Evals before vibes

Before you begin

By the end, you can…

• Build a representative evaluation set
• Choose metrics tied to user outcomes
• Use traces to diagnose regressions

01 / Understand

Evaluation replaces demo-driven development

A few impressive examples reveal possibility, not reliability. Build a dataset from real tasks, common cases, edge cases, failures, and adversarial inputs. Keep a holdout set away from prompt tuning.

Evaluate the whole system: retrieval, answer quality, groundedness, tool choice, safety, latency, cost, and user task success. Aggregate scores can hide catastrophic failures, so track critical slices separately.

02 / Apply

Make failures inspectable

Store traces of inputs, context selection, model and prompt version, tool calls, outputs, validation, and feedback with appropriate privacy controls. A regression should be reproducible.

Use deterministic checks where possible, model graders with calibration where judgment is needed, and human review for high-impact or ambiguous cases. Define thresholds before seeing the new score.

03 / Make

Your studio task

Make — Build a 30-case evaluation set from real tasks, edge cases, and adversarial inputs.

1. Collect 30 representative cases across normal, edge, and adversarial behavior.
2. Define task success, groundedness, safety, latency, and cost measures.
3. Create a holdout set and release thresholds.
4. Run a change, inspect failed slices and traces, then record the ship decision.

04 / Check

Knowledge check