Evals

Prerequisites: The Agent Loop — you need trajectories before you can grade them. Feeds problems: Evaluation without ground truth, Reliability over long horizons, The tool interface gap.

Practitioner

Here’s the trap evals exist to escape. You tweak the system prompt, run your favorite three tasks, they pass, you ship. A week later a category of task you didn’t try is broken, and you don’t know which of the last five tweaks broke it. Agent outputs are samples from a distribution; judging a change from a few runs is reading tea leaves with extra steps.

Start embarrassingly small and real. Twenty tasks pulled from actual usage — real tickets, real repos, real requests — beat five hundred synthetic ones, because synthetic tasks inherit your blind spots: you generate the cases you already thought of, then congratulate yourself for passing them. For each task write down the input, what a good outcome looks like, and how you’ll check it. That last field forces the key design choice:

• Programmatic checks — tests pass, the JSON validates, the answer matches, the file exists. Cheap, objective, incorruptible. Use whenever the domain allows; this is why coding agents improved fastest — their domain grades itself.
• LLM-as-judge — a model grades the output against a rubric you write (“correct? complete? did it cite the actual document?”). This is how you grade the ungradable, but calibrate before trusting: grade twenty outputs yourself, compare with the judge, and fix the rubric until you agree roughly nine times in ten. An uncalibrated judge is a random number generator with gravitas.
• Human grading — the gold standard and the calibration source for judges. Too expensive for every run; essential for anchoring.

Grade the trajectory, not just the destination. Two agents can both succeed while one takes 9 turns and the other takes 34, calls tools it didn’t need, and stumbles into the answer. Outcome metrics (pass rate) tell you whether; trajectory metrics (turns, tokens, wasted calls, tool-selection accuracy) tell you how, and they catch degradation before it reaches the outcome. Log both from day one — turns and tokens are free to collect.

Respect the variance. Run each task several times. Report pass@k or average pass rate, and before celebrating a two-point gain on twenty tasks, notice that’s smaller than your noise. The sample sizes that make single runs meaningful don’t exist at agent scale; the honest posture is “directionally better, confirmed across reruns,” not “up 3.2%.”

The loop that actually improves agents: change one thing → run the eval → read the failing trajectories → fix what you saw → rerun. The italicized step is where the information is. Aggregate scores tell you something broke; only the transcript tells you what — the model never saw the config file, the tool returned an empty string it read as success, the compaction ate the constraint. Practitioners who read trajectories fix causes; practitioners who watch dashboards fix symptoms. Budget real time for it: it’s the code review of this field.

Don’t let the eval fossilize. In production, log full traces. When a real failure comes in, reproduce it as an eval task and leave it there — the suite becomes a ratchet: every fixed failure stays fixed or fails loudly. An eval that stops growing becomes a target you’ve memorized; hitting 95% on it means less every month. Rotate in what production actually serves you.

Expert pointers

Judge reliability is an active research front: judges favor verbose answers, earlier-position candidates, and their own model family, and rubric design to counteract this is craft knowledge worth stealing from published rubrics. Public benchmarks (SWE-bench and kin) are useful for comparing models and nearly useless for predicting your task — treat leaderboard deltas as weather reports. The uncomfortable open problem: evaluating multi-turn quality on genuinely open-ended work (research, writing, judgment) still bottoms out in human taste, and everyone’s judge rubrics are approximations of it.

Misconceptions

• “Evals are the launch gate.” They’re the daily development loop. Teams that eval only before release iterate blind the rest of the quarter.
• “It passed three demos, it works.” You sampled three successes from a distribution. The distribution is the product.
• “A strong benchmark score means it’ll work for us.” Benchmarks measure the benchmark’s task shape. Your eval, on your tasks, is the only number that transfers.

Check yourself

1. Your pass rate fell from 80% to 70% after a model upgrade. What do you look at first, and why isn’t it the prompt?
2. Design the grading for: (a) an agent that fixes failing CI builds, (b) an agent that drafts replies to customer emails. Why do the methods differ, and which needs calibration?
3. A colleague’s new prompt “wins” 8 tasks and “loses” 5 out of 40, single run each. What do you say before this ships?
4. Why does grading only outcomes let quality rot invisibly? Name two trajectory metrics that would catch it.

Apply it

Build a ten-task eval for your Agent Loop agent: ten real questions about a repo you know, each with a programmatic check (expected file named, expected function identified — string matching is fine). Run the suite three times and look at the variance you get for free. Then read one failing trajectory end to end and write down the actual cause. This suite is the measuring stick for every other Apply-it — and a prototype-tier capstone is this suite, grown up.