02 Measure offline

Facilitator companion to the learner lesson 02 Measure offline.

Timing

~15 minutes total.

• 5 min — per-tier accuracy and outcome breakdown on the winning config's Leaderboard detail view.
• 4 min — the agent-arena-llm-judge secondary score, and where it disagrees with correctness.
• 6 min — drill into two or three individual Langfuse traces for wrong/low-scoring questions.

Talk track

• Reframe the goal: winning the Arena tells you a config beat the rest in aggregate; this module is about how it wins and where it's weakest — same idea as knowing not just that a candidate passed an interview, but which questions they nailed.
• Say explicitly that this module produces no new data — everything here was already captured by correctness and by the llm_judge evaluator definition's emitted score, agent-arena-llm-judge, in Module 01. This is a reading exercise, not a re-run.
• Reconfirm the denominator: the repo source has 20 YAML questions, but q019 and q020 are few-shot holdouts, so the Experiment contains 18 scored dataset items.
• On per-tier accuracy: point out that a config can look strong overall while being shaky on the hardest tier, and that's exactly what an aggregate leaderboard number hides.
• On the outcome breakdown: walk through the categories out loud — sandbox-rejected SQL, a ClickHouse error, an empty result, a wrong result set — and that each is a different kind of problem with a different fix, not one undifferentiated "failure."
• On agent-arena-llm-judge: it's binary-correctness's finer-grained sibling — a config can be correct by execution accuracy while writing SQL a reviewer would still flag (an unnecessary subquery, a fragile date comparison). Find one live disagreement between correctness and agent-arena-llm-judge if you can, it's the clearest way to make the distinction land.
• Close by picking two or three wrong/low-scoring questions and reading their full traces end to end live — prompt sent, generated SQL, error or result — looking out loud for a pattern (a phrasing, a join, a date filter the model keeps mishandling). This is the same trace-reading skill Module 04 reuses to investigate the feedback-marked production trace. End by carrying the selected config_id into Module 03 for release.

Common failures

• Langfuse evaluators not configured — there is no agent-arena-llm-judge or correctness score to build this module around. Return to Module 01, fix the evaluator target/filter, and run a small fresh grid before continuing.
• No wrong answers to drill into — if the winning config scored 100% on the demo grid, pick a non-winning config's failures instead; the trace-reading skill is the point, not finding a flaw in the winner specifically.
• ClickHouse not seeded / harness never ran — the Leaderboard detail view (per-tier accuracy, outcome breakdown) is empty. This means Module 01 didn't finish; go back and re-run it before continuing.
• Clicking through to a trace 404s or loads the wrong project — usually means the browser is pointed at a different Langfuse project than the one in .env, or LANGFUSE_BASE_URL/keys don't match the account that ran Module 01's harness.

Reset steps

• If the Leaderboard detail view is empty, re-seed and re-run the cheap subset: scripts/arena.sh up, then python -m eval.harness --run-id demo2 --models qwen3.7-flash,gpt-5.6-luna --prompts P1_zeroshot,P3_dialect.
• Use a fresh --run-id for the re-run so it's unambiguous which Leaderboard rows and which Langfuse Experiment you're reading in the room.
• If only the dashboard rendering is stuck (Experiments exist in Langfuse), restart the local servers without re-seeding: scripts/arena.sh stop && scripts/arena.sh serve.
• If evaluators were the gap, this module's depth depends on fixing that before the next session — there's no in-session substitute for the Langfuse-side scores.