01 Select the base model

Starting point

Module 00 complete: .env sourced, the arena database seeded, Langfuse connected, and the local dashboard reachable at http://localhost:5174 with an empty Leaderboard tab.

Why this is the foundational decision

This is the decision the whole workshop is built around. Before you ship a serious agent, you have to answer a foundational question: which model should power it? Models differ enormously in capability and price, and the best choice depends on your specific task — not on a public leaderboard someone else ran on a different workload. Guessing is costly in both directions: overpay for a frontier model you don't need, or ship a cheap one that quietly gets your real questions wrong.

So instead of guessing, you run a contest: the Arena. A grid of models and prompt strategies all answer the same golden questions, Langfuse grades every answer as an experiment, and the metric that crowns the winner is not raw accuracy but cost per correct answer — quality per dollar for your use case. Concretely, this module answers, with evidence: across a grid of models and prompt strategies, which configuration gets the most correct answers per dollar? "Correct" means execution accuracy — the generated SQL returns the same result set as the golden SQL, not SQL that merely looks plausible. Grading happens inside Langfuse, not inside the harness — Langfuse hosts the evaluators and keeps every Experiment Item, score, and trace. The local leaderboard reads those records through the Langfuse Public API. Everything after this module — measure offline, release and detect, investigate with a human, then prove and monitor the improvement — assumes you've made this choice on evidence.

Concepts — under the hood

Langfuse's data model for this evaluation. The repo source corpus has 20 YAML questions. q019 and q020 are few-shot prompt holdouts, so in a clean project the seeded Dataset (arena-golden) contains 18 experiment questions, each with its expected result set. Every model × prompt configuration you run is one Experiment — a Langfuse Dataset Run — against that same 18-item dataset, so every config is graded on the exact same questions. The evaluator definitions you set up in Step 1 are correctness and llm_judge; their emitted Experiment scores are correctness and agent-arena-llm-judge. One dataset, many experiments, one score per item per experiment — which is what lets the Leaderboard compare configs apples-to-apples.

The prompt strategies are contestants too. The grid isn't just models — it's model × prompt, because how you ask matters as much as who you ask. From config.yaml and agents/prompts.py:

The roster: proprietary vs open-weight. The six contestants split evenly along a second axis that matters just as much as the model name — whether the weights are closed (a vendor API you can only call) or open (a model you could self-host, fine-tune, or keep entirely inside your own data boundary). Open-weight models are often far cheaper per token, while proprietary frontier models may lead on raw capability — but "may" is exactly what this Arena is built to test for your task, not assume. Running both sides through the same golden dataset lets cost-per-correct-answer tell you whether you actually need to pay for the frontier, or whether a cheap open-weight model gets you there for a fraction of the price. The roster below is deliberately low-cost: NL→SQL is a simple enough task that even the priciest contestant here is a mid-tier model, not a frontier one.

Why cost-per-correct, and why execution accuracy. "Correct" is decided by execution accuracy: does running the generated SQL produce the same result set as the golden SQL? That's the honest signal — it doesn't care whether the SQL is byte-for-byte different from the golden query, only whether it answers the question right. The headline ranking metric is then

cost_per_correct_answer = total cost of the run ($) / number of correct answers

which rewards a model that's nearly as accurate but much cheaper over a frontier model that's marginally better but far more expensive — the metric a real cost- conscious team would actually optimize for.

Goal

A populated Leaderboard ranking at least a few model × prompt configurations by cost-per-correct-answer, with every ranking backed by a Langfuse trace you can drill into, and a crowned winner: one config_id.

Step 1 — Set up Langfuse evaluators (one time)

Do this once, following the repo's eval/langfuse_evaluators/README.md. First seed arena-golden and configure the OpenRouter-backed judge through the API:

python -m scripts.provision_langfuse_evaluators

Then configure the deterministic code evaluator in the Langfuse UI:

1. Code evaluator correctness — Evaluators → Set up Evaluator → Code → paste in eval/langfuse_evaluators/correctness_evaluator.py → Target: Experiments → filter dataset = arena-golden. This evaluator compares the agent's result set (from the trace) against the golden result set (the dataset item's expected_output) and emits an execution-accuracy correctness score (0/1) plus an outcome category. It has no network egress — the SQL already ran inside the agent; the evaluator only compares result sets.
2. Manual fallback for evaluator definition llm_judge — Evaluators → Set up Evaluator → LLM-as-a-judge → Custom → use the system/eval prompts and variable mappings from eval/langfuse_evaluators/llm_judge_prompt.md → Target: Experiments, dataset arena-golden → emit the numeric score agent-arena-llm-judge. The evaluator definition and emitted score deliberately have different names. This secondary signal rates SQL quality on top of the primary correctness score — you'll lean on it in Module 02.

The helper is the recommended path; the manual judge step is only a fallback. The deterministic correctness code evaluator remains a one-time UI step.

Step 2 — Run the contest

source .env && python -m eval.harness --run-id demo

What you should see. The harness first prints a summary line (run_id=demo configs=6x3 ...), then one line per question as it runs, e.g.:

claude-sonnet-5__P1_zeroshot q001 pending 812ms $0.00021

Every row starts pending — the SQL ran and the result set was captured on the trace, but the Langfuse evaluators haven't scored it yet. Once every config has finished running, the harness switches to waiting: grading via Langfuse evaluators — waiting on N traces..., printing a countdown as correctness/agent-arena-llm-judge scores land, and finishes with Langfuse scored all N traces; leaderboard ready. That pending → scored handoff is the harness handing grading off to Langfuse; the result and verdict stay together there as the leaderboard's source of truth.

This runs the full model × prompt grid (every model in config.yaml against every prompt strategy, P1_zeroshot through P3_dialect) as Langfuse Dataset Runs (Experiments) against the arena-golden dataset. The harness waits for the exact emitted scores correctness and agent-arena-llm-judge on every item. Exact OpenRouter cost and end-to-end latency are stored on the same Experiment Item.

A single config is <model>__<prompt>, e.g. claude-sonnet-5__P1_zeroshot. The available names come straight from config.yaml:

• Models: the six contestants in the roster table above — three proprietary (claude-sonnet-5, gpt-5.6-luna, gemini-flash-lite) and three open-weight (deepseek-v4-flash, qwen3.7-flash, glm-4.7-flash)
• Prompts: P1_zeroshot, P2_fewshot, P3_dialect

Useful flags:

• --models qwen3.7-flash,gpt-5.6-luna / --prompts P1_zeroshot,P3_dialect — restrict the grid to a CSV subset instead of running everything.
• --run-id <name> — tag the run so it's easy to find on the Leaderboard and in Langfuse's Experiments view.

The SDK may process dataset items in reverse or concurrent order; use the question ID on each line rather than expecting q001, q002, ... output order. A full 18-config grid typically takes 35–45 minutes. Start workshops with a two-model, one-prompt subset and run the full grid only when the schedule and provider limits allow it.

Step 3 — Crown the winner

Open http://localhost:5174 → Leaderboard. Every model × prompt config is ranked by cost-per-correct-answer — the headline metric. A cost × accuracy chart and a "best value" ranking sit above the table.

Cost is computed from live OpenRouter pricing — the harness refreshes model prices from OpenRouter's /models endpoint at the start of each run, so cost-per-correct-answer reflects what the model actually costs today, not a stale number baked into config.yaml.

How to read it. Sort order is cost-per-correct-answer ascending — the winner is the top row, not the row with the highest accuracy. Watch the cost × accuracy chart for a cheap model sitting near an expensive one on the accuracy axis: that gap, at a fraction of the cost, is the whole reason this metric exists instead of a plain accuracy leaderboard.

The top row is your winner: note its config_id. Module 02 goes deep on exactly how good it is, not just that it won.

How to verify you are done

• The Leaderboard table shows at least one model × prompt row with a cost-per-correct-answer value (not empty).
• Clicking a config's row shows per-question results, and clicking a question opens its Langfuse trace with the generated SQL visible.
• You can name the config_id (<model>__<prompt>) the Arena crowned as the winner.

Exercise — predict, then verify

Before you open the Leaderboard for real, make a prediction and write it down:

1. Looking only at the model list and the prompt table above (not the Leaderboard), guess which model × prompt config you think will win on cost-per-correct-answer. Write down the config_id and one sentence on why (e.g. "cheapest model paired with the dialect prompt, because most failures are dialect mistakes, not reasoning mistakes").
2. Now open the Leaderboard and check. Were you right?
3. Whatever the result, answer this: did a cheaper model beat (or come close to) a frontier model? If so, that gap — cheap-and-nearly-as-good beating expensive-and-slightly-better — is the point of ranking by cost-per-correct-answer instead of by raw accuracy. If a frontier model won outright, note by how much it beat the next-cheapest config — that margin is what would justify its price in a real deployment decision.

Wrap-up

You now have evidence, not a guess, for which model and prompt strategy is worth running — ranked by cost per correct answer and backed by Langfuse traces for every run. Take note of your winning config_id; you'll use it in every module from here on.

End state

A ranked leaderboard and a crowned config_id. Continue to 02 Measure offline to see how good that winner really is.