Agent Arena — Langfuse Workshop
Put a roster of LLMs into a contest, crown the winner for NL→SQL over ClickHouse, and keep improving it — all instrumented with Langfuse from the very first step.
Welcome to the Agent Arena playbook. Agent Arena is a head-to-head contest for LLM agents: over the course of this workshop you put a roster of LLMs into a contest for one specific job — natural-language-to-SQL over a ClickHouse-backed e-commerce dataset — and crown a winner using evidence, not a public leaderboard. Langfuse is connected from the very first module, not bolted on at the end: it runs the contest, measures the winner's quality, drives continuous improvement, and carries through into production.
Why this workshop
When you build a serious agent application, one of the first and most consequential decisions is which model to use. Models vary enormously in both capability and price, and the right choice depends on your specific task — not on someone else's public leaderboard. Guess here and you lose in one of two directions: overpay for a frontier model you didn't need, or ship a cheap one that quietly gets your real workload wrong.
The disciplined answer is to run the contest yourself: put a grid of models and prompt strategies through Langfuse experiments against your own golden dataset, and let cost per correct answer — quality per dollar for your use case — pick the winner. That decision is the foundation everything else rests on; there's no point measuring, improving, or releasing an agent built on the wrong model.
This workshop makes that concrete with an NL→SQL chatbot as the use case, and one flow: select the base model → measure offline → release and detect → investigate with a human → prove and monitor the improvement. Langfuse is the thread that ties every step together — the same project, traces, feedback, annotations, evaluators, and datasets from Module 00 through the production improvement loop.
This playbook is dual-track. The Learner track is the lesson you follow in the room. The Instructor track is the facilitator's companion for the same module: timing, talk track, common failures, and reset steps.
Modules
| # | Learner | Instructor | Outcome |
|---|---|---|---|
| 00 | Setup | notes | OpenRouter, ClickHouse, and Langfuse connected — Langfuse wired in before any model is chosen — and the Agent Arena dataset seeded |
| 01 | Select the base model | notes | The contest run as Langfuse experiments; a winner crowned by cost per correct answer — the foundational decision |
| 02 | Measure offline | notes | The winner's quality understood per question and per tier before release, using Langfuse evaluators, datasets, and traces |
| 03 | Release and detect | notes | The selected agent released with a known policy blind spot; executable SQL passes the operational evaluator while real user feedback flags the answer |
| 04 | Investigate | notes | A human follows the negative feedback to the authoritative trace, diagnoses the stale policy, verifies a correction, and records production provenance |
| 05 | Close the loop | notes | The reviewed incident becomes golden data; paired experiments prove the improvement, a general policy evaluator is calibrated and enabled online, and future traffic is monitored |