05 Close the loop

Facilitator companion to 05 Close the loop.

Timing

~25 minutes total.

• 5 min — build and promote the three production-derived records.
• 5 min — launch paired policy-v1 and policy-v2 experiments.
• 5 min — compare correctness and calibrate business-policy-adherence.
• 5 min — enable the guarded observation rule and replay four metric types.
• 5 min — inspect the evidence, discuss rollback, cost, and the next feedback loop.

The two experiments and asynchronous evaluators may outlast these facilitation blocks. Start the runs promptly, use the waiting time for the conceptual talk track, and rehearse provider latency the day before.

Instructor preflight

Run these non-mutating command checks from the lab root and verify the selected configuration exists before the session:

cd ClickHouse_Demos/workshops/agent_arena
.venv/bin/python -m scripts.promote_to_golden --help
.venv/bin/python -m scripts.provision_online_evaluators --help
.venv/bin/python -m eval.harness --help
.venv/bin/python -m scripts.verify_online_scores --help

Then confirm:

• Module 03 has one authoritative root chat_turn with sql-execution-success=true and Boolean user-thumbs=false;
• Module 04's UI-only production-investigation-<session> task is completed, its corrected SQL was executed, its real trace ID is available privately, and its annotation task ID is recorded when Langfuse exposes it;
• reviewed.json will be created from that genuine review and is not the tracked synthetic fixture;
• WINNER_MODEL, WINNER_PROMPT, and WINNER_CONFIG_ID all describe the same room winner;
• the Langfuse LLM connection agent-arena-openrouter can reach its configured judge model and has valid credentials;
• the judge's structured categorical output accepts exactly PASS, FAIL, and NOT_APPLICABLE, with reasoning; and
• the evaluator dispatcher is healthy and the room can receive asynchronous scores.

For every rehearsal, create one unique suffix and keep the baseline/candidate pair together:

export LOOP_RUN_SUFFIX="$(date +%Y%m%d-%H%M%S)"
export BASELINE_RUN_ID="online-loop-baseline-${LOOP_RUN_SUFFIX}"
export CANDIDATE_RUN_ID="online-loop-candidate-${LOOP_RUN_SUFFIX}"

Do not reuse a prior session's run IDs. The harness appends the policy version and configuration, and a unique base ID prevents learners from comparing unrelated or partially overwritten evidence.

Manual-annotation boundary

The human annotation in Module 04 is deliberately not automated. Runtime scripts do not create the queue, fill human judgments, enter corrected output, approve the item, or complete the task. A tracked synthetic fixture is useful for an instructor rehearsal when no real review exists, but it remains source=synthetic-reviewed-fixture and does not prove that a human annotation loop was completed.

For the learner path, use genuine ignored reviewed.json. The original wording is one real user-feedback question. The other two exact inputs are reviewer-authored paraphrases derived from that reviewed incident:

How many active customers do we have?
What is our active customer count right now?
How many customers qualify as active under our business definition?

All three preserve the same real source trace and annotation provenance. Say this out loud so learners do not mistake one production incident for three independent feedback traces.

Promotion reliability and provenance

Before the room runs promotion, inspect reviewed.json without projecting it. It must contain the exact corrected read-only SQL, required production provenance, and three unique safe IDs. Promotion first validates the complete local batch, then performs an authenticated metadata read before ClickHouse execution or dataset upserts. It fails closed when metadata cannot be read and refuses a colliding ID with different genuine production provenance.

An identical genuine promotion is idempotent. A synthetic fixture requires the explicit --synthetic-fixture flag; it cannot be supplied as a direct path or combined with reviewed.json. Never run it after real promotion. If a collision is reported, preserve both sources, investigate the existing dataset item, and choose a new audited ID only when the items genuinely represent different reviewed cases.

Experiment rule versus observation rule

Keep these two contexts visible on a slide or whiteboard:

The provisioner installs one general catalog-driven evaluator, not a customer-count-only evaluator. Its policy-v2 catalog covers active customers, revenue, view-to-purchase conversion, and gross margin; unrelated questions should be NOT_APPLICABLE.

The --business-policy-experiments phase must report online rule enabled=False. Check the Langfuse rule UI as a second guard. The later enable command proves that at least one dataset-scoped business-policy-adherence score exists, but it cannot replace the instructor's full calibration review.

Calibration gates

Compare the baseline and candidate on the same item IDs, model, and prompt. The repo contains 20 YAML questions, but q019 and q020 are few-shot holdouts, so a clean project starts with 18 Experiment items and reaches 21 after the three promotions. The verified reused shared project had 22 items only because one older unrelated approved item remained; its fresh paired run moved from 16/22 to 19/22. Treat that as qualified verification evidence, not the learner's required item count or a promise that stochastic providers reproduce every aggregate exactly.

Do not enable unless all gates pass:

• all three prod-active-* cases move from FAIL and incorrect under policy-v1 to PASS and correct under policy-v2;
• candidate revenue (q005) and conversion (q018) are PASS;
• a plain count (q001) is NOT_APPLICABLE;
• every pre-existing item's correctness is compared per item and there is no 1→0 regression;
• aggregate correctness does not regress; and
• every Experiment item has correctness, agent-arena-llm-judge, and the exact business-policy-adherence score with valid structured output.

A judge that labels every count PASS has failed calibration even if the candidate looks good. Use the NOT_APPLICABLE probe to show that policy applicability is a real classification step.

Asynchronous scoring and no-score troubleshooting

Both experiment and observation evaluation are asynchronous. The harness waits for required experiment scores, while scripts.verify_online_scores polls live trace scores for 180 seconds by default. Do not refresh rapidly, recreate rules, or enable early merely because a score is pending.

If scores do not appear:

1. Confirm the exact expected name. Experiments use business-policy-adherence; serving observations use agent-arena-business-policy-online.
2. Confirm the evaluator dispatcher/execution worker is healthy.
3. Check the agent-arena-openrouter connection and judge-model availability. A provider delay, rate limit, or routing restriction can leave evaluation pending or failed even when the agent response succeeded.
4. Confirm the experiment rule targets the arena-golden dataset and the observation rule targets root chat_turn observations. Mapping must expose $.question and $.sql.
5. Inspect structured output. A missing category, category outside the three allowed values, or absent reasoning is a calibration failure.
6. If provisioning reports an ambiguous rule name, stop and resolve duplicate exact rule names in Langfuse before retrying; do not guess which duplicate was updated.

Preserve failed traces and evaluator evidence. Do not turn a provider outage into a fabricated PASS or skip the missing-score gate.

Replay guidance

After enablement, restart serving explicitly with policy-v2 and use the learner's exact four questions:

How many active customers do we have?
What was revenue in the last 30 days?
What is our view-to-purchase conversion rate for the last 7 days?
How many products are there?

Require operational success for every trace. Active customers, revenue, and conversion must have agent-arena-business-policy-online=PASS; the product count must be NOT_APPLICABLE.

The conversion request has a verified stochastic boundary. Permit at most one retry with the same config, preserve both trace IDs and outcomes, and stop if neither attempt passes. Repeated failure is new production evidence to investigate, not a reason to loop until green.

Sampling, cost, and reliability

Workshop rules use sampling 1 so every eligible trace yields visible evidence. At production volume, an LLM judge on every request adds provider cost, consumes rate limit, and may produce scores after the user response. Choose sampling from traffic, incident risk, evaluator cost/latency, and required coverage. Keep deterministic operational checks broad; reserve expensive semantic judgment for the traffic and metrics where it earns its cost.

The evaluator is monitoring, not request-path authorization. A delayed judge must not silently block the serving response. Route missing scores, category drift, and signal disagreements to operational alerts or a review backlog according to the system's service-level objectives.

Rollback and reset

If the observation judge produces false passes, false failures, malformed output, or unacceptable cost/latency after enablement:

1. Open Langfuse Evaluations, find the exact observation rule agent-arena-business-policy-online, and toggle it to disabled.
2. Confirm new chat_turn observations no longer receive that online-rule score.
3. Keep policy-v2, sql-execution-success, and 👍/👎 running unless their own evidence says otherwise; disabling a faulty judge should not reintroduce the known stale policy.
4. Add affected traces to a suffixed human-annotation queue, improve the evaluator catalog/prompt and golden data, then repeat paired experiment calibration before re-enabling.

To stop local services without deleting remote evidence:

scripts/arena.sh stop

scripts/arena.sh down also removes the workshop ClickHouse database and read-only user, but it does not delete Langfuse datasets, Experiment runs, annotation queues, or scores. Do not use it as an evaluator rollback.

Talk track: the loop stays open

• The original evaluator passed because its contract was only “SQL executed.” The user's 👎 exposed a value failure outside that contract.
• Human review converted an uncertain signal into a tested diagnosis and correction.
• Production provenance made the incident auditable when it entered the golden set; paraphrases improved phrasing coverage without inventing extra user traces.
• Paired experiments separated a policy change from a model/prompt change and tested the general judge before production.
• Online PASS does not make user feedback obsolete. A future agent-arena-business-policy-online=PASS plus user-thumbs=false is precisely the kind of disagreement that should restart investigation.

Completion gate

Do not close the module until the room can show all six artifacts:

1. the production trace with operational pass and negative user signal;
2. the completed human annotation and verified corrected SQL;
3. three golden items with genuine production provenance;
4. same-dataset baseline/candidate evidence with no existing correctness regression;
5. calibrated FAIL, PASS, and NOT_APPLICABLE Experiment evidence; and
6. enabled live scores across active customers, revenue, conversion, and a plain count, while continued 👍/👎 remains available for the next loop.