03 Release and detect

Facilitator companion to 03 Release and detect.

Timing

~15 minutes total.

• 3 min — frame operational evaluation versus user value.
• 4 min — show the preflight and release the room's selected config on policy-v1.
• 4 min — ask the governed question in Chat and collect 👎 on that answer.
• 4 min — locate that Chat trace, verify both scores, and reproduce with curl.

Preflight the day before

Run this with the exact winner you expect the room to select. The verified fallback is qwen3.7-flash__P2_fewshot:

cd ClickHouse_Demos/workshops/agent_arena
source .env
export WINNER_CONFIG_ID="${WINNER_CONFIG_ID:-qwen3.7-flash__P2_fewshot}"
.venv/bin/python -m schema.gen_schema_context
.venv/bin/python -m scripts.check_online_eval_scenario \
  --config-id "$WINNER_CONFIG_ID"
.venv/bin/python -m scripts.provision_online_evaluators --operational

The preflight performs one bounded retry only when a paraphrase's first result is ok/unknown, and retries only that same paraphrase and configuration. All other failures are final, and it never retries more than once.

Do not proceed from memory. Confirm all of the following in the room's actual environment:

• stale_count and current_count are both present and different;
• all three classifications are policy-v1;
• the final line says the incident is reproducible;
• evaluator sql-execution-success exists; and
• rule agent-arena-sql-execution-online is enabled.

For Qwen, OpenRouter Settings → Privacy → Data Policies → Zero Data Retention → Non-frontier must be disabled so the Alibaba route is eligible. Make this change only after reviewing the data-handling requirement for the event. Learners need a capped runtime key, never the instructor's provisioning key.

Talk track

• Start with the room's actual Module 02 winner and write WINNER_CONFIG_ID where everyone can see it. The agent core and selected configuration are unchanged; only the deployed business-policy context is deliberately one version behind.

• Name the evaluator's boundary before showing a result: sql-execution-success can prove that ClickHouse accepted the SQL, not that the SQL implements today's meaning of a governed metric.

• Ask in Chat, show its generated SQL, result, and policy_version=policy-v1, then click 👎 and wait for feedback sent. This Chat root trace is the single authoritative incident.

• Show the audience the current definition side by side:

  SELECT uniqExact(customer_id) FROM v_orders
  WHERE order_ts >= now() - INTERVAL 30 DAY
  AND status NOT IN ('cancelled', 'returned')

• Say explicitly that the generated signup-based SQL is valid under the stale policy-v1 supplied to the model. This is a release/process failure and evaluator blind spot, not a claim that the model ignored clear instructions.

• In Langfuse, filter user-thumbs = false, open the newest matching Chat chat_turn, and show sql-execution-success=true beside user-thumbs=false. The signal prioritizes an investigation; it does not supply the diagnosis or become ground truth by itself.

• Run the mandatory curl reproduction afterward as an unrated diagnostic. Name its identifier CURL_TRACE_ID, verify only sql-execution-success=true, and never POST feedback for it or use it as the Module 04 handoff.

• Record the root trace ID/URL and both counts for Module 04. Keep live identifiers inside the Langfuse project and workshop worksheet.

Expected trace evidence

Open the root chat_turn observation, not only its child llm_call. Expect:

The serving source names this field policy_version; the OpenTelemetry adapter sanitizes metadata keys to alphanumeric characters, so Langfuse displays the emitted key as policyversion.

The operational evaluator is asynchronous. Use the verifier rather than treating a score that has not appeared yet as failure:

.venv/bin/python -m scripts.verify_online_scores "$CHAT_TRACE_ID" \
  sql-execution-success=true user-thumbs=false

Common failures

• Provider blocked by ZDR — Qwen fails before producing SQL when OpenRouter's non-frontier Zero Data Retention requirement is enabled and the eligible Alibaba route is disallowed. Disable that ZDR restriction for this workshop or use the disclosed fallback after reviewing privacy requirements.
• Evaluator dispatcher is not running — the trace arrives but sql-execution-success never appears. Confirm Langfuse's evaluator execution service/dispatcher is healthy and the agent-arena-sql-execution-online rule is enabled; provisioning a rule does not process scores if evaluation workers are unavailable.
• Missing OpenTelemetry dependencies — no trace appears even though /ask returns. Reinstall the pinned lab requirements with .venv/bin/python -m pip install -r requirements.txt; the runtime uses the Langfuse v4 OpenTelemetry path and requires its compatible OTel packages.
• Stale server process — the response reports policy-v2 even though the shell command says policy-v1. An older process still owns port 8100; stop it completely before starting the seeded release.
• Wrong port — the Chat UI defaults to http://localhost:8100. If serving uses a different port, set VITE_SERVING_BASE to the same address or use the raw curl against the actual port.
• Duplicate feedback — feedback uses the deterministic score ID user-thumbs-<trace_id>. Submit one rating per trace. Reusing the same trace for competing ratings can return a feedback-service error or leave an ambiguous demo; create a fresh session/trace instead.
• Score still pending — evaluation is asynchronous. Let scripts.verify_online_scores poll for the score before changing configuration.
• Reference counts match — the seeded contrast is absent in this data snapshot. Do not manufacture a failure; reseed or diagnose the data before the session.

Reset steps

Stop any existing server, then start a clean policy-v1 process:

scripts/arena.sh stop
scripts/arena.sh serve
source .env
.venv/bin/python -m schema.gen_schema_context
AGENT_ARENA_POLICY_VERSION=policy-v1 \
  .venv/bin/uvicorn serving.api:app --port 8100

scripts/arena.sh serve restores the dashboard and web UI in the background before the serving API takes over the terminal. Refresh the Chat page to create a fresh session. If you use the raw API, pass a new session ID or omit it so /ask creates one automatically. Re-run the preflight and provisioner in a second terminal; both are safe to repeat.

Fallback policy

Use the instructor's known-good qwen3.7-flash__P2_fewshot configuration only if the room winner no longer follows the explicit stale policy in the three-question preflight. State the substitution aloud: the fallback preserves a deterministic teaching incident, while the leaderboard winner remains the room's measured result. Do not silently swap models, and do not use the fallback to hide an equal-count, credential, provider-routing, or evaluator-infrastructure failure.

Handoff to Module 04

Before moving on, confirm the worksheet contains the authoritative Chat root trace ID/URL, stale count, current count, sql-execution-success=true, and Boolean user-thumbs=false. Module 04 starts from that exact disagreement and adds human judgment; it must not begin with a prewritten diagnosis detached from the production trace.