04 Investigate

Facilitator companion to 04 Investigate.

Timing

~20 minutes total.

• 3 min — filter negative feedback and verify the authoritative Chat root.
• 4 min — create the three score configs, then the queue with fixed attachments.
• 4 min — open-code observable behavior before discussing a diagnosis.
• 5 min — inspect trace evidence and run policy-v1/policy-v2 SQL side by side.
• 4 min — enter the correction, approve, complete, and record provenance.

Instructor preflight

Before learners arrive, confirm Module 03 produced the one authoritative chat_turn with Boolean user-thumbs=false and sql-execution-success=true. Keep its trace ID private and confirm the root output contains the question, generated SQL, returned rows, and outcome.

Create the three score configs in a disposable test project if you need to rehearse, but do not pre-create the learners' final queue. The queue's attached score-config ID set is fixed at creation, so the room should create the configs first and attach all three:

This annotation exercise is UI-only. Runtime scripts can verify trace scores and later promote a reviewed export, but they do not create the queue, write human judgments, or complete the annotation task.

Talk track

1. Filter Tracing for user-thumbs = false and match the trace ID from the Module 03 worksheet. Say: “feedback decides what we investigate next, not what we conclude.”

2. Use Settings → Scores → Create for each score config. Then use Annotations → Queues → Create, name the queue production-investigation-<session> with a unique session suffix, and attach all three configs.

3. Select the root chat_turn observation, open its Annotate dropdown, and select the new queue. Show the child llm_call, but explain why it is the wrong target: it lacks the end-to-end structured execution result and is not the logical feedback incident.

4. Remind learners that Module 03 disclosed the seeded setup, then ask them to bracket that prior knowledge and practice observable open-coding. A good first note is:

   The SQL executed and returned a count. The observed count differs from the second
   reference count. The query uses a 90-day customer signup window, and the trace
   metadata reports policy-v1.

5. Only after the room records that note, reveal the full evidence: emitted metadata policyversion=policy-v1, the signup-based SQL/result, and the two scores. The source field is policy_version; the OpenTelemetry adapter emits the sanitized Langfuse key policyversion.

6. Run the two policy definitions side by side. The policy-v1 reference is:

   SELECT count() FROM v_customers
   WHERE signup_date >= today() - INTERVAL 90 DAY

   The policy-v2 current-policy reference is:

   SELECT uniqExact(customer_id) FROM v_orders
   WHERE order_ts >= now() - INTERVAL 30 DAY
   AND status NOT IN ('cancelled', 'returned')

7. Only after that comparison, apply the diagnosis: record failure-category=stale-business-policy, switch Corrected Output to plain-text mode, and paste the raw current query. Langfuse does not execute SQL, so verify the exact text through Step 6's read-only client before setting approved-for-golden=true and completing the task.

8. Preserve source_trace_id, failure_category, source_policy_version, the correction, and the optional annotation task ID for Module 05.

Three tempting but wrong diagnoses

• “The model ignored the prompt.” The trace's SQL follows the explicit policy-v1 context. The failure is the stale deployed policy, not disobedience to the supplied instruction.
• “sql-execution-success is broken.” The SQL executed, so the operational evaluator correctly returned true. Its design does not test governed business meaning.
• “A thumbs-down proves the SQL is wrong.” Feedback identifies a trace worth reviewing. It does not disambiguate the user's intent or provide a verified replacement query; the side-by-side policy check and human review do that.

Keep these alternatives visible until learners have open-coded the trace. The seeded answer is known to the instructor, but the investigation should still model a real evidence-first review.

Root-observation targeting

There are two relevant observations in the trace:

If a learner adds the child by mistake, do not complete it as the production investigation. Add the root chat_turn to the correct queue and leave/delete the mistaken task according to the project's retention policy. Preserve the trace ID, not the child observation ID, as source_trace_id.

Fixed queue attachments and reset-safe naming

Langfuse fixes the set of score-config IDs attached to a queue when the queue is created. A queue cannot attach an omitted config later, so creating all configs first is the safest practice. Score configs themselves are mutable: supported name, schema, or categorical-value edits require an audited score-config update, and that update does not rewrite existing scores.

Use a reset-safe suffix such as:

production-investigation-<session>-retry-1

If the queue omitted a config or attached the wrong config ID, create a new suffixed queue with all three correct IDs, add the authoritative root again, and clearly mark the old queue as superseded (or remove it only if project policy permits). For a supported edit to an already attached config, use the audited score-config update instead of recreating the queue. Never reuse a completed queue name in a way that obscures which task supplied the decision.

Corrected-output reliability

The correction becomes future golden ground truth, so syntax and policy both matter. Switch Corrected Output to plain-text mode before pasting raw SQL. Langfuse stores the text but does not execute it; before approval, execute the exact text through Step 6's read-only ClickHouse client. A correct-looking query that is malformed, references raw tables, contains multiple statements, or fails in ClickHouse must remain unapproved.

For malformed SQL:

1. set or leave approved-for-golden=false;
2. do not complete the task as approved;
3. fix the correction against the permitted v_* views;
4. rerun it and inspect the result; and
5. approve and complete only after verification.

If someone completed a malformed correction, create a fresh queue/task suffix and redo the review rather than quietly editing away the audit trail. Module 05's promotion command also validates read-only SQL and executes it, but that guard is not a substitute for human review.

Common failures

• No trace after filtering — confirm the score type is Boolean and the filter is user-thumbs=false; do not search for an old score name. Match the worksheet trace ID rather than selecting the unrated curl diagnostic.
• Wrong target — the task shows only transcript/SQL because llm_call was added. Return to the trace and add the root chat_turn.
• Missing queue dimension — the queue was created without a required attached config ID. Create a new suffixed queue; do not continue with an incomplete review form. Use an audited config update, not queue recreation, for a supported edit to an already attached config.
• Policy metadata appears absent — look for emitted policyversion, not source policy_version, then confirm the root's policy-v1 tag.
• Counts unexpectedly match — stop the diagnosis. Re-run Module 03's scenario preflight and do not manufacture evidence from a data snapshot without contrast.
• Correction is formatted instead of raw SQL — switch Corrected Output to plain-text mode and paste only the query.
• Correction does not execute — Langfuse will not catch this. Keep approval false, fix it, and rerun the exact text through the Step 6 client before completing the task.

Completion and handoff

Before moving to Module 05, verify the completed task is for the authoritative chat_turn, the observation was recorded before the diagnosis, the correction is the exact current-policy SQL, and the task is approved. The learner worksheet must carry:

source=production-feedback
source_trace_id=<authoritative Chat trace ID>
failure_category=stale-business-policy
source_policy_version=policy-v1
annotation_id=<task ID when available>

The negative user score remains on the production trace as the triage signal. The human annotation supplies the reviewed ground truth. Module 05 will preserve both origins when it promotes the correction and builds a preventative evaluator.