Result
12/12 scenario repetitions passed · 384 assertions · 1 manual-only scenario skipped
Each repetition is a full live conversation: a question generates SQL (writer model drafted, reviewer model checked), the query is validated and executed against PostgreSQL, then a follow-up instruction refines the exact current query and the successor is validated and executed again. Executed results are re-checked against an independently-authored gold query (byte-level row-set match) and an independent read-only PostgreSQL cross-check.
Dataset full-20260720T214007Z (openelis-fhir-postgresql): 96 patients · 1152 results · 9 test types · 2025-07-15 – 2026-04-27
| profile | writer model | reviewer model |
|---|---|---|
catalyst-query-gemma-4-12b | gemma-4-12b | qwen2.5-14b |
catalyst-query-qwen-2.5-14b | qwen2.5-14b | gemma-4-12b |
What each scenario does
narrowing-unchanged-base
Question — Show laboratory results since 2026-01-01 with patient, test, status, value, unit, and observed date.
Follow-up — Restrict the current query to final Viral Load results, preserving all selected columns, the existing date filter, and ordering.
3 repetitions — all passed · base query returned 100 rows · refined query returned 100 rows
aggregation-dirty-base-profile-switch
Question — Show numeric laboratory results since 2026-01-01 with patient, test, value, and observed date.
Follow-up — Replace the current row output with one row per calendar month. Return observed_month, result_count, distinct_patient_count, and median_result_value; preserve all filters and order chronologically.
3 repetitions — all passed · base query returned 100 rows · refined query returned 4 rows
unresolved-parameter-correction
Question — Show recent laboratory results.
Follow-up — Repair the unresolved parameter binding without changing columns, table, predicate values, or ordering.
3 repetitions — all passed · base query returned 100 rows · refined query returned 100 rows
semantic-distinct-patient-review
Question — Show patient identifiers represented in laboratory results since 2026-01-01.
Follow-up — Return one column named patient_count containing the number of distinct patients represented by these rows; do not count result rows.
3 repetitions — all passed · base query returned 100 rows · refined query returned 1 rows
bounded-hub-tool-failure
Skipped — manual-only scenario (requires an operator-controlled external failure).
Scenario matrix
| scenario | reported | gold ok | judge medians (base/succ) | assertions | gold FAIL detail |
|---|---|---|---|---|---|
narrowing-unchanged-basenarrowing | PASS | yes | — / — advisory | 33/33 passedsession_created, initial_turn_recorded, initial_evidence_available, base_version_saved, base_version_available, base_validation_recorded, base_execution_succeeded, base_postgres_crosscheck, base_gold_execution_match, followup_http_created, followup_terminal_status, base_classification, manual_version_classification, followup_profile, refresh_restored, timeline_current_turn, followup_evidence_available, invocation_duration_sum, invocation_digests, invocation_timestamp_reconciliation, writer_model, reviewer_model, effective_temperature_and_dry, revision_context_exclusions, exact_selected_output, semantic_reviewer_correction, prior_results_stale_after_successor, successor_validation_recorded, successor_execution_succeeded, successor_postgres_crosscheck, successor_gold_execution_match, successor_visible_under_three_minutes, new_session_isolation | — |
narrowing-unchanged-basenarrowing | PASS | yes | — / — advisory | 33/33 passedsession_created, initial_turn_recorded, initial_evidence_available, base_version_saved, base_version_available, base_validation_recorded, base_execution_succeeded, base_postgres_crosscheck, base_gold_execution_match, followup_http_created, followup_terminal_status, base_classification, manual_version_classification, followup_profile, refresh_restored, timeline_current_turn, followup_evidence_available, invocation_duration_sum, invocation_digests, invocation_timestamp_reconciliation, writer_model, reviewer_model, effective_temperature_and_dry, revision_context_exclusions, exact_selected_output, semantic_reviewer_correction, prior_results_stale_after_successor, successor_validation_recorded, successor_execution_succeeded, successor_postgres_crosscheck, successor_gold_execution_match, successor_visible_under_three_minutes, new_session_isolation | — |
narrowing-unchanged-basenarrowing | PASS | yes | — / — advisory | 33/33 passedsession_created, initial_turn_recorded, initial_evidence_available, base_version_saved, base_version_available, base_validation_recorded, base_execution_succeeded, base_postgres_crosscheck, base_gold_execution_match, followup_http_created, followup_terminal_status, base_classification, manual_version_classification, followup_profile, refresh_restored, timeline_current_turn, followup_evidence_available, invocation_duration_sum, invocation_digests, invocation_timestamp_reconciliation, writer_model, reviewer_model, effective_temperature_and_dry, revision_context_exclusions, exact_selected_output, semantic_reviewer_correction, prior_results_stale_after_successor, successor_validation_recorded, successor_execution_succeeded, successor_postgres_crosscheck, successor_gold_execution_match, successor_visible_under_three_minutes, new_session_isolation | — |
aggregation-dirty-base-profile-switchaggregation-output-shape | PASS | yes | — / — advisory | 31/31 passedsession_created, initial_turn_recorded, initial_evidence_available, base_version_available, base_validation_recorded, base_execution_succeeded, base_postgres_crosscheck, followup_http_created, followup_terminal_status, base_classification, manual_version_classification, followup_profile, refresh_restored, timeline_current_turn, followup_evidence_available, invocation_duration_sum, invocation_digests, invocation_timestamp_reconciliation, writer_model, reviewer_model, effective_temperature_and_dry, revision_context_exclusions, exact_selected_output, semantic_reviewer_correction, prior_results_stale_after_successor, successor_validation_recorded, successor_execution_succeeded, successor_postgres_crosscheck, successor_gold_execution_match, successor_visible_under_three_minutes, new_session_isolation | — |
aggregation-dirty-base-profile-switchaggregation-output-shape | PASS | yes | — / — advisory | 31/31 passedsession_created, initial_turn_recorded, initial_evidence_available, base_version_available, base_validation_recorded, base_execution_succeeded, base_postgres_crosscheck, followup_http_created, followup_terminal_status, base_classification, manual_version_classification, followup_profile, refresh_restored, timeline_current_turn, followup_evidence_available, invocation_duration_sum, invocation_digests, invocation_timestamp_reconciliation, writer_model, reviewer_model, effective_temperature_and_dry, revision_context_exclusions, exact_selected_output, semantic_reviewer_correction, prior_results_stale_after_successor, successor_validation_recorded, successor_execution_succeeded, successor_postgres_crosscheck, successor_gold_execution_match, successor_visible_under_three_minutes, new_session_isolation | — |
aggregation-dirty-base-profile-switchaggregation-output-shape | PASS | yes | — / — advisory | 31/31 passedsession_created, initial_turn_recorded, initial_evidence_available, base_version_available, base_validation_recorded, base_execution_succeeded, base_postgres_crosscheck, followup_http_created, followup_terminal_status, base_classification, manual_version_classification, followup_profile, refresh_restored, timeline_current_turn, followup_evidence_available, invocation_duration_sum, invocation_digests, invocation_timestamp_reconciliation, writer_model, reviewer_model, effective_temperature_and_dry, revision_context_exclusions, exact_selected_output, semantic_reviewer_correction, prior_results_stale_after_successor, successor_validation_recorded, successor_execution_succeeded, successor_postgres_crosscheck, successor_gold_execution_match, successor_visible_under_three_minutes, new_session_isolation | — |
unresolved-parameter-correctionunresolved-correction | PASS | yes | — / — advisory | 31/31 passedsession_created, initial_turn_recorded, initial_evidence_available, base_version_available, base_validation_recorded, base_execution_succeeded, base_postgres_crosscheck, followup_http_created, followup_terminal_status, base_classification, manual_version_classification, followup_profile, refresh_restored, timeline_current_turn, followup_evidence_available, invocation_duration_sum, invocation_digests, invocation_timestamp_reconciliation, writer_model, reviewer_model, effective_temperature_and_dry, revision_context_exclusions, exact_selected_output, semantic_reviewer_correction, prior_results_stale_after_successor, successor_validation_recorded, successor_execution_succeeded, successor_postgres_crosscheck, successor_gold_execution_match, successor_visible_under_three_minutes, new_session_isolation | — |
unresolved-parameter-correctionunresolved-correction | PASS | yes | — / — advisory | 31/31 passedsession_created, initial_turn_recorded, initial_evidence_available, base_version_available, base_validation_recorded, base_execution_succeeded, base_postgres_crosscheck, followup_http_created, followup_terminal_status, base_classification, manual_version_classification, followup_profile, refresh_restored, timeline_current_turn, followup_evidence_available, invocation_duration_sum, invocation_digests, invocation_timestamp_reconciliation, writer_model, reviewer_model, effective_temperature_and_dry, revision_context_exclusions, exact_selected_output, semantic_reviewer_correction, prior_results_stale_after_successor, successor_validation_recorded, successor_execution_succeeded, successor_postgres_crosscheck, successor_gold_execution_match, successor_visible_under_three_minutes, new_session_isolation | — |
unresolved-parameter-correctionunresolved-correction | PASS | yes | — / — advisory | 31/31 passedsession_created, initial_turn_recorded, initial_evidence_available, base_version_available, base_validation_recorded, base_execution_succeeded, base_postgres_crosscheck, followup_http_created, followup_terminal_status, base_classification, manual_version_classification, followup_profile, refresh_restored, timeline_current_turn, followup_evidence_available, invocation_duration_sum, invocation_digests, invocation_timestamp_reconciliation, writer_model, reviewer_model, effective_temperature_and_dry, revision_context_exclusions, exact_selected_output, semantic_reviewer_correction, prior_results_stale_after_successor, successor_validation_recorded, successor_execution_succeeded, successor_postgres_crosscheck, successor_gold_execution_match, successor_visible_under_three_minutes, new_session_isolation | — |
semantic-distinct-patient-reviewlint-clean-semantic-reviewer-correction | PASS | yes | — / — advisory | 33/33 passedsession_created, initial_turn_recorded, initial_evidence_available, base_version_saved, base_version_available, base_validation_recorded, base_execution_succeeded, base_postgres_crosscheck, base_gold_execution_match, followup_http_created, followup_terminal_status, base_classification, manual_version_classification, followup_profile, refresh_restored, timeline_current_turn, followup_evidence_available, invocation_duration_sum, invocation_digests, invocation_timestamp_reconciliation, writer_model, reviewer_model, effective_temperature_and_dry, revision_context_exclusions, exact_selected_output, semantic_reviewer_correction, prior_results_stale_after_successor, successor_validation_recorded, successor_execution_succeeded, successor_postgres_crosscheck, successor_gold_execution_match, successor_visible_under_three_minutes, new_session_isolation | — |
semantic-distinct-patient-reviewlint-clean-semantic-reviewer-correction | PASS | yes | — / — advisory | 33/33 passedsession_created, initial_turn_recorded, initial_evidence_available, base_version_saved, base_version_available, base_validation_recorded, base_execution_succeeded, base_postgres_crosscheck, base_gold_execution_match, followup_http_created, followup_terminal_status, base_classification, manual_version_classification, followup_profile, refresh_restored, timeline_current_turn, followup_evidence_available, invocation_duration_sum, invocation_digests, invocation_timestamp_reconciliation, writer_model, reviewer_model, effective_temperature_and_dry, revision_context_exclusions, exact_selected_output, semantic_reviewer_correction, prior_results_stale_after_successor, successor_validation_recorded, successor_execution_succeeded, successor_postgres_crosscheck, successor_gold_execution_match, successor_visible_under_three_minutes, new_session_isolation | — |
semantic-distinct-patient-reviewlint-clean-semantic-reviewer-correction | PASS | yes | — / — advisory | 33/33 passedsession_created, initial_turn_recorded, initial_evidence_available, base_version_saved, base_version_available, base_validation_recorded, base_execution_succeeded, base_postgres_crosscheck, base_gold_execution_match, followup_http_created, followup_terminal_status, base_classification, manual_version_classification, followup_profile, refresh_restored, timeline_current_turn, followup_evidence_available, invocation_duration_sum, invocation_digests, invocation_timestamp_reconciliation, writer_model, reviewer_model, effective_temperature_and_dry, revision_context_exclusions, exact_selected_output, semantic_reviewer_correction, prior_results_stale_after_successor, successor_validation_recorded, successor_execution_succeeded, successor_postgres_crosscheck, successor_gold_execution_match, successor_visible_under_three_minutes, new_session_isolation | — |
bounded-hub-tool-failurehub-tool-failure | PASS | yes | — / — advisory | — | — |
Turn / version / execution timeline
| scenario | session | versions | executions | gen wall |
|---|---|---|---|---|
narrowing-unchanged-base | 0f85f243-2cef-45b8-be79-0b4e29d965d5 | c91d7cd4-8149-4916-ad8a-dbc2d8ecfe13 → 1af173e0-71b2-491a-93d1-595d3197a51e | 3628f1a5-a620-4cf7-93c8-2f47438508b2 / e99771e1-2792-429a-bce5-fe38a1d3c81e | 100597 ms |
narrowing-unchanged-base | 02e9a456-7f83-4f3f-9e46-a97cdebaee80 | 3dabd1a6-b506-4ab4-b1e6-fc294f6455c0 → 6a31285a-4d70-4f8c-94e4-2f2831130160 | 99bcb633-023d-4636-b4d8-5b493d4462ee / e5479d6c-9f00-4316-bb71-ff4bfa699f5c | 77622 ms |
narrowing-unchanged-base | 630e4e04-72ee-49a7-85b0-d6235e1cd3f2 | cdd499fd-44ea-466f-ad65-d9f1f32b2ef3 → 7fe704e6-3fa7-4b1e-b385-18e983d4676c | a8dd95fa-33d4-45d9-9582-0b01c1caf50d / a0cb2374-46c2-4032-b42d-4185ef8d2b4b | 77626 ms |
aggregation-dirty-base-profile-switch | 61e46f4e-bcc4-4c91-928a-347a15bc1bdd | 2ea8f02c-ecb1-445a-8de9-5716a3a7bf1e → 587edc86-cbfb-44c4-896d-7d236b2ac009 | 4d70a419-fdfc-467d-af8f-e0096a4ba6ef / d93260ce-b0b4-4976-ab16-5dbc4253ddc2 | 74096 ms |
aggregation-dirty-base-profile-switch | 98803511-d92b-436a-949d-9509ec56846b | 2cf367a4-1c64-471f-b50a-f49815e37f66 → ecc184ea-7fc2-49ec-b217-71a2bf2c2e4e | e83f2036-07b1-4430-a38b-b404010b6f37 / 7fe9134b-a85f-4db1-acf3-904343c05044 | 53367 ms |
aggregation-dirty-base-profile-switch | 0008064f-333a-4980-a581-79b4657db937 | 27c6b345-95f9-4068-a9ad-e3a4a2888872 → 102f77ea-a034-4008-9ec9-1a493ca131d9 | 9b388439-5682-4902-a8b6-922d0ee93897 / 4c7e4262-a703-4ee5-a707-1d046e04b53d | 55390 ms |
unresolved-parameter-correction | 9481e959-ff55-490a-bb3c-d4e4ea452c78 | d8b8c5b0-88b6-46be-a054-62d68e7d71c1 → aa90a8bf-36f6-417e-a5cf-98f511e6a6d4 | acc9148d-90fd-4e1b-87b8-be2faab5bc64 / 81b1ed0d-dfb5-4b9a-95fc-393b441d7f3b | 83191 ms |
unresolved-parameter-correction | 626eea53-d4ea-4c15-9adb-ab3e7f95b62c | ae586a6b-18ec-433d-a136-44987842d19c → 7b174f78-6b36-49fc-ba5b-c3922f3efde7 | cbbf632e-1fb2-4b09-9ee6-811a156000b0 / f1e49218-b51d-4a82-b27f-de9f41325c0a | 60824 ms |
unresolved-parameter-correction | 1a15d6f6-f262-4c2c-b451-c09a2fcb85db | a8117929-0843-481e-9906-14a818e44fa7 → 665fd3cc-61b6-42c4-aa9d-925a05d26dfe | d5d211e6-6893-4eff-a42c-6b4229536487 / b7ae8fa4-dd7f-4275-9eee-2831a75ed73e | 58952 ms |
semantic-distinct-patient-review | e4c8546c-2408-4f4d-85c1-b035681d6782 | 799cdab9-565e-4a06-bf77-45cdf9aad755 → a72ec658-c8e4-4ea8-8260-5f3b54734c88 | f3b9f013-57fa-47a6-a70c-35a7ee48d806 / 4e504323-f2e0-4c63-bc5f-c3626ae79e4f | 37260 ms |
semantic-distinct-patient-review | b7d5deb7-b4e0-43bd-8fe4-666bb341056a | 27d194a7-2562-41d5-9854-14a15283b944 → 83d971c4-1204-40eb-a983-869deea66291 | d149abde-376a-4d55-944c-b8152b35ae93 / 837b320e-857a-4623-ad0d-02c394701e16 | 31891 ms |
semantic-distinct-patient-review | 277a6f9f-2414-4cec-b015-03d7ad0b2239 | 648077bf-4151-4a6d-9881-bafc5f0dac93 → ac26959d-65ab-425d-8833-0f106fde402f | 20a670b8-108a-4952-a883-4f43f21492a7 / 6498e78f-70dc-454a-9c10-0b579366c332 | 31127 ms |
bounded-hub-tool-failure | → | / | ms |
SQL diffs
No multi-version SQL diffs found.
Finalized judge (advisory)
Judge not available for this run (development / no-judge path).