Clinical AI validation runs

A different validation family from the chart-QA runs above: Catalyst turns a plain-language clinical question into governed SQL (a Gemma 4 12B writer drafted, a Qwen 2.5 14B reviewer checked, a deterministic policy enforcing read-only execution), then a follow-up instruction refines the exact current query. Four scenario families × 3 repetitions against the live stack and a real PostgreSQL: narrowing an unchanged base, aggregating from a dirty editor with a mid-session profile switch, correcting an unresolved parameter, and a semantic distinct-patient review. Every executed result is re-checked two ways — against an independently-run gold query (byte-level execution match) and by an independent read-only PostgreSQL cross-check.

TakeawayAll 12 scenario repetitions passed — 384/384 assertions, including every gold execution-match: the SQL the models produced returned exactly the rows an independently-authored gold query returns, on every repetition. This is an acceptance gate on one synthetic 96-patient laboratory cohort, not a benchmark — it demonstrates the governed notebook loop (draft → review → validate → execute → refine) is reliable end-to-end on real infrastructure, not that the models never err (the advisory validator and deterministic policy exist precisely because they do).

A fast 12-scenario dev sweep, holding two writers (Gemma-12B dense, Gemma-26B-A4B MoE ~4B active) against {bare, +coverage prompt, +rewrite-validator, +both}. Judged AFTER fixing a scoring bug where the judge couldn't see table answers (which had been deflating completeness). N=12/arm — directional, not definitive.

TakeawayTwo findings. (1) A SCORING bug was capping the benchmark: the judge scored only the prose answer and never saw the structured table answers the prompts ask for, deflating completeness ~8pts; fixed. (2) The optimal config is WRITER-DEPENDENT: the dense 12B is best with coverage+rewrite-validator (83.5, 0 harm) vs bare 80.6, but the cheap A4B MoE is best BARE (83.7, 0 harm, highest completeness) and EVERY lever hurts it. Bare A4B edges the best scaffolded 12B — the cheap MoE alone is competitive, the on-device headline. The orchestrator hurts both (12B +rewrite: 72 w/ orch vs 78 w/o). Caveat: N=12, single judge, hard-weighted dev subset; needs full-run confirmation.

Does a small-orchestrator + Qwen-2.5-14B-validator team beat the bare single writer on the two strong writers (Gemma-12B, Gemma-26B-A4B)? Plus a cite-or-abstain prompt lever. 297 cells, judged.

TakeawayThe validator's value scales inversely with writer strength: it rescues the weaker A4B (2 harm to 0, +~3pts) but doesn't help the already-safe 12B, best as a bare single (~81, 0 harm). cite-or-abstain lowers the 12B via over-caution.

Six single Google Gemma models reading the same patient charts, varied along two axes: precision (8-bit vs 4-bit) and size/architecture. Two are run at BOTH precisions — a 4-billion on-device model (E4B) and a 12B dense model — to isolate quantization; alongside them a 31B dense model and a 26B sparse Mixture-of-Experts (which fires only ~4B parameters per token), both at 4-bit. Each makes a fast Answer plus a separate In-Depth, both graded. 62 clinical questions across three patients (two adults with HIV/TB, a pediatric infant), judged against the full chart by an automated clinical rubric.

TakeawayCutting a model to 4-bit costs the 12B about 7 benchmark points AND its safety — at 8-bit it scored 84 with zero unsafe answers; at 4-bit, 76 with two. Recovering that loss by going BIGGER barely works: a 31B at 4-bit buys back only ~1 point, while simply restoring 8-bit on the 12B buys back all 7 — for this model class, spend memory on precision, not parameters. The efficiency standout was the sparse Mixture-of-Experts (26B-A4B): second-best quality (80) at HALF the latency of any other model, with just one unsafe answer. And 4-bit is most dangerous where there's least capacity to spare: the small 4B model at 4-bit was the floor (68), with the most unsafe answers (3) and the most invented facts for things not in the chart (6). Directional results from a single automated judge on a small sample, not a formal benchmark.

Eleven single-model setups, each making a fast Answer plus a separate In-Depth (both graded): five model classes at matched Q8 AND Q4 — Google Gemma 4 12B, Alibaba Qwen3-14B, Qwen2.5-14B, IBM Granite 3.3 8B, Mistral NeMo 12B — plus a reasoning model (Qwen3.6-35B). 32 clinical questions across three patients, judged against the full chart on two independent axes.

TakeawayGemma 4 12B tops BOTH axes (Answer 75.8, In-Depth 65.7) — and its lead is NOT a precision artifact: Gemma at Q4 matched (even edged) Gemma at Q8, while quantization moved the other models inconsistently and mostly within noise, so the earlier Gemma win was the model, not its higher precision. Mistral NeMo was the weakest clinical reader (Q8 worst, 55.8, 7 unsafe answers). The separate In-Depth axis earned its keep: it exposed safety failures the Answer hides — Granite's Q4 In-Depth produced 12 unsafe elaborations, Mistral-Q8's 8 — that an answer-only view would miss entirely. The reasoning model (Qwen3.6) was safe but not top. Directional: one judge, three patients, 32 questions.

The new two-call architecture: every setup returns a fast Answer and a separate, slower In-Depth, each independently timed and graded — for single models AND the multi-agent team, on equal footing, via the same shared in-depth prompt. Three patients, 32 clinical questions, five setups (Google Gemma 12B, Alibaba Qwen 14B, Google MedGemma 27B, Liquid LFM2 24B singles, plus a parity AI team).

TakeawayHolding the in-depth prompt fixed, the team's elaboration does NOT beat a single model's. The single Gemma 12B topped BOTH axes — best Answer (73) AND best In-Depth (added-value 6.9, zero unsafe elaborations) — while the parity team's in-depth was no better (6.2), and the fast on-device Liquid 24B's in-depth was actively dangerous (10 unsafe elaborations). Separate latencies confirm the design: a quick answer (Gemma ~20s, Liquid ~2s) followed by a slower in-depth (~12-24s). One strong general model gives the best answer AND the best background; the team scaffolding adds neither. Directional results from a single automated judge on a small sample, not a formal benchmark.

Three patients (two adults with HIV/TB and a pediatric infant), 32 clinical questions, seven setups: four single-model baselines across classes — Google Gemma 12B, Alibaba Qwen 14B, Google MedGemma 27B (a medical-specialist model), and the fast on-device Liquid LFM2 24B — plus a single model with an added elaboration and two multi-agent AI-team configurations. Every answer was graded against the full patient chart by an automated clinical rubric.

TakeawayA strong GENERAL single model wins: Gemma 12B (72) tied the full multi-agent team, and Qwen 14B (71) was just behind AND the safest single arm (1 unsafe answer vs Gemma's 3). The surprise: the medical-specialist MedGemma 27B scored BELOW the general models (65) with more unsafe answers (5) — domain fine-tuning did not help here — and the fast on-device Liquid 24B was the weakest and least safe (49, 8 unsafe). Directional results from a single automated judge on a small sample, not a formal benchmark.

Three patients — two adults with HIV/TB and a pediatric perinatal-HIV infant — 62 clinical questions, six AI setups. A 2×2 ablation swaps the AI team's coordinator and answer-writer between a Google (Gemma) and a fast on-device (Liquid LFM2) model — every team cross-checked by a strong non-Liquid validator — alongside two single-model anchors (Gemma 12B and Liquid 24B). Every answer was graded against the full patient chart by an automated clinical rubric.

TakeawayThe model that WRITES the answer decides clinical quality. Setups with a Qwen writer scored ~70; those with a Liquid writer ~43–51 — a ~25-point gap that tracks the writer alone, whichever model coordinates. A single strong general model (Gemma 12B, 71.4) matched or beat the full multi-agent team, and a strong non-Liquid validator did not rescue a Liquid writer (13–17 unsafe answers with it switched on). Pick a strong writer model before adding team scaffolding. Directional results from a single automated judge, not a formal benchmark.

A follow-up to the Liquid-team run: same two patients, 16 clinical questions, four AI setups — our proper multi-agent team, a single strong model (Gemma 12B), and the fast on-device 'Liquid' team in two forms: as-is, and with a strong non-Liquid Gemma-12B 'validator' that independently re-checks each answer against the chart before it's returned. Every answer was graded against the full chart by an automated clinical rubric.

TakeawayAdding a strong non-Liquid validator to the Liquid team helped — it eliminated the made-up 'I don't know' answers and removed one unsafe response, raising the headline score from 43 to 52 — but it did not make the team safe: the validated Liquid team still produced four unsafe answers (including a harmful regimen error the validator passed) and stayed far below both a proper model team (81) and a single strong model (74). A validator catches over-confident gaps, not the subtler clinical fabrications. These are directional results from a single automated judge on a small sample, not a formal benchmark.

Two patients, 8 clinical questions, five AI setups: a strong single model (Gemma 12B), our proper multi-agent AI team, a stripped-down 'parity' team, and two teams built around fast on-device 'Liquid' (LFM2) models — one all-Liquid, one that keeps a medical-specialist model. Every answer was graded against the full patient chart by an automated clinical rubric.

TakeawayPutting fast on-device 'Liquid' models inside a careful multi-agent team improved their sourcing — they now cite real chart records instead of nothing — but did not make them safe: both Liquid teams still fabricated medications, reversed a weight trend, and hid an AIDS-defining CD4 count, landing far below both the proper model team and a single strong model. Team scaffolding helped grounding, not clinical accuracy. These are directional results from a single automated judge on a small sample, not a formal benchmark.

Two patients, 8 clinical questions, five AI setups: Google's Gemma 12B and 4B single models, two brand-new on-device 'Liquid' (LFM2) single models (24B and 1.2B), and our multi-agent AI team. Every answer was graded against the full patient chart by an automated clinical rubric.

TakeawayThe Gemma 12B single model was the most accurate and the safest — no unsafe answers, and it cited a chart record for every claim. The two new Liquid models were the fastest but the least safe: they often answered confidently without grounding their claims in the chart (almost never citing a record) and produced several unsafe answers — including a 'no allergies' reply for a patient whose chart documents a penicillin allergy. The AI team cited its sources and was far safer than the Liquid models, though it didn't beat the strong single model here. These are directional results from a single automated judge on a small sample, not a formal benchmark.

Two patients, 16 questions that hinge on timing: the most-recent lab value, weight trends over months, what's changed recently, and whether a medication regimen is current. We compared plain single models (Gemma 4B → 12B → 26B) against our multi-agent AI team at four configurations.

TakeawayRe-scored with our updated temporal rubric. The Gemma 12B single model led on the overall benchmark (80/100) and accuracy (8.7/10); the AI team's Advanced tier gave the most complete answers (7.8/10). The stricter date/trend scoring surfaced unsafe answers the first pass missed: a few setups fabricated dates or trends — the Basic AI team 3, and the Gemma 4B and 26B single models one each — while the Gemma 12B single and the Standard/Advanced/matched teams had none. Bigger isn't automatically better, and getting the timing right is where several setups slip.

21 high-priority questions about one complex HIV/TB patient (Aloice), comparing two single models (Gemma 2B and 26B) with three AI-team configurations.

TakeawayThe 26B single model led on accuracy (9.1/10); the AI team's Advanced tier led on completeness (8.4/10). No unsafe answers in this run.

An earlier, broader sweep of 22 scenarios on the same patient, comparing the Gemma 12B single model with several AI-team configurations. This run predates our safety cross-checker and the temporal-reasoning fixes.

TakeawayThe Advanced AI team led on both accuracy and completeness — but the smaller team tiers produced several unsafe answers (Basic tier: 5). That's exactly what motivated the cross-checking 'validator' added in the later runs.