Clinical AI validation runs

TakeawayMed Agent Hub scored 89.8/100 (9.32/10 accuracy, 9.03/10 completeness, 9.12/10 clarity) versus 75.4/100 for built-in ChartSearchAI — a +14.4-point lead with zero unsafe answers, zero wrong dates (vs. 3 on built-in), zero unnecessary refusals (vs. 4 on built-in), and 100% verified chart citations (94/94). On the 6-month orders question that motivated this test, Med Agent Hub jumped from 53.0 to 94.0/100 by capturing all eight qualifying prescriptions across both January and December visits while excluding December 14 labs and chest X-rays that fell six days outside the cutoff. Trade-off and remaining gap: Med Agent Hub averaged 17.0s per answer vs. 4.4s on built-in ChartSearchAI, and its one weak spot (53.0/100) was a 5-turn counseling follow-up about stopping an older HIV medication.

We tested whether tightening Med Agent Hub's date-window filters fixes a stubborn failure mode: mixing older lab or imaging orders into a strict 6-month lookback, or dropping earlier drug refills inside that same window. Both setups — OpenMRS's built-in ChartSearchAI and Med Agent Hub's single-call answer path — ran the exact same Google Gemma 4 12B model on a shared 24 GB cloud GPU across three HIV patients and 17 clinical scenarios (35 questions per setup, 70 answers total), anchored to June 20, 2026 and graded blind by two independent AI judges.

Judged in 2 passes by claude-opus-5-5, reasoning high, Claude Code CLI 2.1.316 (Claude Code), pass 1 of 1, brief 6553834; gpt-6-astra, reasoning high, Codex CLI 0.165.0, pass 1 of 1, brief 6553834; the headline is the median of each 0-10 axis and the majority of each label across the passes, a harm tie counting as harmful.

TakeawayMed Agent Hub beat built-in ChartSearchAI by +8.7 points on clinical quality (87.2 vs. 78.5/100, zero unsafe answers, zero wrong dates, 82/82 verified citations vs. 1 wrong date and 4 unnecessary refusals on built-in), led by major gains on upcoming appointments (100 vs. 77/100), 5-turn medication adherence dialogs (+14.8 points), and pediatric growth tracking. Under multi-user load, Med Agent Hub scaled throughput 3.5x from 1 to 4 concurrent clinicians (3.7 to 13.0 answers/min, keeping median follow-up first-word wait at 3.2s with zero errors), whereas built-in ChartSearchAI saturated at 9.2 answers/min with a 17.8s median wait for the first word.

In a busy clinic, several providers may query patient charts simultaneously on one modest GPU server. We compared Med Agent Hub's single-call answer path against built-in ChartSearchAI on a shared 24 GB cloud GPU running Google Gemma 4 12B (configured for up to four concurrent requests), evaluating both clinical answer quality across 17 HIV scenarios (35 questions per setup) and live response speed with 1, 2, and 4 simultaneous clinicians.

Judged in 2 passes by claude-opus-5-5, reasoning high, Claude Code CLI 2.1.316 (Claude Code), pass 1 of 1, brief 6553834; gpt-6-astra, reasoning high, Codex CLI 0.165.0, pass 1 of 1, brief 6553834; the headline is the median of each 0-10 axis and the majority of each label across the passes, a harm tie counting as harmful.

TakeawayReplacing the second AI pass with deterministic checks cut Med Agent Hub's median answer wait in half (21.7s down to 10.8s, with post-answer checking dropping from 4.9s to under 0.5s) while outscoring built-in ChartSearchAI by +16.1 points (85.6 vs. 69.5/100). Med Agent Hub recorded zero unsafe answers and zero wrong dates, compared with 3 unsafe answers and 2 wrong dates on built-in ChartSearchAI.

Earlier Med Agent Hub setups ran a second AI model pass after every answer to double-check citations, doubling wait times on local hardware. On an Apple Silicon laptop running Google Gemma 4 12B, we tested replacing that second AI pass with fast rule-based date and citation checks — comparing Med Agent Hub's true single-call path head-to-head against built-in ChartSearchAI across three HIV patients and 35 clinical questions per setup.

Judged in 2 passes by claude-opus-5-5, reasoning high, Claude Code CLI 2.1.316 (Claude Code), pass 1 of 1, brief 6553834; gpt-6-astra, reasoning high, Codex CLI 0.165.0, pass 1 of 1, brief 6553834; the headline is the median of each 0-10 axis and the majority of each label across the passes, a harm tie counting as harmful.

TakeawayGoogle Gemma 4 is the clear winner for a single-model clinical pathway, sweeping the top 7 spots out of all 27 setups (67.2–82.1/100) ahead of the best Qwen 3 (65.4/100) and Llama 3.1 (55.3/100) setups — and across every model 4B and above in all three families, routing through Med Agent Hub monotonically raised scores and cut unsafe answers by 33%–71%. Why Gemma 4 dominates: across all 216 answers from Gemma 4 12B and 26B (all 6 setups), neither model ever produced a fabricated or unsupported [N] citation (0/216 vs. 2–3/36 per setup on Qwen 3 14B, while Llama 3.1 8B frequently dropped inline [N] citations with only 2–5/36 answers fully supported). Gemma 4 26B + Med Agent Hub led overall at 82.1/100 (8.62 accuracy, 8.39 completeness, 8.76 relevance, 3/36 unsafe vs. 72.9/100 for Direct + KB and 71.3/100 with 6/36 unsafe for Direct AI — a 50% reduction in unsafe answers). Gemma 4 12B + Med Agent Hub (80.5/100, 8.53 accuracy, 8.68 relevance) emerged as the practical single-model sweet spot: it came within 1.6 points of 26B while recording the lowest unsafe-answer rate (2/36 vs. 7/36 on Direct 12B — a 71% reduction) and lowest confabulation count (1/36) in the entire sweep, beating raw Gemma 4 26B (71.3/100) by +9.2 points. For edge or laptop deployments, compact Gemma 4 E4B (4B effective parameters) + Med Agent Hub scored 70.4/100 (8.01 accuracy, 4/36 unsafe vs. 8/36 on Direct E4B), outperforming both raw Gemma 4 12B (67.2/100) and Qwen 3 14B + Med Agent Hub (65.4/100) at roughly one-third the active parameter footprint. The same pathway gains held across non-Gemma families: Qwen 3 14B climbed from 53.4/100 (13/36 unsafe) on Direct AI to 61.8/100 (6/36 unsafe) with Direct + KB and 65.4/100 (5/36 unsafe, −62% unsafe) with Med Agent Hub; Qwen 3.5 9B climbed from 49.4 to 57.1/100; Qwen 3 4B climbed from 34.7 (12/36 unsafe) to 43.5/100 (8/36 unsafe); and Llama 3.1 8B climbed from 50.6 (14/36 unsafe) to 55.3/100 (8/36 unsafe, −43% unsafe). Only the ultra-small 3.2B tier (Llama 3.2 3B, 33.8–44.2/100) struggled to synthesize dense multi-clinic FHIR + rule context simultaneously, establishing ~4B effective parameters as the capability floor.

To separate the impact of model size, clinical rule checks, and full Med Agent Hub orchestration, we evaluated nine open-weight models across three model families — Google Gemma 4 (E4B, 12B, 26B), Alibaba Qwen 3 (4B, 9B, 14B), and Meta Llama 3.x / Hermes 3 (3.2B, 8B, Hermes 3 8B) — under three pathways at a standard 8k chart budget: (1) Direct AI on the condensed chart alone, (2) Direct AI + Stage-2 4-Layer Clinical KB alerts, and (3) the full Med Agent Hub pipeline (condensed chart + clinical conflict rules + patient entity-relationship subgraph + Tier-0 safety net recovery). Every setup was tested across 36 clinical scenarios spanning 10 complex multi-clinic and ICU patients (20 clinical accuracy & conflict reconciliation questions and 16 conversational privacy, caregiver-boundary, and coercion stress tests — 972 total answers) and graded in a blinded review by two independent AI judges (Claude Opus 4.6 and GPT-5.4) anchored to a clinician-verified ground-truth rubric.

Judged in 2 passes by claude-opus-5-5, reasoning high, Claude Code CLI 2.1.316 (Claude Code), pass 1 of 1, brief 6553834; gpt-6-astra, reasoning high, Codex CLI 0.165.0, pass 1 of 1, brief 6553834; the headline is the median of each 0-10 axis and the majority of each label across the passes, a harm tie counting as harmful.

TakeawayGiving the local server two active memory slots raised Med Agent Hub's chart-memory reuse from 0% to 84%, cutting median wait for the first word by 3.3x (34.0s down to 10.3s) and full-answer time by 3.0x (35.9s down to 12.1s). At the same time, Med Agent Hub maintained its clinical quality and safety lead over built-in ChartSearchAI (79.8 vs. 75.9/100, with zero unsafe answers and zero invented facts vs. 1 of each on built-in).

When a local AI server only keeps one conversation in active memory at a time, any background helper task evicts the patient chart and forces the AI to re-read thousands of words on the next question. On a 64 GB Apple Silicon MacBook running Gemma 4 12B, we configured two concurrent memory slots and compared Med Agent Hub against built-in ChartSearchAI across 17 HIV chart scenarios (35 questions per setup).

Judged in 2 passes by claude-opus-5-5, reasoning high, Claude Code CLI 2.1.316 (Claude Code), pass 1 of 1, brief 6553834; gpt-6-astra, reasoning high, Codex CLI 0.165.0, pass 1 of 1, brief 6553834; the headline is the median of each 0-10 axis and the majority of each label across the passes, a harm tie counting as harmful.

TakeawayBoth setups answered all 35 questions with zero system errors and smooth live word-by-word streaming once generation started, but the single memory slot exposed a major caching bottleneck. Built-in ChartSearchAI reused 99.9% of the patient chart in memory across follow-up questions (1.3s median wait for the first word), whereas Med Agent Hub's short post-answer citation-matching call overwrote that single memory slot — forcing a 34.0s chart reload on the next question and motivating our two-slot and single-call fixes.

To diagnose why local laptop deployments sometimes pause for 30 seconds before streaming the first word, we instrumented live token-by-token streaming and memory-cache tracking on a 64 GB MacBook running Gemma 4 12B with a single active memory slot, comparing built-in ChartSearchAI and Med Agent Hub across 35 HIV chart questions each.

TakeawayGiving the AI enough memory completely eliminated the 32k chart collapse, and scoping Med Agent Hub's clinical rules to the question being asked put the KB-enriched setup on top at every chart size with zero unsafe answers. At the standard 8k chart, Med Agent Hub + clinical rules led overall at 92.5/100 (9.42 accuracy, 9.33 completeness, 9.08 relevance, 0 unsafe answers), ahead of Hub chart-only (89.2/100, 0 unsafe) and calling the AI model directly (86.5/100, 1 unsafe). At the large 32k chart, Med Agent Hub + clinical rules also led at 91.3/100 (0 unsafe), ahead of Hub chart-only (89.8/100, 0 unsafe) and Direct AI (85.5/100, 1 unsafe). On the compact 1k summary — where older prescriptions and labs were squeezed out of the static chart — Med Agent Hub's clinical rules and Tier-0 overflow recovery rescued the missing safety records and scored 68.2/100 (0 unsafe), beating Hub chart-only (49.2/100, 1 unsafe) by +19.0 points and Direct AI (47.0/100, 2 unsafe) by +21.2 points.

In our earlier cloud GPU test, large patient charts failed because the shared server silently cut off anything past 16,000 tokens of memory. Here we ran the same Google Gemma 4 12B model on a 64 GB MacBook with a 96,000-token memory window — large enough to hold the entire chart at once — across two complex multi-clinic patients and six clinical questions (medication conflicts, hospital history, and knowing when to say 'not in chart'). At each chart size (compact 1k, standard 8k, and large 32k), we compared three setups — calling the AI model directly on the chart, routing through Med Agent Hub with the chart alone, and routing through Med Agent Hub with its automated clinical rule checks — and graded all 54 answers with two independent blinded AI judges (Claude Opus 4.6 and GPT-5.4).

Judged in 2 passes by claude-opus-5-5, reasoning high, Claude Code CLI 2.1.316 (Claude Code), pass 1 of 1, brief 6553834; gpt-6-astra, reasoning high, Codex CLI 0.165.0, pass 1 of 1, brief 6553834; the headline is the median of each 0-10 axis and the majority of each label across the passes, a harm tie counting as harmful.

TakeawayFeeding a bigger chart to an AI backfires if the server's active memory window is smaller than the prompt. Scores peaked at the standard 8k summary, which fit comfortably inside the server's 16k memory limit: Gemma 4 26B with clinical rules led at 88.0/100 (8.75 accuracy, 9.12 completeness, zero unsafe answers), followed by Gemma 4 12B with clinical rules (80.5/100) and chart-only (80.2/100), while Qwen 3 14B scored 67.0/100. On the tight 1k summary, Med Agent Hub's clinical rule safety net beat chart-only by +19 points (43.8 vs. 24.8/100). When we sent 32k to 128k of chart text to that same 16k server, the server silently cut off the beginning of the prompt — erasing active allergies, current medications, and recent hospital stays — so chart-only scores collapsed to 0.0–2.8/100 (with 4/4 unsafe answers). Because Med Agent Hub attaches its pre-computed clinical conflict alerts at the very end of the prompt, those alerts survived the cutoff and rescued the medication conflict portion of the answer (27.0–32.8/100), though top-of-chart allergy checks remained cut off.

How much of a patient's medical record should we feed an AI at once? We tested five chart summary sizes — from a tiny 1k-token emergency snapshot up to a massive 128k-token multi-decade history — across four patients with records from multiple hospitals and clinics (48 runs total). All models ran on a standard shared 24 GB cloud GPU with a fixed 16,000-token active memory limit, comparing answers with and without Med Agent Hub's clinical rule checks, and were graded in a blinded review by two independent AI judges (Claude Opus 4.6 and GPT-5.4).

Judged in 2 passes by claude-opus-5-5, reasoning high, Claude Code CLI 2.1.316 (Claude Code), pass 1 of 1, brief 6553834; gpt-6-astra, reasoning high, Codex CLI 0.165.0, pass 1 of 1, brief 6553834; the headline is the median of each 0-10 axis and the majority of each label across the passes, a harm tie counting as harmful.

TakeawayAttaching automated clinical rule checks improved every general AI model by 6 to 21 points. Google's Gemma 4 12B with clinical rules led overall (91.0/100, zero unsafe answers), followed by Alibaba's Qwen 3 14B (83.1/100, zero unsafe answers). Reading the chart alone, even strong models frequently missed duplicate brand-name vs. generic prescriptions (such as taking Plavix and clopidogrel together) or multi-step vaccine timing rules.

Seven open-weight AI models (Google Gemma 4 12B, Alibaba Qwen 3 14B, Mistral NeMo 12B, Meta Llama 3.1 8B, IBM Granite 3.3 8B, Liquid LFM2 8B, and medical-tuned MedGemma 4B) answered eight clinical questions across five patients. Each model was tested twice — once reading the condensed patient chart on its own, and once with our automated clinical rule checks attached — and graded by two independent AI judges.

Judged in 2 passes by claude-opus-5-5, reasoning high, Claude Code CLI 2.1.316 (Claude Code), pass 1 of 1, brief 6553834; gemini-4-argon-high, reasoning high, Gemini CLI 1.4.12, pass 1 of 1, brief 6553834; the headline is the median of each 0-10 axis and the majority of each label across the passes, a harm tie counting as harmful.

TakeawayA single AI model reading the chart on its own passed only 17 of 32 questions (80.0/100) because it overlooked brand-vs-generic drug duplicates and age-specific vaccine rules. Attaching automated clinical rule alerts raised the pass rate to 24/32 (80.7/100), adding a second reviewer model reached 29/32 (83.1/100), and the four-agent specialist team passed all 32 questions (88.6/100) while writing in clearer, more patient-friendly language.

When a patient sees doctors at several different clinics, their combined record often contains duplicate prescriptions under different brand names, conflicting kidney-disease stages, or outdated medications. We tested four AI setups on 32 clinical questions: a single model reading the condensed chart alone, a single model with automated clinical rules attached, a two-model writer-and-reviewer pair, and a four-agent specialist team.

Judged by our automated clinical checklist and source-citation verifier.

TakeawayA static 2k-token summary keeps 100% of current allergies, active medications, and two-year lab trends, but drops older surgical and hospital details — passing only 18 of 32 questions (70.4/100). Giving that same 2k summary an on-demand search tool to pull older notes when asked recovered every missing historical detail (32/32 passed, 88.6/100), matching the 16k summary at a fraction of the size.

Patient health exports can span thousands of pages, far too large for an AI to read reliably all at once. We compressed charts down to four summary sizes (16k, 8k, 4k, and a tiny 2k-token wallet summary) and tested 32 questions ranging from current medications to hospital surgeries years ago. We then tested whether giving the tiny 2k summary an on-demand search tool to look up older notes could match the full 16k summary.

Judged by our automated clinical checklist and source-citation verifier.

TakeawayThe condenser and rule engine passed all 32 test cases at every summary size (8k, 4k, and 2k tokens), shrinking a 53-year chart by over 99.6% while keeping 100% of active medications and allergies, catching all 15 cross-clinic conflicts with clickable source links, and blocking 100% of sensitive records in external-share mode.

Before putting an AI model in the loop, we tested our chart condenser and clinical rule engine on 32 test cases across 53 years of multi-clinic records. The test checked whether shrinking an 18 MB health export into 2k–8k token summaries preserves all active conditions and allergies, catches 15 planted cross-clinic prescription and staging conflicts, and blocks sensitive mental-health or substance-use records when sharing externally.

Judged by our automated clinical checklist and source-citation verifier.

TakeawayThe improvements generalized cleanly to unseen patients: Med Agent Hub outscored built-in ChartSearchAI 81.5 to 75.3/100 across all 120 questions, cut unsafe answers by two-thirds (3 vs. 9), reduced invented facts sixfold (2 vs. 12), and eliminated all six time-window overreach errors seen on built-in ChartSearchAI.

To make sure our clinical date-window, lab-trend, and prescription rules were not overfitted to our three development charts, we drew 15 fresh, previously unseen HIV patient records and asked 8 clinical questions per patient (120 answers per setup, 240 total). Both OpenMRS's built-in ChartSearchAI and Med Agent Hub's rule-checked single-model path ran Google Gemma 4 12B and were graded blind by two independent AI judges.

Judged in 2 passes by claude-opus-5-5, reasoning high, Claude Code CLI 2.1.312 (Claude Code), pass 1 of 1, brief 049e8da; gpt-6-astra, reasoning high, Codex CLI 0.165.0, pass 1 of 1, brief 049e8da; the headline is the median of each 0-10 axis and the majority of each label across the passes, a harm tie counting as harmful.

TakeawayRefining the deterministic date and chart checks eliminated false-positive warning banners and achieved zero unsafe answers and zero wrong dates across all three Med Agent Hub setups (79.0–81.1/100), compared with 70.6/100, 1 unsafe answer, and 3 wrong dates on built-in ChartSearchAI. Adding a second 9B AI verifier nudged the headline score by less than 1 point (81.1 vs. 80.4/100) at the cost of 27 extra seconds per question — showing fast deterministic checks do the real safety work.

Across 17 HIV chart scenarios (35 questions per setup), we compared four ways of running Google Gemma 4 12B: OpenMRS's built-in ChartSearchAI, Med Agent Hub's fast answer-only path, Med Agent Hub with deterministic chart and date checks, and a two-model setup that adds a separate Qwen 9B verifier model after every answer.

Judged in 2 passes by claude-opus-5-5, reasoning high, Claude Code CLI 2.1.312 (Claude Code), pass 1 of 1, brief 049e8da; gpt-6-astra, reasoning high, Codex CLI 0.165.0, pass 1 of 1, brief 049e8da; the headline is the median of each 0-10 axis and the majority of each label across the passes, a harm tie counting as harmful.

TakeawayBoth Med Agent Hub setups (82.0 checked, 81.6 answer-only) outscored built-in ChartSearchAI (55.2) by more than 26 points on the exact same model. Built-in ChartSearchAI struggled with date calculations (8 date errors vs. 0 on the Hub) because it does not pass today's reference date or run deterministic date and guideline checks.

A repeat of the 17-question HIV chart comparison across three patients, running all three setups through the same Gemma 4 12B model: ChartSearchAI's built-in answer path on its default settings, Med Agent Hub's plain answer path, and Med Agent Hub's checked profile (which reviews each answer and flags unsupported claims).

Judged by claude-opus-5-5, reasoning high, Claude Code CLI, pass 1 of 1, brief 8b3b7a1. Also scored by gpt-6-astra, reasoning high, Codex CLI 0.160.1, pass 1 of 1, brief 8b3b7a1.

TakeawayOn the same engine, the Hub's plain answer path scored 78.9 and bundled ChartSearchAI 59.8. The gap comes from things the bundled path does not have: it does not tell the model today's date, it has no guideline knowledge base or date checks, and its repetition penalty garbles long lists when it reads the whole chart. The checked profile (78.4) was the most complete but marked most In-Depth sections red, mostly guideline statements that cite a chart record which does not back them.

Seventeen questions about three HIV patients, some with follow-ups (35 answers per setup), asked through three setups that all send their model calls to the same Gemma 4 12B engine: ChartSearchAI's bundled answer path on its default settings, the Hub's plain answer path, and the Hub's checked profile, which reviews each answer and labels what its checks found. One automated judge graded each question.

Judged by claude (model not pinned), Claude Code CLI, pass 1 of 1, brief 8b3b7a1.

Historical development report. Its methods, acceptance rules and conclusions apply to that recorded run, not the current release.

TakeawayNo team cleared the acceptance bar (≥90% overall, ≥80% per scenario). Best: the Qwen-reviewed team at 8/12. An independent AI judge rated the SQL itself consistently well-built (median 100/100 across 44 queries) — the failures came from misreading ambiguous questions, the same two follow-ups for every team. Next lever: question understanding, not query craft.

Catalyst turns plain-English clinical questions into database queries it runs itself. Three AI team configurations each faced the same twelve questions about an HIV dataset — including vague ones they had to ask about, and impossible ones they had to refuse — with every answer checked row-for-row against an independently written reference.

Historical development report. Its methods, acceptance rules and conclusions apply to that recorded run, not the current release.

TakeawayAll 12 scenario repetitions passed — 384/384 assertions, including every gold execution-match: the SQL the models produced returned exactly the rows an independently-authored gold query returns, on every repetition. This is an acceptance gate on one synthetic 96-patient laboratory cohort, not a benchmark — it demonstrates the governed notebook loop (draft → review → validate → execute → refine) is reliable end-to-end on real infrastructure, not that the models never err (the advisory validator and deterministic policy exist precisely because they do).

A different validation family from the chart-QA runs above: Catalyst turns a plain-language clinical question into governed SQL (a Gemma 4 12B writer drafted, a Qwen 2.5 14B reviewer checked, a deterministic policy enforcing read-only execution), then a follow-up instruction refines the exact current query. Four scenario families × 3 repetitions against the live stack and a real PostgreSQL: narrowing an unchanged base, aggregating from a dirty editor with a mid-session profile switch, correcting an unresolved parameter, and a semantic distinct-patient review. Every executed result is re-checked two ways — against an independently-run gold query (byte-level execution match) and by an independent read-only PostgreSQL cross-check.

Historical catalog entry. The linked report was unavailable when checked on 15 September 2026; no original summary was supplied.

TakeawayAutomated date checking fixed the 4B model's past-appointment mistake, and the fully checked 4B profile scored highest overall while attaching verified source records. However, two strict six-month date-window questions still slipped through, showing where date filtering needed to be tightened.

Twelve questions focused on timing and chart grounding, comparing three answer paths on Google's Gemma 4B and 12B models: a plain single-pass answer, an answer with deterministic date checking, and a fully checked profile with model review.

Judge not recorded for this run.

TakeawayThe checked 12B profile is strong on exact dates, visit ordering, appointments, and weight trends, but the independent judge caught two six-month cutoff mistakes that the built-in date check missed (including an X-ray six days outside the requested window). That shows why built-in runtime checks and independent judging both matter.

Twelve date-sensitive clinical questions answered by the checked Gemma 4 12B profile. All answers passed the built-in automated checks and scored 88.4/100 from an independent AI judge with zero unsafe answers and 66/66 source citations verified.

Judge not recorded for this run.

TakeawayEvery setup ran end to end and recorded stage timings, while surfacing formatting and table-citation bugs that were fixed before the scored runs above.

An 18-question engineering test of the 4B single model, 12B single model, and multi-agent team profiles to verify stage timing and formatting before running a full scored benchmark.

TakeawayExplicit date-window instructions boosted both single-model tiers: Gemma 4 31B rose from 84.2 to 86.5/100 (zero unsafe answers) and Gemma 4 12B rose from 80.2 to 83.5/100, outperforming both the 26B Mixture-of-Experts model (77.3/100) and the multi-agent team setups (73.1/100; the oversized 31B/35B team timed out on most questions).

Twelve date-sensitive HIV and TB chart questions tested across seven setups — comparing Google Gemma 4 12B (8-bit), Gemma 4 26B Mixture-of-Experts (4-bit), Gemma 4 31B (4-bit), and two multi-agent teams with and without explicit date-window prompt instructions alongside our automated temporal warning check.

Judge not recorded for this run.

TakeawayA separate validator model helps weaker or more error-prone writers (cutting unsafe answers from 2 to 0 on the 26B-A4B setup and adding ~3 points), but does not improve an already-safe 12B model (~81/100 with zero unsafe answers on its own). Overly strict 'cite or abstain' rules made the 12B model too cautious.

Across 297 graded answers, we tested whether pairing a writer model (Gemma 12B or Gemma 26B-A4B) with a separate Qwen 14B validator model and stricter 'cite or abstain' instructions beats running the writer model alone.

Judge not recorded for this run.

TakeawayCutting a model to 4-bit costs the 12B about 7 benchmark points AND its safety — at 8-bit it scored 84 with zero unsafe answers; at 4-bit, 76 with two. Recovering that loss by going BIGGER barely works: a 31B at 4-bit buys back only ~1 point, while simply restoring 8-bit on the 12B buys back all 7 — for this model class, spend memory on precision, not parameters. The efficiency standout was the sparse Mixture-of-Experts (26B-A4B): second-best quality (80) at HALF the latency of any other model, with just one unsafe answer. And 4-bit is most dangerous where there's least capacity to spare: the small 4B model at 4-bit was the floor (68), with the most unsafe answers (3) and the most invented facts for things not in the chart (6). Directional results from a single automated judge on a small sample, not a formal benchmark.

Six single Google Gemma models reading the same patient charts, varied along two axes: precision (8-bit vs 4-bit) and size/architecture. Two are run at BOTH precisions — a 4-billion on-device model (E4B) and a 12B dense model — to isolate quantization; alongside them a 31B dense model and a 26B sparse Mixture-of-Experts (which fires only ~4B parameters per token), both at 4-bit. Each makes a fast Answer plus a separate In-Depth, both graded. 62 clinical questions across three patients (two adults with HIV/TB, a pediatric infant), judged against the full chart by an automated clinical rubric.

Judge not recorded for this run.

TakeawayGemma 4 12B tops both axes (Answer 75.8, In-Depth 65.7) — and its lead is not a precision artifact: Gemma at 4-bit matched (and slightly edged) Gemma at 8-bit, while quantization moved the other models inconsistently and mostly within noise. Mistral NeMo was the weakest clinical reader (55.8 at 8-bit, 7 unsafe answers). Grading the separate In-Depth explanation exposed safety failures the short Answer hides — Granite 4-bit's In-Depth produced 12 unsafe elaborations and Mistral 8-bit's produced 8. The reasoning model (Qwen 3.6 35B) was safe but did not top Gemma 12B.

Eleven single-model setups, each making a fast Answer plus a separate In-Depth explanation (both graded): five model families tested at both 8-bit and 4-bit precision — Google Gemma 4 12B, Alibaba Qwen 3 14B, Qwen 2.5 14B, IBM Granite 3.3 8B, and Mistral NeMo 12B — plus a reasoning model (Qwen 3.6 35B). 32 clinical questions across three patients, judged against the full chart on two independent axes.

Judge not recorded for this run.

TakeawayHolding the in-depth prompt fixed, the team's elaboration does NOT beat a single model's. The single Gemma 12B topped BOTH axes — best Answer (73) AND best In-Depth (added-value 6.9, zero unsafe elaborations) — while the parity team's in-depth was no better (6.2), and the fast on-device Liquid 24B's in-depth was actively dangerous (10 unsafe elaborations). Separate latencies confirm the design: a quick answer (Gemma ~20s, Liquid ~2s) followed by a slower in-depth (~12-24s). One strong general model gives the best answer AND the best background; the team scaffolding adds neither. Directional results from a single automated judge on a small sample, not a formal benchmark.

The new two-call architecture: every setup returns a fast Answer and a separate, slower In-Depth, each independently timed and graded — for single models AND the multi-agent team, on equal footing, via the same shared in-depth prompt. Three patients, 32 clinical questions, five setups (Google Gemma 12B, Alibaba Qwen 14B, Google MedGemma 27B, Liquid LFM2 24B singles, plus a parity AI team).

Judge not recorded for this run.

TakeawayA strong GENERAL single model wins: Gemma 12B (72) tied the full multi-agent team, and Qwen 14B (71) was just behind AND the safest single arm (1 unsafe answer vs Gemma's 3). The surprise: the medical-specialist MedGemma 27B scored BELOW the general models (65) with more unsafe answers (5) — domain fine-tuning did not help here — and the fast on-device Liquid 24B was the weakest and least safe (49, 8 unsafe). Directional results from a single automated judge on a small sample, not a formal benchmark.

Three patients (two adults with HIV/TB and a pediatric infant), 32 clinical questions, seven setups: four single-model baselines across classes — Google Gemma 12B, Alibaba Qwen 14B, Google MedGemma 27B (a medical-specialist model), and the fast on-device Liquid LFM2 24B — plus a single model with an added elaboration and two multi-agent AI-team configurations. Every answer was graded against the full patient chart by an automated clinical rubric.

Judge not recorded for this run.

TakeawayThe model that WRITES the answer decides clinical quality. Setups with a Qwen writer scored ~70; those with a Liquid writer ~43–51 — a ~25-point gap that tracks the writer alone, whichever model coordinates. A single strong general model (Gemma 12B, 71.4) matched or beat the full multi-agent team, and a strong non-Liquid validator did not rescue a Liquid writer (13–17 unsafe answers with it switched on). Pick a strong writer model before adding team scaffolding. Directional results from a single automated judge, not a formal benchmark.

Three patients — two adults with HIV/TB and a pediatric perinatal-HIV infant — 62 clinical questions, six AI setups. A 2×2 ablation swaps the AI team's coordinator and answer-writer between a Google (Gemma) and a fast on-device (Liquid LFM2) model — every team cross-checked by a strong non-Liquid validator — alongside two single-model anchors (Gemma 12B and Liquid 24B). Every answer was graded against the full patient chart by an automated clinical rubric.

Judge not recorded for this run.

TakeawayAdding a strong non-Liquid validator to the Liquid team helped — it eliminated the made-up 'I don't know' answers and removed one unsafe response, raising the headline score from 43 to 52 — but it did not make the team safe: the validated Liquid team still produced four unsafe answers (including a harmful regimen error the validator passed) and stayed far below both a proper model team (81) and a single strong model (74). A validator catches over-confident gaps, not the subtler clinical fabrications. These are directional results from a single automated judge on a small sample, not a formal benchmark.

A follow-up to the Liquid-team run: same two patients, 16 clinical questions, four AI setups — our proper multi-agent team, a single strong model (Gemma 12B), and the fast on-device 'Liquid' team in two forms: as-is, and with a strong non-Liquid Gemma-12B 'validator' that independently re-checks each answer against the chart before it's returned. Every answer was graded against the full chart by an automated clinical rubric.

Judge not recorded for this run.

TakeawayPutting fast on-device 'Liquid' models inside a careful multi-agent team improved their sourcing — they now cite real chart records instead of nothing — but did not make them safe: both Liquid teams still fabricated medications, reversed a weight trend, and hid an AIDS-defining CD4 count, landing far below both the proper model team and a single strong model. Team scaffolding helped grounding, not clinical accuracy. These are directional results from a single automated judge on a small sample, not a formal benchmark.

Two patients, 8 clinical questions, five AI setups: a strong single model (Gemma 12B), our proper multi-agent AI team, a stripped-down 'parity' team, and two teams built around fast on-device 'Liquid' (LFM2) models — one all-Liquid, one that keeps a medical-specialist model. Every answer was graded against the full patient chart by an automated clinical rubric.

Judge not recorded for this run.

TakeawayThe Gemma 12B single model was the most accurate and the safest — no unsafe answers, and it cited a chart record for every claim. The two new Liquid models were the fastest but the least safe: they often answered confidently without grounding their claims in the chart (almost never citing a record) and produced several unsafe answers — including a 'no allergies' reply for a patient whose chart documents a penicillin allergy. The AI team cited its sources and was far safer than the Liquid models, though it didn't beat the strong single model here. These are directional results from a single automated judge on a small sample, not a formal benchmark.

Two patients, 8 clinical questions, five AI setups: Google's Gemma 12B and 4B single models, two brand-new on-device 'Liquid' (LFM2) single models (24B and 1.2B), and our multi-agent AI team. Every answer was graded against the full patient chart by an automated clinical rubric.

Judge not recorded for this run.

TakeawayRe-scored with our updated temporal rubric. The Gemma 12B single model led on the overall benchmark (80/100) and accuracy (8.7/10); the AI team's Advanced tier gave the most complete answers (7.8/10). The stricter date/trend scoring surfaced unsafe answers the first pass missed: a few setups fabricated dates or trends — the Basic AI team 3, and the Gemma 4B and 26B single models one each — while the Gemma 12B single and the Standard/Advanced/matched teams had none. Bigger isn't automatically better, and getting the timing right is where several setups slip.

Two patients, 16 questions that hinge on timing: the most-recent lab value, weight trends over months, what's changed recently, and whether a medication regimen is current. We compared plain single models (Gemma 4B → 12B → 26B) against our multi-agent AI team at four configurations.

Judge not recorded for this run.

TakeawayThe 26B single model led on accuracy (9.1/10); the AI team's Advanced tier led on completeness (8.4/10). No unsafe answers in this run.

21 high-priority questions about one complex HIV/TB patient (Aloice), comparing two single models (Gemma 2B and 26B) with three AI-team configurations.

Judge not recorded for this run.

TakeawayThe Advanced AI team led on both accuracy and completeness — but the smaller team tiers produced several unsafe answers (Basic tier: 5). That's exactly what motivated the cross-checking 'validator' added in the later runs.

An earlier, broader sweep of 22 scenarios on the same patient, comparing the Gemma 12B single model with several AI-team configurations. This run predates our safety cross-checker and the temporal-reasoning fixes.

Judge not recorded for this run.