Three-layer read-only evaluation reference
- Layer 1 · Deterministic
- HTTP/schema validity, exact chart-reference UUID resolution, and temporal guardrail gate status — computed without an LLM judge.
- Layer 2 · LLM Judge
- advisory Scout autorater composite (0–100), Safety-Adjusted Benchmark (hard-gated on severe harm / critical omission), and per-axis scores.
- Layer 3 · Human & PPI
- clinician/human adjudication recorded via
python -m harness.validate adjudicate <run_dir>, paired with Prediction-Powered Inference (PPI) calibration and Cohen’s κ. - ⏱ latency
- end-to-end response time; orange = first turn per backend (carries model warmup).
- chart refs
- count of chart records cited — a COUNT, not a grounding/quality signal.
- not surfaced
- tokens / finish_reasons / response model — not returned by
/chat (OTel-deferred).