Radical honesty
We publish what works and what doesn't — negative results included. A medical decision-support tool earns trust by being auditable, not by hiding its limits.
The production MS head is the V7 SSL-foundation ensemble — Dice 0.645 against the MSSEG-2016 consensus test (38 patients; reproduced on 2026-10-05 with the original evaluation code, result file archived). Compared like with like, the model agrees with an individual expert at Dice 0.53, while the seven experts agree with each other at 0.60: the model is below expert-to-expert agreement (gap −0.07, 95% CI −0.10 to −0.035), mostly on scans with a small lesion load. The table below is the early (v1) experiment history, not the current model.
| Model | Data | Val Dice | Status |
|---|---|---|---|
| SegResNet (FLAIR-only) | 60 real MS patients, patient-level split | 0.52 | v1 — superseded by V7 |
| SegResNet (T1+T2+FLAIR, naive registration) | same, affine-header alignment | 0.45 | rejected |
| SegResNet (T1+T2+FLAIR, proper SimpleITK registration) | same, intensity-based registration | 0.47 | rejected — still < FLAIR-only |
Honest finding: even with correct registration, multi-sequence did not beat FLAIR-only here (the ground-truth mask is FLAIR-derived). We kept the simpler, stronger model rather than ship a worse one for novelty.
FLAIR → MNI152 affine registration (SimpleITK) + a Harvard-Oxford region atlas. Each lesion is classified by tight abutment: ≤3 mm to a ventricle = periventricular, ≤1 mm to cortex = juxtacortical, brain-stem/cerebellum = infratentorial.
An earlier atlas-free guess was tested and dropped as unreliable — anatomical labels that contradict reality are worse than none.
Confidence is recalibrated (ECE), and a conformal selector is designed to bound committed-error ≤ α with probability ≥ 1−δ. No guarantee is certified today: the current calibration run is infeasible (fitted on synthetic cases, with no clinician labels yet), and the threshold-selection procedure is being redesigned because searching many thresholds weakens the stated bound. Until then the agent abstains by policy, without a statistical guarantee.
Deterministic, auditable orchestrator (not a generic agent loop) · FlammaGuard blocks final-diagnosis language and forces specialist review · FlammaCrossCheck catches image↔report contradictions (e.g. laterality flips) even at high confidence · self-consistency voting + multi-agent debate on hard cases · every output carries the disclaimer and is logged for clinician validation.
The exact vision model in production, its held-out Dice, training data and robustness — plus the full version lineage. Every assessment can state which agent version produced it.