Parsing and summarizing results

Parsing and summarizing results#

Running a model over the benchmark (see Running evaluations) only produces raw per-sample predictions. Turning those into scores is two more steps:

  1. Parse — read each model’s raw JSONL, pull the numeric answer out of the model’s free-text response, and score every sample. This writes a parsed/ copy of the results.

  2. Summarize — aggregate the parsed per-sample scores into per-anatomy and per-model tables.

Both steps run offline on CPU and are safe to re-run; nothing here touches the model or the GPU. For what the metrics mean (IoU, MAE, MRE, nMAE, SuccessRate, …) see the Benchmark overview.

Where the files live#

Evaluation writes one folder per model under a task tag:

Results/<task_tag>/<model>/
    <task_id>_samples_....jsonl     # raw per-sample predictions (one JSON object per line)
    <task_id>_results.json          # lmms-eval run summary for that task

Parsing adds a sibling parsed/ folder; summarizing then drops aggregate files into it and a human-readable table at the task level:

Results/<task_tag>/<model>/parsed/
    <task_id>_samples_....jsonl       # same samples, now with per-sample metrics attached
    <task_id>_results.json            # run summary updated with avgMAE / avgMRE / avgIoU / SuccessRate
    summary_metrics_<task>_Task.json  # aggregated metrics per anatomy label (written by summarize)
    summary_values_<task>_Task.json   # raw targets + predictions per group (written by summarize)
Results/<task_tag>/
    summary_<task>_task.txt           # formatted per-model tables printed to console + saved here

Step 1 — Parse raw outputs#

python -m medvision_bm.benchmark.parse_outputs \
    --task_type Detection \
    --task_dir Results/MedVision-detect-v2 \
    -p 32

Point --task_dir at a task folder and every model subfolder under it is processed in turn. (You can instead target one model with --model_dir <path>; exactly one of the two is required.)

For each raw *.jsonl the parser sorts samples by doc_id, extracts the last numeric values from within the <answer></answer> tags of the model’s response (1 value for A/D, 2 for T/L, 4 box coordinates for Detection), scores each sample, and writes the annotated copy to <model>/parsed/. It also refreshes that model’s <task_id>_results.json with the aggregate avgMAE, avgMRE, avgIoU, SuccessRate, and the binned MRE<0.1 … MRE<1.0 fractions.

Flag

Meaning

--task_type {AD,TL,Detection}

Required. Selects the scoring logic and the expected number of predicted values.

--task_dir <dir>

Task folder; loops over every model subfolder inside it.

--model_dir <dir>

Alternative to --task_dir: parse a single model folder.

-p, --processes <N>

Worker processes (one JSONL file per worker). Omit for single-process.

--limit <N>

Score only the first N samples per file — handy for a quick smoke test.

--skip_existing

Leave files that already have a parsed counterpart untouched.

--rm_old

Delete the model’s existing parsed/ folder before re-parsing (clean rebuild).

Tip

--skip_existing resumes an interrupted parse cheaply; --rm_old forces a full re-parse when you have changed the scoring code or re-run the evaluation. Do not combine them — they pull in opposite directions.

Step 2 — Summarize into tables#

Each task type has its own summarizer. They read the parsed/ folders, group samples by anatomy, and emit summary_metrics_<task>_Task.json / summary_values_<task>_Task.json per model plus a task-level summary_<task>_task.txt.

# Angle / Distance
python -m medvision_bm.benchmark.summarize_AD_task \
    --task_dir Results/MedVision-AD-v2-CoT -p 32 --skip_model_wo_parsed_files

# Tumour / Lesion size  (note the removed-samples caveat below)
python -m medvision_bm.benchmark.summarize_TL_task \
    --task_dir Results/MedVision-TL-v2-CoT -p 32 --skip_model_wo_parsed_files \
    --removed_samples_dir Data/Datasets

# Detection
python -m medvision_bm.benchmark.summarize_detection_task \
    --task_dir Results/MedVision-detect-v2 -p 32 --skip_model_wo_parsed_files

Shared flags across all three:

Flag

Meaning

--task_dir <dir> / --model_dir <dir>

Aggregate every model under a task tag, or a single model. One is required.

-p, --processes <N>

Parallelize the per-label metric computation.

--limit <N>

Match a limited parse run; also suffixes output filenames with _limit<N>.

--skip_model_wo_parsed_files

Ignore model folders that were never parsed. Valid only with --task_dir.

What each summarizer groups on:

  • A/D groups by dataset_metricType_metricKey labels and also reports rolled-up group averages (FeTA-Distance, Ceph-Angle, Ceph-Distance). Samples whose ground-truth value is below the near-zero threshold (0.1) are dropped so relative error stays meaningful.

  • T/L groups by anatomy label crossed with imaging modality and slice plane, weighting per-label averages by sample count.

  • Detection groups by anatomy/modality/slice, then additionally splits regions into an anatomy bucket vs a tumour/lesion (T/L) bucket. Regions tagged miscellaneous/others and regions with fewer than the minimum group size (50 samples) are excluded from the grouped means, and a random_detection baseline folder, if present, is skipped.

T/L only: excluding multi-cluster samples#

The published T/L benchmark is scored against v1.0.0 annotations, but dataset release v1.1.0 identified a set of samples whose target spanned multiple disconnected clusters. To reproduce the reported numbers you must drop those samples at summarize time:

--removed_samples_dir Data/Datasets

With this set, the summarizer looks for multi_cluster_samples_v1.0.0_to_v1.1.0.json inside each dataset folder under that root (override the filename with --removed_samples_filename) and skips any matching sample. The resulting output files gain a _filtered suffix so they never overwrite an unfiltered run. This flag exists only on summarize_TL_task.

Warning

Omitting --removed_samples_dir for T/L silently includes the multi-cluster samples and yields numbers that do not line up with the paper.

Beyond the published numbers#

Two follow-on analyses build directly on the parsed/ records this page produces:

  • LLM-judge output parsing re-reads every raw response with a second, LLM-based reader so answers written outside the <answer> tags are recovered — separating can’t measure from didn’t follow the output format.

  • Clinical Decision Agreement pushes each parsed measurement through published clinical cutoffs (tumour staging, skeletal classification) and asks whether the model’s error would change the clinical decision.