LLM-judge output parsing

LLM-judge output parsing#

The published benchmark numbers come from a strict regex parser: an answer only counts if it sits inside <answer></answer> tags (see Parsing and summarizing). That strictness has a cost — a model that measures correctly but writes \boxed{407.02, 325.62} or **Answer:** 408.2, 326.4 is scored as a miss, which mixes “the model can’t measure” together with “the model didn’t follow the output format”.

The LLM-judge pipeline (script/llm-parsing/) separates the two. It re-reads every raw response with a second, offline language model whose only job is to find the answer wherever it was written. You end up with two versions of every report: the published one, and a format-robust one. The gap between them is how much of a model’s apparent failure was formatting.

Across the 18-model roster the regex rejects a substantial, non-random slice:

Task	Responses	Rejected by the regex	Worst model
Tumour/Lesion	43,938	23.0%	Llama-3.2-11B, 85%
Angle/Distance	37,080	26.2%	Qwen2.5-VL-32B, 90.8%
Detection	415,278	6.6%	GLM-4.6V, 40.0%

Measured effect of the re-parse: T/L success 77.0% → 89.7%, A/D 73.8% → 90.1%, Detection 93.4% → 98.4%. Llama-3.2-11B’s T/L success goes from 14.5% to 97.9% — almost all of its “failure” was formatting.

What the judge does — and is not allowed to do#

The judge is an extraction device, never an evaluator. It never sees the image, never sees the ground truth, and is never asked whether a value is correct — so it cannot flatter or penalise a model.

It is also not trusted to copy numbers. The judge must quote the sentence it found the answer in; the pipeline then locates that quote in the original response (span verification) and re-reads the digits out of the quote itself. A number the model never wrote therefore cannot enter the results, even if the judge claims otherwise.

Two invariants keep the published numbers safe:

If the regex already found an answer, that answer stands. The judge can only add answers the regex missed; it can never revise a published number.
The metrics are the existing ones. MAE, MRE, IoU and SuccessRate come from the same summarizer code that produced the published numbers, just pointed at the re-parsed records. There is no second definition of any metric.

For T/L and A/D the judge also extracts the intermediate steps the prompt asked for (landmark coordinates, axis endpoints, the computed value). Those are saved but not scored.

What you get#

Re-parsed records land in a sibling llm-parsed/ folder next to each model’s parsed/; the original parsed/ files are never touched. Summary reports gain a __llm-parsed suffix so they sit beside the published ones instead of overwriting them.

Every record carries one of four outcomes:

Outcome	Meaning	Counts as parsed
answer in expected format	the regex already had it	yes
answer in another format	found and span-verified by the judge	yes
no answer stated	the response never gives one (declined, or stopped early)	no
undetermined	the judge was unusable and the regex failed	no

“Undetermined” is the pipeline’s own error rate (currently ≈ 0.4% of all responses) — recoveries that were never attempted, so the reported improvement is a floor, not a ceiling. Each recovered record also keeps the quoted sentence, so any recovered number can be checked by eye.

Running the pipeline#

The pipeline lives in script/llm-parsing/ and runs in four resumable stages: build per-task work queues from the model roster, sweep them on GPU with offline vLLM, verify spans and merge into llm-parsed/, then report. The judge reader comes from a registry; the current (and only) entry is google/gemma-4-31B-it — ~62 GB of bf16 weights, run with 2 GPUs per process.

1. Build the judge environment#

The judge needs its own Python environment (a newer vLLM than the evaluation code pins, and Transformers 5.x, which cannot resolve against vLLM’s declared bounds in a single pip pass — the install runs in two phases and prints expected dependency-conflict errors, then imports every package that objected to prove the bounds were conservative):

bash script/llm-parsing/setup_judge_env.sh    # builds <repo>/.cache/judge-env_gemma-4-31b
export PYTHON=<target>/bin/python             # the script prints this line

The setup ends with a GPU allocation check, so a bad CUDA/PyTorch pairing fails here, in minutes — not thirteen hours into a sweep. If your driver is older than CUDA 12.8, point the build at a matching wheel index: TORCH_INDEX_URL=https://download.pytorch.org/whl/cu126 bash script/llm-parsing/setup_judge_env.sh.

2. Run everything#

bash script/llm-parsing/run_llm_parsing.sh              # all stages, in order
bash script/llm-parsing/run_llm_parsing.sh --list       # show the steps, run nothing
bash script/llm-parsing/run_llm_parsing.sh analyze      # only the CPU stages + reports
bash script/llm-parsing/run_llm_parsing.sh --fresh      # start over (archives old judge output)

The driver stops at the first failed check and is resumable — finished work is skipped, so it can be killed and restarted freely. All settings are optional environment variables; the ones you are most likely to touch:

Variable	Meaning
`PYTHON`	interpreter with the judge’s PyTorch + vLLM (from step 1)
`JUDGE`	which registered reader to use (default `gemma-4-31b`)
`TASKS`	default `TL AD Detection`
`TP`	GPUs one judge process spans (default per reader; 2 for gemma-4-31b)
`NUM_SHARDS`	independent processes (default: visible GPUs ÷ `TP`)
`MOCK=1`	exercise the whole pipeline on CPU with a stand-in — never report its numbers

Every visible GPU is used: the work list is split across independent processes (cheap, near-linear speedup) and each process spans the fewest GPUs the model fits in (expensive tensor parallelism, capacity not speed). Rough cost on two H100s: a one-off model download, minutes of CPU preparation, ~13 hours of GPU for the full sweep, and ~1 hour of CPU for the reports.

Tip

Every judge answer is stamped with a fingerprint of the exact prompt and token budget that produced it, and the pipeline refuses to mix stamps — a prompt edit invalidates the queues instead of silently blending two prompts into one report. To repair a minority of bad rows without re-judging everything, see “Repairing a partial run” in the pipeline README.

Summarizing the re-parsed records#

The standard summarizers (see Parsing and summarizing) read judge output through two flags, so the strict and format-robust columns share one code path:

python -m medvision_bm.benchmark.summarize_TL_task \
    --task_dir Results/MedVision-TL-v2-CoT -p 32 --skip_model_wo_parsed_files \
    --removed_samples_dir Data/Datasets \
    --parsed_dirname llm-parsed --resps_key LLM_filtered_resps

Flag	Meaning
`--parsed_dirname <dir>`	per-model subfolder to read: `parsed` (default) or an `llm-parsed*` folder
`--resps_key <key>`	record field holding the prediction: `filtered_resps` (regex) or `LLM_filtered_resps` (judge). The summarizer aborts if the key is absent, rather than silently dropping every record.
`--models <names…>`	restrict the run to the configured roster (a results tree also carries superseded `_bugfix-*` variants and baselines)

The driver’s analyze step runs these for you; the flags matter when you aggregate by hand.

Reading the results#

Two reports per task land side by side — the published one and its __llm-parsed twin. Diff them. The judge report additionally splits each model’s failures into wrong format vs no answer given, and reports how often the judge agreed with the regex on responses the regex could already read — a free reliability check, which is why every response is re-read rather than only the failures.

Warning

Judge output is not reproducible run to run, even with greedy decoding on identical hardware — the cause is numerical non-determinism in the inference stack, not sampling. Treat the saved judge-out_*.jsonl files as the artefact of record: release those, don’t re-run and expect the same rows back. The full root-cause analysis is in docs/LLM-Judge-Reproducibility.md (measured on a since-retired reader; its headline rates do not transfer to the current one, but the operating rules do).

A low success rate that survives re-parsing is not automatically a measurement failure either: if a model’s responses pile up against its token limit it ran out of room, which must be checked against the run’s generation settings.