MedVision

MedVision#

MedVision MedVision

Dataset and benchmark for quantitative medical image analysis

Paper · Project page · Dataset · Models · Demo · PyPI · Code

MedVision measures whether vision–language models (VLMs) can read quantities off a medical image — the size of a tumour in millimetres, the coordinates of a bounding box, the angle between two anatomical landmarks — rather than only naming what they see. It ships a large annotated dataset (22 source datasets, ~30.8M annotated samples across CT, MRI, PET, ultrasound and X-ray), a reproducible benchmark harness, and post-training recipes (SFT, RFT/RL, chain-of-thought, LoRA).

The medvision_bm package is the codebase behind the benchmark. This site documents how to run the benchmark, fine-tune models, and call the core API.

The three tasks#

🎯 Detection

Predict a bounding box (BoxCoordinate) around a target structure. Scored with IoU, precision, recall and F1.

📏 Tumour / Lesion size

Estimate the physical size of a tumour or lesion in millimetres (TumorLesionSize). Scored with MAE and MRE.

📐 Angle / Distance

Measure an angle (degrees) or distance (mm) between landmarks (BiometricsFromLandmarks). Scored with MAE and MRE.

Quickstart#

Install the package and run a small evaluation on one task, then aggregate the scores:

# 1. Install from PyPI
pip install medvision-bm

# 2. Point the loader at a data directory and pin a dataset version
export MedVision_DATA_DIR=/path/to/Data
export MedVision_PLANNER_VERSION=1.1.1

# 3. Evaluate an open-weight VLM on a handful of Angle/Distance samples
python -m medvision_bm.benchmark.eval__qwen2_5_vl \
    --model_hf_id Qwen/Qwen2.5-VL-7B-Instruct \
    --model_name Qwen2.5-VL-7B-Instruct \
    --data_dir /path/to/Data \
    --tasks_list_json_path tasks_list/tasks_MedVision-AD-CoT.json \
    --task_status_json_path completed_tasks/AD.json \
    --results_dir Results/MedVision-AD-CoT \
    --sample_limit 10

# 4. Parse per-sample outputs, then summarise per-anatomy metrics
python -m medvision_bm.benchmark.parse_outputs --task_type AD --task_dir Results/MedVision-AD-CoT -p 8
python -m medvision_bm.benchmark.summarize_AD_task --task_dir Results/MedVision-AD-CoT -p 8

See Quickstart for the annotated walkthrough, and Installation for Docker and the full dependency setup.

How the documentation is organised#

  • Getting started — install the package, then run your first end-to-end evaluation.

  • Dataset — the data model (config naming, annotation types, physical units) and how to load it.

  • Benchmarking — the three-step eval → parse → summarize pipeline and the metrics.

  • Fine-tuning — supervised (SFT) and reinforcement (RFT) post-training recipes.

  • Reference — the complete command-line reference and the autogenerated Python API.

  • Extending — how to add a new model or a new task to the harness.

The benchmarking codebase is published on PyPI as medvision-bm (pip install medvision-bm).

Citation#

If MedVision, the medvision_bm code, or the MedVision-V0 models are useful in your research, please cite the preprint:

@misc{yao2026medvisionbenchmarkingquantitativemedical,
      title={MedVision: Benchmarking Quantitative Medical Image Analysis},
      author={Yongcheng Yao and Yongshuo Zong and Raman Dutt and Yongxin Yang and Sotirios A Tsaftaris and Timothy Hospedales},
      year={2026},
      eprint={2511.18676},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.18676},
}