Release v1.1.1

Release v1.1.1#

Release notes for the medvision_bm codebase, v1.1.1. This is the version this documentation set describes; use the version switcher (bottom-right on Read the Docs) to read the docs for a different release.

New Model Evaluations#

  • Claude (Anthropic API / OpenRouter): adaptive thinking + a 28px-grid pre-resize so the prompt’s stated image/pixel size matches what the model perceives (ead5ae0); finalized eval scripts and pinned requirements_eval_claude.txt (91e14d9)

  • Gemini (Google Gemini API / OpenRouter): pass-through resize with a 3072px client-side guard and per-model image caps (47210fb)

  • OpenAI/GPT (OpenAI API / OpenRouter): patch-budget resize rule and per-model image caps (a0c5d84); added non-Pro GPT-5.5 scripts and capped --reasoning_effort/--max_tokens on all 6 GPT scripts to control runaway cost (cace62c)

  • Kimi K2.6: OpenAI-compatible API model with MoonViT floor-28 fixed-point resize and a per-model patch-budget table (e3cf881)

  • GLM-4.6V and GLM-4.6V-Flash: vLLM wrapper covering both the MoE and dense checkpoints, with reasoning-model sampling and a perceived-size resize probe (238f84b); fix GLM-4.6V environment setup (a76f382)

  • MiniMax-M3: vLLM wrapper with a perceived-size probe (a070f93); MiniMax-M3-INT4 (AWQ) on 4xH100 with CUDA-13 forward-compat fixes (1e67af7); experimental MiniMax-M3 (MXFP8) for >=8x80GB GPUs with a fail-fast aggregate-VRAM guard (91a0cc8)

  • Gemma 4 and Qwen3-VL(-Thinking): vLLM wrappers with thinking-mode support and reasoning-content merge (98b0aa9)

  • Unify HealthGPT-L14/XL32 into a single healthgpt model key, selectable via a new --model_choice CLI arg (cf52d7d, 7efb72a)

  • MedVision-V0-7B: benchmark eval scripts and environment (ae972d8); full HF model card covering the two-stage SFT+RFT recipe, <think>/<answer> output contract, and licensing, later revised for clarity and CC-BY 4.0 (575254e, 1002189, 84e8aa7)

New Metrics#

  • F1/Precision/Recall for Detection and per-sample nMAE for TL/AD added to the parsed JSONL output (1f4cc6c)

  • Step 1/2 raw L2 distance replaced by normalized L2 distance (÷√2) in the TL/AD process-accuracy scripts, making localization error comparable to the step 3/4 nMAE (986d4ea)

  • nMAE for -scaledPS files reconstructed via a BLAKE2B hash when pixel_size_scale wasn’t stored in the raw JSONL (677ccc4)

New Analyses#

  • Detection box-size analysis pipeline: per-bin Recall/Precision/F1 vs. a simulated random-detection baseline, plus a per-label × box-size visualization (4575d66, a9e2326, 9242e1b, 80508e0)

  • Removed-samples filtering added to the TL equation-/process-accuracy scripts to exclude samples dropped between dataset v1.0.0→v1.1.0 (a4a835f); later dropped from the response-visualization scripts as obsolete once dataset v1.1.1 changed which samples were removed (0719acb)

Visualization#

  • Per-sample overlay and response-visualization scripts for TL/AD/Detection tasks (c6bfea4, fcac4cf), made robust to missing or unparseable predictions (eb97ad0)

  • Radar-chart and cross-model comparison-grid compilation scripts with batch runners (4739c79, c4ca65f)

  • plot_utils extended with AD landmark plotting and a GT axis overlay for TL (130583f)

  • export_webpage_cases for the project webpage’s interactive case viewer, with dual-origin off-the-shelf rendering and a demo gallery seeded from MedVision-V0-7B (9a76fd4, 563e9c7, c205333, 6a9fbdf)

  • Image-vs-real-space ellipse-fit comparison figures illustrating the v1.1.1 TL bug (0a9ca53)

  • All figures capped at arXiv’s 34-megapixel per-image limit; transparent backgrounds and --save_as_png/--save_as_pdf output flags added (c0afb27, d4bc9d3)

Key Bugfixes — Image Perception & Generation Truncation#

  • Critical: get_resized_img_shape() returned a single shape used for both the stated image size and the pixel size; split into a (perceived-canvas, content) pair so padding models — LLaVA-OneVision, InternVL3, Llama-3.2-Vision, LLaVA-Med, HuatuoGPT-Vision, HealthGPT-L14 — report the correct TL/AD image size and pixel scale; off-the-shelf runs on non-square inputs should be re-evaluated (0a4c5e2)

  • Critical regression: lmms-eval’s fewshot-delimiter "\n\n" was being forwarded as a vLLM generation stop for every model, truncating CoT answers before <answer> (0% parseable answers observed for InternVL3-38B/Llama-3.2-Vision); removed the default until-forwarding and gated string stops on --stop_strings only, generalizing the earlier Gemma 4-specific fix (2eb7706, 18aa18a)

  • --stop_strings wired end-to-end through HuatuoGPT-Vision and LLaVA-Med to stop truncated outputs (de6ae15, a766b35)

  • min_new_tokens=16 floor prevents empty LLaVA-Med responses from greedy premature-EOS (556f0b1)

  • Fix scaledPS process-accuracy scoring against the scaled ground truth instead of the unscaled one; restore the missing MedVision-V0 scaledPS eval scripts (f6fc5e6)

  • Balance parentheses in the angle-formula CoT prompt (ae29149)

Key Bugfixes — Metrics & Summarization Pipeline#

  • Exit nonzero when lmms_eval swallows an evaluation error, so vLLM-driver eval scripts correctly mark crashed tasks as incomplete instead of “done” (813b350; paired with raising eval subprocess verbosity to INFO across the 9 remaining vLLM drivers in ee7d5f9)

  • Detection-metric parse failures now count as 0 (not NaN) in IoU/F1/Precision/Recall, matching the authoritative summarizer; fixed a crash in the no-CoT verl detection formatter (6e90af4)

  • Exclude _proc_acc/_eq_acc/_filtered sidecar JSONLs and cross-analysis outputs from the summarize/collection steps, preventing crashes and double-counted results (8644f91, 313ff45, 641517b, ba111a2)

Robustness Hardening (Audit-Driven)#

  • Replace exec() of model-generated code in tool-execution with an AST-restricted evaluator that rejects dunder/attribute/subscript/import escapes; redact secrets from evaluation-tracker logs; deterministic per-task dataset ordering to fix non-deterministic RFT crash-resume; report angle pixel size in millimeters, not degrees (742c484)

  • Replace eval() with ast.literal_eval for JSONL targets, add a robust numeric parser, and seed the weighted sampler and SFTConfig for reproducibility (28f6c67)

Dataset v1.1.1 Integration#

  • Release the MedVision dataset v1.1.1, fixing a TL anisotropic ellipse-fit bug (transposed in-plane voxel spacing); ~22% fewer TL samples on anisotropic data (fe27464)

  • Set MedVision_ACK_RELEASE=1.1.1 across benchmark/SFT scripts still pinned to older MedVision_PLANNER_VERSIONs, acknowledging the new release-acknowledgement gate (5f35017); set MedVision_PLANNER_VERSION explicitly in SFT training scripts, required since medvision_ds v1.1.0 (01557f3)

  • Add versioned v1.1.1 task lists (A/D, T/L, Detection) and a per-task pixel-size-distribution summarizer over MedVision configs (3673558, 77a00ea, a9947a3)

  • Share lmms_eval_specific_kwargs via a YAML include across landmark base configs; drop stale TL-OOD task lists (7a18805)

OOD Ablation#

  • Add MedVision-V0-7B eval scripts for plane-OOD and task-OOD ablation on TL and Detection tasks, evaluating generalization to unseen slicing planes and unseen anatomical targets; renamed the corresponding task-list JSONs to the -CoT-<split> naming convention used elsewhere (2673ce5)

SFT / RFT#

  • Fix: SFT loss masked to train only on the assistant response (not the full prompt+response sequence) in the Qwen2.5-VL collate (9945b05)

  • Feat (experimental): full-parameter and LoRA SFT scripts for MedGemma-27B, Gemma-4-31B, and Qwen3-VL-27B (0e2c0eb)

  • Feat: parquet dataset building scripts for RFT in verl (7f638e3)

Infrastructure / Install#

  • Persist per-sample model outputs to a resumable, prompt-hash-keyed JSONL cache so an interrupted eval run only re-does the in-flight sample (783dddb)

  • Build the medvision_bm wheel in a private temp dir instead of the repo tree, fixing intermittent build failures on shared CephFS checkouts (473a4fd)

  • Write task-status JSON atomically (temp file + fsync + os.replace) to survive full-disk writes (3c01bea)

  • Standardize install method and pin MedVision_PLANNER_VERSION='1.0.0' across all 38 benchmark eval scripts; rename eval script filenames to consistent Title Case and drop dead gemini keys (702c392, 7ba7f9a)

  • Relax scipy/nibabel/matplotlib version pins to fix environment setup conflicts (69f7afe)

  • Auto-format all Python and shell sources with black/isort/shfmt (cosmetic only) (bfcf773)

Documentation#

  • Add a Read the Docs documentation site (Sphinx + MyST) with autodoc/autosummary API reference over all importable modules (f1102c5)

  • Change codebase license to CC-BY 4.0 (3633fea, 1002189)

  • Document each model’s perceived-resolution strategy in docs/Model-Image-Processing.md; restructure the doc and drop the superseded perceived-size bugfix page (e930941, 8573eca)

  • Refresh README News, Quick Start, benchmarked-models list, and reference links throughout the release cycle (a168350, cb40b43, 7564b9a, 0b7106d, 58a63e3, 4a184f0, 9fe1f96, 6ffe47f, 0719acb)