Reinforcement fine-tuning (RFT)

Reinforcement fine-tuning (RFT)#

RFT closes the loop after supervised fine-tuning: instead of imitating chain-of-thought traces, the model is optimised directly against a reward computed from the geometry of its predictions. MedVision runs this with GRPO (group-relative policy optimisation).

Who does what#

The work is split across two repositories, and it helps to keep them straight:

This package (medvision_bm) builds the data. It converts MedVision tasks into verl-ready Parquet datasets: rendered images, chat-formatted prompts, ground-truth answers, and the per-sample metadata the reward functions need.
The verl fork runs the training. The GRPO loop, the rollout/actor machinery, and the reward functions themselves all live on the medvision-rl branch. medvision_bm never launches training — it only produces the Parquet files that the fork consumes.

Note

The reward functions on the fork (under verl/utils/reward_score/medvision_rewards/) are the consumers of the metadata built here. When this page mentions a field like landmark_1_wh or a data_source such as medvision-detection, that is the contract between the two repos.

Building the Parquet dataset#

The entry point is a plain argparse module you invoke with python -m:

python -m medvision_bm.rft.verl.build_parquet_ds \
    --model_family_name qwen25vl \
    --model_hf "Qwen/Qwen2.5-VL-7B-Instruct" \
    --data_dir "$MedVision_DATA_DIR" \
    --tasks_list_json_path_AD     tasks_list/tasks_MedVision-AD__train_SFT.json \
    --tasks_list_json_path_detect tasks_list/tasks_MedVision-detect__train_SFT.json \
    --tasks_list_json_path_TL     tasks_list/tasks_MedVision-TL__train_SFT.json \
    --train_sample_limit_task_AD        5500 --val_sample_limit_task_AD        45 \
    --train_sample_limit_task_Detection 110000 --val_sample_limit_task_Detection 105 \
    --train_sample_limit_task_TL        5500 --val_sample_limit_task_TL        50 \
    --train_sample_limit 121000 --val_sample_limit 200 \
    --new_shape_hw 512 512

The script/rft/ directory ships ready-to-run wrappers around this call — each one pins a conda env, rebuilds the wheel, runs medvision_bm.sft.env_setup, and then invokes the builder with a fixed task mix:

Script	Task mix (train / val)
`build_parquet_ds__verl__D0k-AD5.5k-TL0k__512x512.sh`	A/D only (5.5K / 45)
`build_parquet_ds__verl__D0k-AD0k-TL5.5k__512x512.sh`	T/L only (5.5K / 50)
`build_parquet_ds__verl__D110k-AD0k-TL0k__512x512.sh`	Detection only (110K / 105)
`build_parquet_ds__verl__D110k-AD5.5k-TL5.5k__512x512.sh`	all three combined (121K / 200)
`build_parquet_ds__verl__D1000k-AD0k-TL0k__512x512__checkpointed.sh`	Detection at scale (1M / 500)

Builder variants#

Three modules share the same CLI surface; pick by dataset size and intent:

medvision_bm.rft.verl.build_parquet_ds — the default. Loads, formats, and writes each task in one pass.
medvision_bm.rft.verl.build_parquet_ds__checkpointed — writes intermediate shards so a run can survive an out-of-memory kill and resume. Use it for the very large detection splits (the ~1M-sample script above relies on it).
medvision_bm.rft.verl.build_parquet_ds_with_testset (and its __checkpointed twin) — also carves out a held-out test split. GRPO training itself only reads train and validation; the test split exists for debugging and offline inspection.

Key flags#

Flag	Meaning
`--data_dir`	Root data folder (mirror of `MedVision_DATA_DIR`); output lands under `<data_dir>/verl_datasets/<model_family_name>/`.
`--model_family_name`	Image-processor group (e.g. `qwen25vl`). Determines how images are resized.
`--model_hf`	Hugging Face id used to load that processor.
`--tasks_list_json_path_AD` / `_detect` / `_TL`	One task-list JSON per task type. Supply one, two, or all three; missing ones are skipped.
`--train_sample_limit_task_AD` / `_Detection` / `_TL` (+ `val_` twins)	Per-task caps applied while each task is loaded (`-1` = no limit).
`--train_sample_limit` / `--val_sample_limit`	Global caps applied after the tasks are concatenated. Keep these equal to the sum of the per-task limits so nothing is silently truncated.
`--new_shape_hw H W`	Resize target as height then width (e.g. `512 512`); omit to keep native resolution.
`--without_cot_instruction`	Emit the lite (no reasoning-format) prompt. Deprecated — the intended pipeline is SFT-CoT followed by RFT, so keeping the CoT instruction avoids a train-time distribution shift.

The builder writes train_verl.parquet and validation_verl.parquet. Each row carries the fields verl expects — prompt, images, ground_truth, data_source, ability, reward_model, and extra_info.

Warning

A built dataset is tied to the model_family_name it was made for: the image resize ratio and final pixel dimensions come from that model’s processor. Reusing a Parquet file with a different model family will feed it mispreprocessed images and mismatched prompts. The output directory name encodes the model and sample limits so caches can’t collide.

The reasoning-format prompt#

Every sample’s system message tells the model to separate its reasoning from its final answer. Two variants live in rft_prompts.py:

SYSTEM_PROMPT (the CoT builders) asks for reasoning inside <think>...</think> and the result inside <answer>...</answer>, and additionally requires each intermediate step to be wrapped as <step-k-reasoning> / <step-k-answer> pairs. That per-step structure is what makes a process reward possible — the grader can read off each intermediate landmark.
SYSTEM_PROMPT_LITE (deprecated) — the --without_cot_instruction path; keeps only the outer <think> / <answer> split, with no per-step tags.

So a CoT rollout looks like:

<think>
  <step-1-reasoning> ... </step-1-reasoning>
  <step-1-answer> ... </step-1-answer>
  ...
</think>
<answer> 12.480, 7.930 </answer>

How the reward is composed#

The reward lives entirely on the verl fork; here it is enough to know its shape and what data feeds it. Conceptually it sums three parts:

Format reward — did the rollout produce a well-formed <think>/<answer> (and, for CoT, <step-k-*>) structure that can be parsed at all.
Process reward — how close each intermediate step is to ground truth. The CoT builders populate extra_info with the true intermediate coordinates so this can be scored: landmark pairs (landmark_P1_wh…landmark_P4_wh) for T/L, the two points or two lines for A/D, and the box corners for detection.
Answer reward — the final metric against ground_truth.

Each row also stamps a data_source / ability that routes it to the right reward function: medvision-tl for tumour/lesion size, medvision-ad (further split into medvision-angle / medvision-distance by metric type) for angle-distance, and medvision-detection for detection.

For the exact field contracts and the formatter functions that emit them, see the API reference for rft.verl.verl_utils. For the GRPO configuration and the reward implementations, follow the verl fork.

After training#

Point the standard benchmark pipeline at the fine-tuned checkpoint, exactly as you would for any other model — see Running evaluations. The script/benchmark-*/ drivers include an eval__MedVision-V0-7B__detect.sh example for scoring an RFT’d detection model.