Loading data

Loading data#

MedVision ships as a Hugging Face dataset with a custom loading script. Each config corresponds to one task on one source dataset (and one split), and asking for a config triggers download and building of the underlying Arrow files on first use. This page covers the two ways to get data onto disk — direct load_dataset() calls and the batch download_datasets CLI — plus the environment variables that control versioning and re-downloading.

For what the configs, tasks, and annotation types actually mean, see Dataset concepts. Once data is present, wire it into a benchmark run via Running evaluations.

Loading a single config#

The dataset builder reads its target directory from the MedVision_DATA_DIR environment variable, so set that before calling load_dataset(). Because the loading script runs remote code, pass trust_remote_code=True.

import os
from datasets import load_dataset

# Where raw data, caches, and built Arrow files live (a full copy is ~1TB).
os.environ["MedVision_DATA_DIR"] = "/path/to/Data"
os.environ["MedVision_PLANNER_VERSION"] = "latest"

config = "OAIZIB-CM_BoxSize_Task01_Axial_Test"

ds = load_dataset(
    "YongchengYAO/MedVision",
    name=config,
    trust_remote_code=True,
    split="test",  # "test" for *_Test configs, "train" for *_Train configs
)

Config names encode the source dataset, annotation type, sub-task, plane, and split (for example OAIZIB-CM_BoxSize_Task01_Axial_Test). The split argument you pass must match the split baked into the config name.

Warning

Pin datasets==3.6.0. The trust_remote_code mechanism that MedVision relies on to run its custom builder was removed in datasets>=4.0.0, so newer versions cannot load the dataset. This pin is already declared in medvision_bm’s dependencies.

Note

Requesting any single config pulls the raw imaging data for that source dataset in full — both the train and test halves — because the builder fetches the underlying archives before slicing. Selecting a _Test config does not mean only test-set bytes are downloaded.

Planner version and the acknowledgement gate#

MedVision_PLANNER_VERSION selects which release of the annotation logic (the “planner”) builds the samples. It is required from dataset v1.1.0 onward — loading fails loudly if it is unset. Accepted values:

Value

Resolves to

latest

the newest release (currently 1.1.1)

a pinned version — 1.1.1, 1.1.0, or 1.0.0

that exact annotation release

Different planner versions can change the exact set and framing of samples, so keep this value fixed across a benchmark to stay reproducible.

Warning

Pinning below the latest version additionally requires MedVision_ACK_RELEASE. Set it to the current latest (1.1.1) to acknowledge you have read the release note; without it, loading legacy data is blocked.

export MedVision_PLANNER_VERSION='1.1.0'
export MedVision_ACK_RELEASE='1.1.1'

Batch download: the download_datasets CLI#

To fetch many datasets ahead of time (data downloading and building is slow), use the CLI instead of scripting load_dataset() calls by hand:

# From a task-list JSON (keys are task names):
python -m medvision_bm.benchmark.download_datasets \
  --tasks_json <task-list.json> \
  --data_dir <data-folder>

# ...or from a configs CSV (config names in the first column):
python -m medvision_bm.benchmark.download_datasets \
  --configs_csv docs/dataset-configs/ConfigurationsList_Test.csv \
  --data_dir <data-folder>

Arguments:

  • --data_dir — required; the folder that becomes MedVision_DATA_DIR (datasets and the fetched dataset source code land here).

  • --tasks_json — path to a task-list JSON; its top-level keys are read as task names (the same format used under tasks_list/).

  • --configs_csv — path to a CSV whose first column lists config names. Ready-made lists ship in docs/dataset-configs/: ConfigurationsList_All.csv, ConfigurationsList_Test.csv, and ConfigurationsList_Train.csv.

  • --split — test (default) or train; controls which split of each task/config is requested.

  • --force_download_data — store-true flag that forces re-download of the raw imaging data.

Provide exactly one of --tasks_json or --configs_csv — supplying neither or both is an error. When you pass tasks, each task name is expanded to a config by appending _Test/_Train; as part of that expansion BoxCoordinate in a task name is rewritten to BoxSize to match the dataset’s config naming.

Warning

--force_download_data is a debugging aid. Because several tasks/configs can share one source dataset, it will re-download the same raw archives repeatedly. Leave it off for normal use.

Reuse, rebuild, and re-download#

Loading has two independent caches, and the controls for each are separate:

  • Built Arrow files — the cached, ready-to-serve dataset. On a second call with the same config, Hugging Face serves these directly and does not run the builder script. Pass download_mode="force_redownload" to ignore the cache and re-run the script.

  • Raw imaging data — the source images, masks, and landmarks. Whether these are re-fetched is decided by:

    • MedVision_FORCE_DOWNLOAD_DATA — set to True to force re-downloading raw data.

    • .downloaded_datasets.json — a tracker file that records which datasets have been fetched. Deleting a dataset’s entry causes its raw data to be re-downloaded next time the builder runs.

The catch is that both raw-data controls are only consulted while the builder runs. If a valid Arrow cache exists, the script is skipped and neither the env var nor the tracker is checked. So the rule is:

  • To rebuild only the built fields: download_mode="force_redownload".

  • To refresh raw data: download_mode="force_redownload" and either MedVision_FORCE_DOWNLOAD_DATA=True or an edited .downloaded_datasets.json.

import os
from datasets import load_dataset

os.environ["MedVision_DATA_DIR"] = "/path/to/Data"
os.environ["MedVision_PLANNER_VERSION"] = "latest"
os.environ["MedVision_FORCE_DOWNLOAD_DATA"] = "True"

ds = load_dataset(
    "YongchengYAO/MedVision",
    name="OAIZIB-CM_BoxSize_Task01_Axial_Test",
    trust_remote_code=True,
    split="test",
    download_mode="force_redownload",  # required, or the raw-data flags are never read
)

Note that the CLI’s --force_download_data maps to MedVision_FORCE_DOWNLOAD_DATA; it forces the builder to run and re-fetch raw data for every requested item.

Gated source datasets#

A few source datasets are access-restricted and need credentials before they will download. Set the relevant tokens in your environment first:

  • FeTA24 — hosted on Synapse behind a data-use agreement; provide SYNAPSE_TOKEN.

  • SKM-TEA — Stanford AIMI knee-MRI released under a data-use agreement, so MedVision serves it from a gated Hugging Face mirror rather than fetching it freely; provide MedVision_SKMTEA_HF_ID (a mirror repo you have been granted access to) together with HF_TOKEN.

  • ToothFairy2 — a gated Hugging Face repo; provide MedVision_ToothFairy2_HF_ID and HF_TOKEN.

Without these, requesting a config from a gated dataset will fail at download time. See the dataset card for the exact per-dataset access steps.