benchmark pipeline

benchmark pipeline#

The scoring backend that turns per-sample model outputs into aggregated metrics. These modules are normally invoked as command-line entry points (see Command-line reference); the functions below are the reusable pieces they are built from.

eval_utils#

medvision_bm.benchmark.eval_utils.parse_sample_indices(s)[source]#

Parse a --sample_indices string into a list of integer indices.

Two formats are accepted:

  • [start:stop] -> list(range(start, stop))

  • [start,stop] or [start,stop,step] -> list(range(start, stop[, step]))

Surrounding square brackets are optional and stripped before parsing.

Parameters:

s (str) – The raw --sample_indices argument, e.g. "[0:10]" or "0,10,2".

Returns:

The list of integer indices described by s.

Return type:

list

Raises:

ValueError – If the string matches neither accepted format (a colon form without exactly two parts, or a comma form without two or three parts).

parse_outputs#

Reads the raw per-sample JSONL written during evaluation and computes per-sample metrics. main() is the CLI entry point; --task_type selects the scoring logic.

Parse benchmark output JSONL files and compute per-sample metrics.

This module post-processes the raw *.jsonl prediction files emitted by the benchmark harness. For each sample it extracts the model response, applies the answer-tag number extraction, and scores it with cal_metrics() according to the task type (AD, TL, or Detection). The augmented records are written to a parsed subdirectory alongside an updated *_results.json summary.

Run as a CLI (e.g. --task_type AD --model_dir ...); see parse_args() for the accepted arguments.

medvision_bm.benchmark.parse_outputs.main(**kwargs)[source]#

Parse benchmark JSONL files for one or more model directories.

Dispatches on whether task_dir or model_dir is given. In task_dir mode every model subdirectory is processed in turn; in model_dir mode only that single directory is processed. Each JSONL file is scored according to task_type and its parsed output plus an updated results summary are written to the model’s parsed subdirectory.

Parameters:

  • task_dir (str, optional) – Directory holding one model subdirectory per model. Mutually exclusive with model_dir.

  • model_dir (str, optional) – A single model results directory containing *.jsonl files. Mutually exclusive with task_dir.

  • task_type (str) – Scoring mode, one of "AD", "TL", or "Detection"; selects which metrics are computed for each sample.

  • limit (int, optional) – Maximum number of samples to process per JSONL file. None processes all samples.

  • skip_existing (bool) – Skip files that already have parsed outputs.

  • processes (int, optional) – Number of worker processes for parsing.

  • rm_old (bool) – Remove the existing parsed directory before processing.

Raises:

ValueError – If neither task_dir nor model_dir is provided.

medvision_bm.benchmark.parse_outputs.parse_args()[source]#

summarize_AD_task#

Aggregates parsed Angle/Distance results per anatomy.

Summarize Angle/Distance (AD) benchmark results across models.

Reads the parsed *.jsonl prediction files under each model’s parsed subdirectory, groups samples by dataset/metric label, and aggregates MAE, MRE, nMAE, success rate, and threshold-based accuracies. Per-model summaries are written as JSON (raw values and metrics) and a formatted cross-model report is saved to a text file.

Run as a CLI; see parse_args() for the accepted arguments.

medvision_bm.benchmark.summarize_AD_task.cal_metrics_AD_task(results)[source]#

Calculate metrics for a single Angle/Distance (AD) estimation sample.

Parses the model’s predicted scalar from filtered_resps and compares it with the ground-truth target to compute the mean absolute error (MAE), mean relative error (MRE), and a success flag. When distance metadata is available, a normalized MAE (nMAE) is either read from a precomputed value or reconstructed by dividing the MAE by the physical image diagonal.

Parameters:

results (dict) –

A single sample with keys:

  • filtered_resps (list): One-element list holding the prediction string.

  • target (str): Ground-truth value, parsed with ast.literal_eval.

  • doc_meta (dict, optional): Metadata used to derive nMAE, including any precomputed nmae_precomputed entry and the metric_type/scale fields.

Returns:

Metric entries whose keys match the task YAML metric fields:

  • avgMAE: {"MAE": float, "success": bool}

  • avgMRE: {"MRE": float, "success": bool}

  • SuccessRate: {"success": bool}

  • nMAE: {"NMAE": float, "success": bool}

Return type:

dict

Note

MAE/MRE are np.nan and success is False when the prediction cannot be parsed or does not contain exactly one value. The metric key names must match the metric field in the task’s YAML configuration.

medvision_bm.benchmark.summarize_AD_task.process_label_group(label, data)[source]#

Helper function to process metrics for a single label group. Used for both sequential and parallel processing.

medvision_bm.benchmark.summarize_AD_task.calculate_summary_metrics_per_anatomy_AD_task(all_data, processes=None)[source]#

Calculate summary metrics grouped by label (anatomy/metric type).

This function aggregates predictions by label and computes comprehensive metrics for each unique label (e.g., dataset_metricType_metricKey combinations).

Parameters:

  • all_data (list) – List of dictionaries, each containing: - ‘label’ (str): Label identifier (e.g., “FeTA24_distance_BPD”) - ‘targets’ (str): Ground truth value - ‘responses’ (list): Model predictions

  • processes (int, optional) – Number of processes to use for parallel calculation.

Returns:

Dictionary mapping each label to its computed metrics:

  • avgMAE, avgMRE, SuccessRate

  • MAE<k and MRE<k for k in [0.1, 0.2, …, 1.0]

  • num_samples

Return type:

dict

Note

Groups data by label before computing metrics to enable per-anatomy analysis

medvision_bm.benchmark.summarize_AD_task.find_and_group_jsonl_files(model_path)[source]#

Find and group JSONL files in a model directory by dataset and task.

Different datasets use different grouping strategies:

  • Ceph-Biometrics-400: each file is kept separate (one file per group).

  • FeTA24: all files for the same task are grouped together.

Parameters:

model_path (str) – Path to the directory containing JSONL files.

Returns:

Dictionary mapping group keys to lists of file paths. A

Ceph-Biometrics-400 file maps to a single-element list keyed by its filename, while all FeTA24 files for a task map to the combined key "FeTA24_BiometricsFromLandmarks_Task01_combined".

Return type:

dict

medvision_bm.benchmark.summarize_AD_task.process_jsonl_file(jsonl_path, limit)[source]#

Process a single JSONL file and extract label, target, and response data.

Each line in the JSONL file represents one prediction sample. This function extracts the biometric profile information and constructs a structured label.

Parameters:

  • jsonl_path (str) – Path to the JSONL file

  • limit (int, optional) – Maximum number of samples to process. None for all samples.

Returns:

List of dictionaries, each containing:

  • ’label’ (str): Constructed label (dataset_metricType_metricKey)

  • ’targets’ (str): Ground truth value

  • ’responses’ (list): Model predictions

Return type:

list

Raises:

  • ValueError – If JSON parsing fails

  • AssertionError – If metric_key is None

medvision_bm.benchmark.summarize_AD_task.process_combined_jsonl_files(jsonl_paths, limit)[source]#

Process multiple JSONL files and combine their data into a single list.

Useful for combining data from multiple files that belong to the same dataset or task group (e.g., multiple FeTA24 files).

Parameters:

  • jsonl_paths (list) – List of paths to JSONL files to combine

  • limit (int, optional) – Maximum number of samples to process per file. None for all samples.

Returns:

Combined list of dictionaries with ‘label’, ‘targets’, and ‘responses’ keys

Return type:

list

medvision_bm.benchmark.summarize_AD_task.process_parsed_file_in_model_folder(model_dir, limit=None, processes=None)[source]#

Process all parsed JSONL files in a model folder and write AD summaries.

Steps performed:

  1. Find and group JSONL files in the parsed subdirectory.

  2. Extract targets and model predictions from each file.

  3. Calculate comprehensive metrics per anatomy/metric type.

  4. Save two JSON files in the parsed directory: a values file with the raw targets and predictions, and a metrics file with the aggregated metrics per label.

If the parsed subdirectory does not exist, a message is printed and the function returns without writing any output.

Parameters:

  • model_dir (str) – Path to the model folder (expected to contain a parsed subdirectory).

  • limit (int, optional) – Maximum number of samples to process per file. None for all samples.

  • processes (int, optional) – Number of processes to use for parallel calculation.

medvision_bm.benchmark.summarize_AD_task.print_model_summaries(task_dir, limit=None, skip_model_wo_parsed_files=False)[source]#

Print and save summary metrics for all models in a task directory.

This function generates a comprehensive summary report that includes: 1. Overall weighted averages across all anatomies/metrics 2. Group-level averages (FeTA-Distance, Ceph-Angle, Ceph-Distance) 3. Label-specific detailed metrics

The output is both printed to console and saved to a text file.

Parameters:

  • task_dir (str) – Path to the task directory containing model folders

  • limit (int, optional) – Maximum number of samples to process per file

  • skip_model_wo_parsed_files (bool) – Whether to skip models without parsed folders

Input:

  • Reads from {model_dir}/parsed/summary_AD_metrics.json for each model

Output:

  • Prints formatted tables to console

  • Saves summary to {task_dir}/summary_AD_task.txt

medvision_bm.benchmark.summarize_AD_task.main(**kwargs)[source]#

Process AD model folders based on the provided arguments.

Supports two modes:

  1. task_dir mode: process all model directories within a task directory and generate a cross-model summary.

  2. model_dir mode: process a single model directory in isolation.

Parameters:

  • task_dir (str, optional) – Path to task directory (mutually exclusive with model_dir).

  • model_dir (str, optional) – Path to model directory (mutually exclusive with task_dir).

  • limit (int, optional) – Maximum number of samples to process per file. None for all samples.

  • skip_model_wo_parsed_files (bool) – Whether to skip model directories without parsed folders (task_dir mode only).

  • processes (int, optional) – Number of processes to use for parallel calculation.

Raises:

ValueError – If neither task_dir nor model_dir is provided.

medvision_bm.benchmark.summarize_AD_task.parse_args()[source]#

Parse and validate command line arguments.

Returns:

Parsed arguments with the following attributes:

  • task_dir: Path to task directory (or None)

  • model_dir: Path to model directory (or None)

  • limit: Sample limit (or None for all samples)

  • skip_model_wo_parsed_files: Boolean flag

  • processes: Number of worker processes (or None)

Return type:

argparse.Namespace

Raises:

SystemExit – If arguments are invalid (via parser.error)

summarize_TL_task#

Aggregates parsed Tumour/Lesion-size results per anatomy.

Summarize Tumor/Lesion (TL) size-estimation benchmark results across models.

Reads the parsed *.jsonl prediction files under each model’s parsed subdirectory, groups samples by anatomy/modality/slice, and aggregates MAE, MRE, nMAE, success rate, and threshold-based accuracies. Per-model summaries are written as JSON (raw values and metrics) and a formatted cross-model report is saved to a text file. Samples listed in per-dataset removed-samples files can optionally be excluded.

Run as a CLI; see parse_args() for the accepted arguments.

medvision_bm.benchmark.summarize_TL_task.cal_metrics_TL_task(results)[source]#

Calculate metrics for a single Tumor/Lesion (TL) size-estimation sample.

Parses the two predicted size values from filtered_resps and compares them with the ground-truth target to compute the mean absolute error (MAE), mean relative error (MRE), and a success flag. When metadata is available, a normalized MAE (nMAE) is read from a precomputed value or reconstructed by dividing the MAE by the physical image diagonal.

Parameters:

results (dict) –

A single sample with keys:

  • filtered_resps (list): One-element list holding the prediction string (expected to contain two comma-separated values).

  • target (str): Ground-truth values, parsed with ast.literal_eval.

  • doc_meta (dict, optional): Metadata used to derive nMAE.

Returns:

Metric entries whose keys match the task YAML metric fields:

  • avgMAE: {"MAE": float, "success": bool}

  • avgMRE: {"MRE": float, "success": bool}

  • SuccessRate: {"success": bool}

  • nMAE: {"NMAE": float, "success": bool}

Return type:

dict

Note

MAE/MRE are np.nan and success is False when the prediction cannot be parsed or does not contain exactly two values.

medvision_bm.benchmark.summarize_TL_task.process_label_group_TL(parent_class, data)[source]#

Helper function to process metrics for a single anatomy group (parent_class). Used for both sequential and parallel processing.

medvision_bm.benchmark.summarize_TL_task.calculate_summary_metrics_per_anatomy_TL_task(grouped_data, processes=None)[source]#

Calculate summary metrics for each anatomy group.

Parameters:

  • grouped_data – Dictionary with parent_class as keys and task_data as values

  • processes (int, optional) – Number of processes to use for parallel calculation.

Returns:

Dictionary with summary metrics per parent class and task type

medvision_bm.benchmark.summarize_TL_task.process_jsonl_file_TL_task(jsonl_path, limit=None, removed_set=None)[source]#

Process a JSONL file and extract relevant fields for TL task evaluation.

Parameters:

  • jsonl_path – Path to the JSONL file

  • limit – Maximum number of samples to process (None for no limit)

Returns:

(imgModality, label_name, target, filtered_resps, task_id, slice_dim)

Return type:

List of tuples

medvision_bm.benchmark.summarize_TL_task.process_parsed_file_in_model_folder(model_dir, limit=None, processes=None, removed_samples_dir=None, removed_samples_filename=None)[source]#

Process all JSONL files in a model folder and generate summary metrics.

Parameters:

  • model_dir – Path to the model folder

  • limit – Maximum number of samples to process per file (None for no limit)

  • processes (int, optional) – Number of processes to use for parallel calculation.

  • removed_samples_dir (str, optional) – Root directory containing per-dataset removed_samples JSON files. When provided, matching samples are excluded.

  • removed_samples_filename (str, optional) – Filename within each dataset subdirectory.

medvision_bm.benchmark.summarize_TL_task.print_model_summaries(task_dir, limit=None, skip_model_wo_parsed_files=False, removed_samples_dir=None)[source]#

Print and save summary metrics for all models in a task directory.

Parameters:

  • task_dir – Path to the task directory containing model folders

  • limit – Maximum number of samples to process per file (None for no limit)

  • skip_model_wo_parsed_files – Whether to skip models without parsed folders

  • removed_samples_dir (str, optional) – When provided, reads filtered metrics files.

medvision_bm.benchmark.summarize_TL_task.main(**kwargs)[source]#

Process TL model folders based on the provided arguments.

Dispatches to task_dir mode (loop over all model directories and generate a cross-model summary) or model_dir mode (process a single model directory).

Parameters:

  • task_dir (str, optional) – Path to task directory (mutually exclusive with model_dir).

  • model_dir (str, optional) – Path to model directory (mutually exclusive with task_dir).

  • limit (int, optional) – Maximum number of samples to process per file.

  • skip_model_wo_parsed_files (bool) – Whether to skip model directories without parsed folders.

  • processes (int, optional) – Number of processes to use for parallel calculation.

  • removed_samples_dir (str, optional) – Root directory with per-dataset removed_samples JSON files.

  • removed_samples_filename (str, optional) – Filename within each dataset subdirectory.

Raises:

ValueError – If neither task_dir nor model_dir is provided.

medvision_bm.benchmark.summarize_TL_task.parse_args()[source]#

Parse command line arguments.

summarize_detection_task#

Aggregates parsed detection results per anatomy.

Summarize object-detection (bounding-box) benchmark results across models.

Reads the parsed *.jsonl prediction files under each model’s parsed subdirectory, groups samples by anatomy/modality/slice, and aggregates MAE, IoU, F1, precision, recall, success rate, and threshold-based statistics. Regions are further grouped into anatomy vs. tumor/lesion categories. Per-model summaries are written as JSON and a formatted cross-model report is saved to a text file.

Run as a CLI; see parse_args() for the accepted arguments.

medvision_bm.benchmark.summarize_detection_task.calculate_summary_metrics_per_anatomy_detection_task(grouped_data)[source]#

Calculate summary metrics for each anatomy group.

Parameters:

grouped_data – Dictionary with parent_class as keys and task_data as values

Returns:

Dictionary with summary metrics per parent class and task type

medvision_bm.benchmark.summarize_detection_task.group_anatomy_vs_tumor_lesion(model_path, limit=None)[source]#

Group anatomical regions into ‘anatomy’ vs ‘tumor/lesion’ (T/L) categories and calculate weighted mean metrics for each group.

This function: 1. Reads per-region metrics from SUMMARY_FILENAME_DETECT_METRICS 2. Classifies regions as anatomy or tumor/lesion based on keywords 3. Filters out regions marked as miscellaneous/others or with insufficient samples 4. Calculates sample-weighted mean metrics for each group 5. Saves results to SUMMARY_FILENAME_GROUPED_ANATOMY_VS_TUMOR_LESION_DETECT_METRICS

Parameters:

  • model_path – Path to the model folder containing summary metrics file

  • limit – Maximum samples to process per file (None = all)

medvision_bm.benchmark.summarize_detection_task.process_jsonl_file_detection_task(jsonl_path, limit=None)[source]#

Parse a JSONL results file and extract detection task data.

This function: 1. Extracts dataset name from filename 2. Parses each line for label, target, response, task_id, etc. 3. Resolves label names using benchmark plan configuration 4. Returns structured data for downstream processing

Parameters:

  • jsonl_path – Path to the JSONL file

  • limit – Maximum number of samples to process (None = process all)

Returns:

(imgModality, label_name, target,

filtered_resps, task_id, slice_dim)

Return type:

List of tuples

medvision_bm.benchmark.summarize_detection_task.process_parsed_file_in_model_folder(model_dir, limit=None, processes=None)[source]#

Process all JSONL files in a model’s parsed folder and generate summary metrics.

This function performs the complete pipeline: 1. Finds all JSONL files in model_dir/parsed/ 2. Parses each file to extract detection data 3. Groups data by anatomy-modality-slice combinations 4. Calculates summary metrics per group 5. Saves intermediate and final results as JSON files 6. Generates anatomy vs tumor/lesion grouped metrics

Parameters:

  • model_dir – Path to the model folder

  • limit – Maximum number of samples to process per file (None = all)

  • processes – Number of worker processes to use (None = single process)

medvision_bm.benchmark.summarize_detection_task.print_summary_metrics(task_dir, limit=None, skip_model_wo_parsed_files=False)[source]#

Print and save summary metrics for all models in a task directory.

This function: 1. Collects metrics from all model directories 2. Prints formatted summary table to console 3. Saves metrics to JSON file 4. Saves console output to text file

Parameters:

  • task_dir – Path to task directory containing model folders

  • limit – Maximum samples to process per file (None = all)

  • skip_model_wo_parsed_files – If True, skip models without parsed folders

medvision_bm.benchmark.summarize_detection_task.main(**kwargs)[source]#

Process detection model folders based on the provided arguments.

Dispatches to task_dir mode (loop over all model directories and generate a cross-model summary) or model_dir mode (process a single model directory).

Parameters:

  • task_dir (str, optional) – Path to task directory (mutually exclusive with model_dir).

  • model_dir (str, optional) – Path to model directory (mutually exclusive with task_dir).

  • limit (int, optional) – Maximum number of samples to process per file.

  • skip_model_wo_parsed_files (bool) – Whether to skip model directories without parsed folders.

  • processes (int, optional) – Number of worker processes to use.

Raises:

ValueError – If neither task_dir nor model_dir is provided.

medvision_bm.benchmark.summarize_detection_task.parse_args()[source]#

Parse command line arguments.

Supports two modes: - Task mode (–task_dir): Process all models in a task directory - Model mode (–model_dir): Process a single model directory

Returns:

Parsed command line arguments