Auto-Pipelines

This section documents the automated pipelines provided by OmniGenBench, which include AutoBench and AutoTrain. These tools are designed to streamline benchmarking and training workflows for genomic foundation models, making it easy for users to evaluate and train models with minimal manual setup.

OmniGenBench’s auto-pipelines offer:

• Automated benchmarking of genomic models using standardized protocols and datasets.

• Automated training workflows with flexible configuration options.

• Integration with model, dataset, and pipeline hubs for easy access to resources.

• Command-line interfaces for running benchmarks and training directly from the terminal.

Key Components:

• AutoBench: Automates the benchmarking process for genomic models.

• AutoTrain: Automates the training process for genomic models.

• BenchHub, ModelHub, PipelineHub: Provide access to benchmarks, models, and pipelines.

• CLI commands: Allow users to run benchmarks and training via the command line.

Example Usage:

from omnigenbench import AutoBench, AutoTrain

# Run automated benchmarking
bench = AutoBench("RGB", "model_name")
bench.run()

# Train a model
trainer = AutoTrain("RGB", "model_name")
trainer.run()

Refer to the API documentation below for details on each auto-pipeline component, including configuration, usage, and extension options.

Auto Bench

class omnigenbench.auto.auto_bench.auto_bench.AutoBench(benchmark, config_or_model, tokenizer=None, **kwargs)

Bases: object

Automated benchmarking framework for evaluating genomic foundation models across standardized benchmark suites with reproducible protocols and statistical rigor.

This class orchestrates the complete evaluation pipeline: benchmark dataset acquisition, model loading, distributed inference, metric calculation, multi-seed averaging, and results visualization. It implements best practices for genomic machine learning evaluation, including proper cross-validation, ignored label handling, and task-specific metric selection.

Design Philosophy: AutoBench follows the “Convention over Configuration” principle, providing sensible defaults while allowing full customization. By default, it uses the native trainer for single-GPU evaluation (optimizing for control and debuggability), while the CLI defaults to accelerate for distributed evaluation (optimizing for throughput).

Benchmark Suites Supported:

• RGB: RNA Genome Benchmarks (12 tasks) - RNA structure and function prediction

• BEACON: Broad Evaluation Across Computational geNOmics (13 tasks) - Multi-domain RNA

• PGB: Plant Genomics Benchmarks (7 categories) - Plant-specific sequence analysis

• GUE: Genomics Understanding Evaluation (36 datasets) - DNA general understanding

• GB: Genomics Benchmarks (9 datasets) - Classic DNA classification tasks

Evaluation Protocol:

1. Dataset Loading: Automatically downloads benchmark datasets from HuggingFace Hub or local cache, validates data format, and applies task-specific preprocessing

2. Model Initialization: Loads pre-trained models with proper task-specific heads, handling multi-label classification, regression, and token-level prediction

3. Multi-Seed Evaluation: Runs independent training/evaluation with different random seeds (typically 3-5) to quantify variance and ensure statistical significance

4. Metric Calculation: Computes task-appropriate metrics (MCC, F1, AUPRC for classification; MSE, Spearman for regression) with proper handling of ignored labels

5. Result Aggregation: Calculates mean ± standard deviation across seeds, generates visualizations, and serializes results with MetricVisualizer

Trainer Backend Selection:

• native (Python API default): Pure PyTorch training loop for single-GPU evaluation, providing explicit control over training dynamics and simplified debugging

• accelerate (CLI default): HuggingFace Accelerate for distributed evaluation across multiple GPUs, enabling efficient parallel inference on large benchmarks

• hf_trainer: HuggingFace Trainer API integration for users familiar with that ecosystem

Variables:

• benchmark (str) – Name or local path of the benchmark suite to evaluate on.

• config_or_model (str) – HuggingFace Hub identifier or local path to the model.

• tokenizer – Tokenizer instance for sequence preprocessing. Auto-loaded if None.

• autocast (str) – Mixed precision mode (‘fp16’, ‘bf16’, ‘fp32’) for memory efficiency.

• overwrite (bool) – Whether to overwrite existing evaluation results or resume from cache.

• trainer (str) – Training backend (‘native’, ‘accelerate’, ‘hf_trainer’).

• mv_path (str) – Path to MetricVisualizer file for result serialization and visualization.

• mv (MetricVisualizer) – Active visualizer instance for tracking metrics across seeds.

• bench_metadata – Benchmark configuration metadata loaded from benchmark’s metadata.py.

bench_info()

Prints and returns information about the current benchmark setup.

Returns:

str – A string containing benchmark information.

Example

>>> info = bench.bench_info()
>>> print(info)

run(**kwargs)

Runs the benchmarking process. This method iterates through the tasks in the benchmark, loads the corresponding configurations, initializes the model, tokenizer, and datasets, and then trains and evaluates the model.

Parameters:

**kwargs – Additional keyword arguments that will override the default parameters in the benchmark configuration.

Example

>>> # Run benchmarking with default settings
>>> bench.run()
>>> # Run with custom parameters
>>> bench.run(learning_rate=1e-4, batch_size=16)

omnigenbench.auto.auto_bench.auto_bench_cli.bench_command(args: list | None = None)

This function parses command-line arguments, initializes the AutoBench, and runs the evaluation.

omnigenbench.auto.auto_bench.auto_bench_cli.create_parser() → ArgumentParser

Creates the argument parser for the benchmark CLI.

Returns:

An argparse.ArgumentParser instance.

omnigenbench.auto.auto_bench.auto_bench_cli.run_bench()

This function sets up logging, constructs the command to execute (potentially with accelerate launch), and runs it.

omnigenbench.auto.auto_bench.config_check.config_check(args)

Performs a basic check on the configuration arguments.

Parameters:

args – A dictionary of configuration arguments.

Raises:

RuntimeError – If a configuration check fails.

Auto Train

class omnigenbench.auto.auto_train.auto_train.AutoTrain(dataset, config_or_model, tokenizer=None, **kwargs)

Bases: object

This class provides a comprehensive framework for training genomic models on various datasets with minimal configuration. It handles dataset loading, model initialization, training configuration, and result tracking.

AutoTrain supports various training scenarios including:

• Single dataset training with multiple seeds

• Different trainer backends (native, accelerate, huggingface)

• Automatic metric visualization and result tracking

• Configurable training parameters

Variables:

• dataset (str) – The name or path of the dataset to use for training.

• config_or_model (str) – The name or path of the model to train.

• tokenizer – The tokenizer to use for training.

• autocast (str) – The autocast precision to use (‘fp16’, ‘bf16’, etc.).

• overwrite (bool) – Whether to overwrite existing training results.

• trainer (str) – The trainer to use (‘native’, ‘accelerate’, ‘hf_trainer’).

• mv_path (str) – Path to the metric visualizer file.

• mv (MetricVisualizer) – The metric visualizer instance.

run(**kwargs)

This method loads the dataset configuration, initializes the model and tokenizer, and runs training across multiple seeds. It supports various training backends and automatic result tracking.

The method now supports loading configs from both local datasets and HuggingFace Hub datasets automatically.

Parameters:

**kwargs –

Additional keyword arguments that will override the default parameters in the dataset configuration.

Special kwargs: - config (AutoConfig): Provide a pre-loaded config instead of

auto-loading from the dataset directory.

Example

>>> # Run training with default settings (auto-loads config)
>>> trainer = AutoTrain("translation_efficiency_prediction", "yangheng/OmniGenome-186M")
>>> trainer.run()

>>> # Run with custom parameters
>>> trainer.run(learning_rate=1e-4, batch_size=16)

>>> # Run with custom config
>>> custom_config = AutoConfig.from_hub("my_dataset")
>>> trainer.run(config=custom_config)

train_info()

Print and return information about the current training setup.

Returns:

str – A string containing training setup information.

Example

>>> info = trainer.train_info()
>>> print(info)

omnigenbench.auto.auto_train.auto_train_cli.create_parser() → ArgumentParser

Creates the argument parser for the auto-train CLI.

Returns:

An argparse.ArgumentParser instance.

omnigenbench.auto.auto_train.auto_train_cli.run_train()

This function is the entry point for the ‘autotrain’ console script.

omnigenbench.auto.auto_train.auto_train_cli.train_command(args: list | None = None)

Entry point for the OmniGenome auto-train command-line interface.

Parameters:

args – A list of command-line arguments. If None, sys.argv is used.

Bench Hub

class omnigenbench.auto.bench_hub.bench_hub.BenchHub

Bases: object

A hub for accessing and managing benchmarks. This class is intended to provide a centralized way to list, download, and inspect available benchmarks for OmniGenome.

run()

Placeholder for running functionality related to the benchmark hub.