Metrics

Base Metrics

class omnigenbench.src.metric.metric.Metric(metric_func=None, ignore_y=-100, *args, **kwargs)[source]

Bases: OmniMetric

A flexible metric class that provides access to all scikit-learn metrics and custom metrics for evaluation.

This class dynamically wraps scikit-learn metrics and provides a unified interface for computing various evaluation metrics. It handles different input formats including HuggingFace trainer outputs and supports custom metric functions.

Variables:

  • metric_func – Custom metric function if provided

  • ignore_y – Value to ignore in predictions and true values

  • kwargs – Additional keyword arguments for metric computation

  • metrics – Dictionary of available metrics including custom ones

Example

>>> from omnigenbench import Metric
>>> metric = Metric(ignore_y=-100)
>>> y_true = [0, 1, 2, 0, 1]
>>> y_pred = [0, 1, 1, 0, 1]
>>> result = metric.accuracy(y_true, y_pred)
>>> print(result)
{'accuracy': 0.8}
compute(y_true, y_score, *args, **kwargs)[source]

Compute the metric, based on the true and predicted values.

Parameters:

  • y_true – The true values

  • y_score – The predicted values

  • *args – Additional positional arguments for the metric

  • **kwargs – Additional keyword arguments for the metric

Returns:

The computed metric value

Raises:

NotImplementedError – If no metric function is provided and compute is not implemented

omnigenbench.src.metric.metric.mcrmse(y_true, y_pred)[source]

Compute Mean Column Root Mean Square Error (MCRMSE).

MCRMSE is a multi-target regression metric that computes the RMSE for each target column and then takes the mean across all targets.

Parameters:

  • y_true (np.ndarray) – Ground truth values with shape (n_samples, n_targets)

  • y_pred (np.ndarray) – Predicted values with shape (n_samples, n_targets)

Returns:

float – Mean Column Root Mean Square Error

Raises:

ValueError – If y_true and y_pred have different shapes

Example

>>> y_true = np.array([[1, 2], [3, 4], [5, 6]])
>>> y_pred = np.array([[1.1, 2.1], [2.9, 4.1], [5.2, 5.8]])
>>> mcrmse(y_true, y_pred)
0.1833...

Classification Metrics

class omnigenbench.src.metric.classification_metric.ClassificationMetric(metric_func=None, ignore_y=-100, *args, **kwargs)[source]

Bases: OmniMetric

This class provides a comprehensive interface for classification metrics in the OmniGenome framework. It integrates with scikit-learn’s classification metrics and provides additional functionality for handling genomic classification tasks.

The class automatically exposes all scikit-learn classification metrics as callable attributes, making them easily accessible for evaluation. It also handles special cases like Hugging Face’s EvalPrediction objects and provides proper handling of ignored labels.

Variables:

  • metric_func (callable) – A callable metric function from sklearn.metrics.

  • ignore_y (any) – A value in the ground truth labels to be ignored during metric computation. Defaults to -100.

  • kwargs (dict) – Additional keyword arguments for metric computation.

compute(y_true, y_pred, *args, **kwargs)[source]

Compute the metric, based on the true and predicted values.

This method computes the classification metric using the provided metric function. It handles preprocessing and applies any additional keyword arguments.

Parameters:

  • y_true – The true values (ground truth labels).

  • y_pred – The predicted values (model predictions).

  • *args – Additional positional arguments for the metric function.

  • **kwargs – Additional keyword arguments for the metric function.

Returns:

dict – A dictionary with the metric name as key and its value.

Raises:

NotImplementedError – If no metric function is provided and the method is not implemented by the subclass.

Example

>>> metric = ClassificationMetric(metrics.accuracy_score)
>>> result = metric.compute(y_true, y_pred)
>>> print(result)  # {'accuracy_score': 0.85}

Regression Metrics

class omnigenbench.src.metric.regression_metric.RegressionMetric(metric_func=None, ignore_y=-100, *args, **kwargs)[source]

Bases: OmniMetric

This class provides access to regression-specific metrics from scikit-learn and handles different input formats including HuggingFace trainer outputs. It dynamically wraps scikit-learn metrics and provides a unified interface for computing various regression evaluation metrics.

Variables:

  • metric_func – Custom metric function if provided

  • ignore_y – Value to ignore in predictions and true values

  • kwargs – Additional keyword arguments for metric computation

  • metrics – Dictionary of available metrics including custom ones

Example

>>> from omnigenbench import RegressionMetric
>>> metric = RegressionMetric(ignore_y=-100)
>>> y_true = [1.0, 2.0, 3.0, 4.0, 5.0]
>>> y_pred = [1.1, 1.9, 3.1, 3.9, 5.2]
>>> result = metric.mean_squared_error(y_true, y_pred)
>>> print(result)
{'mean_squared_error': 0.012}
compute(y_true, y_score, *args, **kwargs)[source]

Compute the regression metric, based on the true and predicted values.

Parameters:

  • y_true – The true values

  • y_score – The predicted values

  • *args – Additional positional arguments for the metric

  • **kwargs – Additional keyword arguments for the metric

Returns:

The computed regression metric value

Raises:

NotImplementedError – If no metric function is provided and compute is not implemented

omnigenbench.src.metric.regression_metric.mcrmse(y_true, y_pred)[source]

Compute Mean Column Root Mean Square Error (MCRMSE). MCRMSE is a multi-target regression metric that computes the RMSE for each target column and then takes the mean across all targets.

Parameters:

  • y_true (np.ndarray) – Ground truth values with shape (n_samples, n_targets)

  • y_pred (np.ndarray) – Predicted values with shape (n_samples, n_targets)

Returns:

float – Mean Column Root Mean Square Error

Raises:

ValueError – If y_true and y_pred have different shapes

Example

>>> y_true = np.array([[1, 2], [3, 4], [5, 6]])
>>> y_pred = np.array([[1.1, 2.1], [2.9, 4.1], [5.2, 5.8]])
>>> mcrmse(y_true, y_pred)
0.1833...

Ranking Metrics

class omnigenbench.src.metric.ranking_metric.RankingMetric(*args, **kwargs)[source]

Bases: OmniMetric

A specialized metric class for ranking tasks and evaluation.

This class provides access to ranking-specific metrics from scikit-learn and handles different input formats including HuggingFace trainer outputs. It dynamically wraps scikit-learn metrics and provides a unified interface for computing various ranking evaluation metrics.

Variables:

  • metric_func – Custom metric function if provided

  • ignore_y – Value to ignore in predictions and true values

Example

>>> from omnigenbench import RankingMetric
>>> metric = RankingMetric(ignore_y=-100)
>>> y_true = [0, 1, 2, 0, 1]
>>> y_pred = [0.1, 0.9, 0.8, 0.2, 0.7]
>>> result = metric.roc_auc_score(y_true, y_pred)
>>> print(result)
{'roc_auc_score': 0.8}
compute(y_true, y_score, *args, **kwargs)[source]

Compute the ranking metric, based on the true and predicted values.

This method should be implemented by subclasses to provide specific ranking metric computation logic.

Parameters:

  • y_true – The true values

  • y_score – The predicted values (scores for ranking)

  • *args – Additional positional arguments for the metric

  • **kwargs – Additional keyword arguments for the metric

Returns:

The computed ranking metric value

Raises:

NotImplementedError – If compute method is not implemented in the child class