scikit-fallback is a scikit-learn-compatible Python package for machine learning with a reject option.

To allow your probabilistic pipeline to fallback—i.e., abstain from predictions—you can wrap it with a skfb rejector. Training a rejector means both fitting your model and learning the best rules to accept or reject predictions. Evaluation of a rejector depends on fallback mode (inference with or without fallback labels) and measures the ability of the rejector to both accept correct predictions and reject ambiguous ones.

For example, skfb.estimators.RateFallbackClassifierCV fits the base estimator and then finds the best confidence threshold s.t. the fallback rate on the held-out set is <= the provided value. If fallback_mode == "store", then the rejector returns skfb.core.array.FBNDArray of predictions and a sparse fallback-mask property, which lets us summarize the accuracy of both predictions and rejections.

from skfb.estimators import RateFallbackClassifierCV
from sklearn.linear_model import LogisticRegressionCV

rejector = RateFallbackClassifierCV(
    LogisticRegressionCV(cv=4, random_state=0),
    fallback_rates=(0.05, 0.06, 0.07),
    cv=5,
    fallback_label=-1,
    fallback_mode="store",
)
rejector.fit(X_train, y_train)  # Train base estimator and learn best threshold
rejector.score(X_test, y_test)  # Compute acceptance-correctness accuracy score

For more information, see the project's Wiki.

scikit-fallback requires:

• Python (>=3.9,< 3.13)
• scikit-learn (>=1.3)
• matplotlib (>=3.0) (optional)

pip install -U scikit-fallback

See the examples/ directory for various applications of fallback estimators and scorers to scikit-learn-compatible pipelines.

1. Hendrickx, K., Perini, L., Van der Plas, D. et al. Machine learning with a reject option: a survey. Mach Learn 113, 3073–3110 (2024). https://doi.org/10.1007/s10994-024-06534-x