Welcome to the EquiBoots Documentation!

Note

This documentation is for equiboots version 0.0.0a13.

EquiBoots is a fairness-aware model evaluation toolkit for auditing performance disparities across demographic groups in machine learning models. It provides robust, bootstrapped evaluation metrics for binary, multi-class, and multi-label classification tasks, as well as regression models.

The library supports:

• Group-wise performance slicing

• Fairness diagnostics and disparity metrics

• Confidence intervals via bootstrapping

• Customizable and publication-ready visualizations

• Statistical tests to assess performance differences

EquiBoots is suited for applications in clinical, social, and policy contexts: domains where transparency, bias mitigation, and equitable outcomes are essential for responsible AI/ML deployment.

Project Links

1. PyPI Page

2. GitHub Repository

3. Classification: Google Colab Example Notebook

4. Regression: Google Colab Example Notebook

Prerequisites

Before you install equiboots, ensure your system meets the following requirements:

• Python (version 3.7.4 or higher)

Additionally, equiboots depends on the following packages, which will be automatically installed when you install equiboots:

• matplotlib: version 3.5.3 or higher, but capped at 3.10.1

• numpy: version 1.21.6 or higher, but capped at 2.2.4

• pandas: version 1.3.5 or higher, but capped at 2.2.3

• scikit-learn: version 1.0.2 or higher, but capped at 1.5.2

• scipy: version 1.8.0 or higher, but capped at 1.15.2

• seaborn: version 0.11.2 or higher, but capped at 0.13.2

• statsmodels: version 0.13 or higher, but capped at 0.14.4

• tqdm`: version 4.66.4 or higher, but capped below 4.67.1

Installation

You can install equiboots directly from PyPI:

pip install equiboots

Description

This guide provides detailed instructions and examples for using the functions provided in the equiboots library and how to use them effectively in your projects.

---

Table of Contents

Getting Started

• Welcome to the EquiBoots Documentation!
  • Project Links
  • Prerequisites
  • Installation
  • Description

Point Estimate Metrics

• Point Estimate Evaluation
  • Supported Metrics
  • Initial Set-up
    • Step 1: Import and Initialize EquiBoots
    • Step 2: Slice Groups and Compute Point Estimates
  • Metrics DataFrame
    • metrics_dataframe()
  • Statistical Tests
    • analyze_statistical_significance()
    • Example
  • Statistical Significance Plots
    • Test Setup
• Group Metrics Point Plot
  • eq_group_metrics_point_plot()
  • Statistical Metrics table
    • metrics_table()
• Group Curve Plots
  • eq_plot_group_curves()
  • ROC AUC Curve
  • Precision-Recall Curves
  • Calibration Plots
    • Example 1 (Calibration Overlay)
    • Example 2 (Calibration Subplots)
    • Example 3 (LOWESS Calibration)
    • Example 4 (Calibration Histograms)
• Group Metrics Forest Plot
  • eq_plot_metrics_forest()
  • Example 1: Prevalence
• Effect Size
  • plot_effect_sizes()
• Residual-Based Point Estimates
  • eq_plot_residuals_by_group()
  • Residual Plot Examples
    • Example 1: Residuals by Age Group (Overlayed)

Bootstrap Estimate Metrics

• Bootstrap Estimate Evaluation
  • 1. Bootstrap Setup
    • Step 1.1: Instantiate EquiBoots with Bootstrapping
    • Step 1.2: Slice by Group and Compute Metrics
  • 2. Disparity Analysis
    • Plot Disparity Ratios
      • eq_group_metrics_plot()
  • 3. Metric Differences
  • 4. Statistical Significance Testing
    • 4.1: Metrics Table with Significance Annotations
    • 4.2: Visualize Differences with Significance
  • Bootstrapped Group Curve Plots
    • eq_plot_bootstrapped_group_curves()
    • ROC AUC Curves
    • Precision-Recall Curves
    • Calibration Curves
  • Bootstrapped Forest Plots
    • eq_plot_bootstrap_forest()
    • Tabular Breakdown of Bootstrapped Metrics
      • calculate_bootstrap_stats()
  • Summary

Classes, Attributes, & Methods

• EquiBoots Class
  • EquiBoots
    • EquiBoots.calculate_differences()
    • EquiBoots.calculate_disparities()
    • EquiBoots.calculate_groups_differences()
    • EquiBoots.calculate_groups_disparities()
    • EquiBoots.check_classification_task()
    • EquiBoots.check_fairness_vars()
    • EquiBoots.check_group_empty()
    • EquiBoots.check_group_size()
    • EquiBoots.check_task()
    • EquiBoots.get_groups_metrics()
    • EquiBoots.get_metrics()
    • EquiBoots.list_adjustment_methods()
    • EquiBoots.list_available_tests()
    • EquiBoots.set_fix_seeds()
    • EquiBoots.set_reference_groups()
  • Overview
  • Constructor
  • Main Methods
    • grouper()
    • slicer()
    • get_metrics()
    • calculate_disparities()
    • calculate_differences()
    • analyze_statistical_significance()
    • set_fix_seeds()
    • list_available_tests()
    • list_adjustment_methods()
  • Non-Main/Internal Methods
    • set_reference_groups()
    • check_task()
    • check_classification_task()
    • check_fairness_vars()
    • check_group_size()
    • check_group_empty()
    • sample_group()
    • groups_slicer()
    • get_groups_metrics()
    • calculate_groups_disparities()
    • calculate_groups_differences()
  • Example Usage
  • Point Estimates Without Bootstrapping
  • With Bootstrapping
• StatisticalTester
  • Overview
  • Classes
    • StatTestResult
      • StatTestResult
    • StatisticalTester
      • StatisticalTester
  • Class Attributes
    • METRIC_LIST
  • Function Signatures
    • StatTestResult
    • StatisticalTester
      • StatisticalTester._bootstrap_test()
      • StatisticalTester.get_ci_bounds()
      • StatisticalTester.calc_p_value_bootstrap()
      • StatisticalTester._chi_square_test()
      • StatisticalTester.get_contingency_table()
      • StatisticalTester._calculate_effect_size()
      • StatisticalTester._adjust_p_values()
      • StatisticalTester.analyze_metrics()
      • StatisticalTester.adjusting_p_vals()
      • StatisticalTester._validate_config()
      • StatisticalTester.cohens_d()
      • StatisticalTester._analyze_single_metrics()
      • StatisticalTester._analyze_bootstrapped_metrics()
  • Usage Example

From Model to Prediction

• Preparing Model Outputs for Fairness Analysis
  • Step 1: Install and Import Dependencies
  • Step 2: Load the Dataset
  • Step 3: Clean the Data
  • Step 4: Encode the Target Variable
  • Step 5: Prepare Features and Labels
  • Step 6: Train-Test Split
  • Step 7: Train the XGBoost Model
  • Step 8: Generate Predictions and Evaluate
  • Step 9: Extract Model Outputs

Mathematical Framework

• Calibration Curves and Area Under the Curve
  • Calibration Curves and Area Interpretation
    • Example 1: Calibration with y = x²
    • Example 2: Calibration with y = x² + 4x
• Regression Residuals
• Chi-Square Tests and Cochran’s Rule
  • Chi-Square Statistic
  • Cochran’s Rule
    • Worked Example: Sparse K x 2 Table
    • Reference

iPython Notebooks

• Problem Types
  • Binary Classification
  • Regression

About EquiBoots

• Acknowledgements
• Contributors
• Citing EquiBoots
• Changelog
  • Version 0.0.1a13
  • Version 0.0.1a12
  • Version 0.0.1a11
  • Version 0.0.1a10
  • Version 0.0.1a9
  • Version 0.0.1a8
  • Version 0.0.1a7
  • Version 0.0.1a6
  • Version 0.0.1a5
  • Version 0.0.1a4
  • Version 0.0.1a3
  • Version 0.0.1a2
  • Version 0.0.1a1
  • Version 0.0.1a
  • Version 0.0.0a10
    • Add histogram-only mode to calibration curves (plot_hist flag)
    • Handle Seaborn < 0.13.2 legend errors for boxplot/violinplot
    • Add lowess_kwargs support & show LOWESS AUC in legend
  • Version 0.0.0a9
  • What’s Changed
  • Version 0.0.0a8
  • Version 0.0.0a7
  • Version 0.0.0a6
  • Version 0.0.0a5
    • New Contributor
  • Version 0.0.0a4
    • New Contributor
  • Version 0.0.0a3
  • Version 0.0.0a2
  • Version 0.0.0a1
    • New Contributors
• References