Model Tuner Documentation

Getting Started

• Welcome to Model Tuner’s Documentation!
  • What Does Model Tuner Offer?
  • Prerequisites
  • Installation

Usage Guide

• iPython Notebooks
  • Binary Classification Examples
  • Regression Example
• Input Parameters
  • Model
• Key Methods and Functionalities
  • __init__(...)
  • reset_estimator()
  • process_imbalance_sampler()
  • calibrateModel()
  • Get train, val, test data
    • get_train_data()
    • get_valid_data()
    • get_test_data()
  • calibrate_report()
  • fit()
  • return_metrics()
    • return_metrics()
  • predict()
  • grid_search_param_tuning()
  • print_selected_best_features()
  • tune_threshold_Fbeta()
  • train_val_test_split()
  • get_best_score_params()
  • conf_mat_class_kfold()
  • regression_report_kfold()
  • regression_report()
  • report_model_metrics()
  • find_optimal_threshold_beta()
• Helper Functions
  • kfold_split()
  • get_cross_validate()
  • _confusion_matrix_print()
  • print_pipeline()
• Pipeline Management
  • Specifying Pipeline Steps
  • Helper Methods for Pipeline Extraction
    • get_preprocessing_and_feature_selection_pipeline()
    • get_feature_selection_pipeline()
    • get_preprocessing_pipeline()
  • Extracting Feature names
    • get_feature_names()
  • Summary
• Binary Classification
  • AIDS Clinical Trials Group Study
    • Step 1: Import Necessary libraries
    • Step 2: Load the dataset, define X, y
    • Step 3: Check for zero-variance columns and drop accordingly
    • Step 4: Create an instance of the XGBClassifier
    • Step 5: Define Hyperparameters for XGBoost
    • Step 6: Initialize and configure the Model
    • Step 7: Perform grid search parameter tuning and retrieve split data
    • Step 8: Fit the model
    • Step 9: Return metrics (optional)
    • Step 10: Calibrate the model (if needed)
  • F1 Beta Threshold Tuning
    • Understanding F1 Beta Score
    • Example usage: default (beta = 1)
    • Example usage: custom betas (higher recall)
    • Example usage: custom betas (higher precision)
    • Optimizing Model Threshold for Precision/Recall Trade-off
  • Imbalanced Learning
    • Generating an imbalanced dataset
    • Define hyperparameters for XGBoost
    • Define the model object
    • Addressing Class Imbalance in Machine Learning
      • Techniques to Address Class Imbalance
      • Purpose of Using These Techniques
    • Synthetic Minority Oversampling Technique (SMOTE)
      • Step 1: Initalize and configure the model
      • Step 2: Perform grid search parameter tuning and retrieve split data
      • SMOTE: Distribution of y values after resampling
      • Step 3: Fit the model
      • Step 4: Return metrics (optional)
  • Recursive Feature Elimination (RFE)
    • Elastic Net for feature selection with RFE
  • SHAP (SHapley Additive exPlanations)
    • Step 1: Transform the test data using the feature selection pipeline
    • Step 2: Retrieve the trained XGBoost classifier from the pipeline
    • Step 3: Extract feature names from the training data, and initialize the SHAP explainer for the XGBoost classifier
    • Step 4: Compute SHAP values for the transformed test dataset and generate a summary plot of SHAP values
    • Step 5: Generate a summary plot of SHAP values
    • Feature Importance and Impact
• Multi-Class Classification
  • Iris Dataset with XGBoost
    • Step 1: Import Necessary Libraries
    • Step 2: Load the dataset. Define X, y
    • Step 3: Define the preprocessing steps
    • Step 4: Define the estimator and hyperparameters
    • Step 5: Initialize and configure the model
    • Step 6: Perform grid search parameter tuning
    • Step 7: Generate data splits
    • Step 8: Fit the model
    • Step 9: Return metrics (optional)
      • Report Model Metrics
    • Step 10: Predict probabilities and generate predictions
• Regression
  • California Housing with XGBoost
    • Step 1: Import necessary libraries
    • Step 2: Load the dataset
    • Step 3: Create an instance of the XGBRegressor
    • Step 4: Define Hyperparameters for XGBRegressor
    • Step 5: Initialize and configure the Model
    • Step 6: Perform grid search parameter tuning and retrieve split data
    • Step 7: Fit the model
    • Step 8: Return metrics (optional)
• Performance Evaluation Metrics
  • Using report_model_metrics()
  • Using return_metrics()
  • Threshold Tuning
    • How to Automatically Tune Thresholds
    • Using threshold in report_model_metrics()
    • Reporting Threshold in return_metrics
  • Classification report (optional)
• Bootstrap Metrics
  • check_input_type()
  • sampling_method()
  • evaluate_bootstrap_metrics()
  • return_bootstrap_metrics()
  • Bootstrap metrics example

Caveats

• Zero Variance Columns
  • Effects on Model Training
• Dependent Variable
  • Target Variable Shape and Its Effects
  • Solution
• Scaling Before Imputation
  • Accurate Imputation with Normalized Data
  • Consistency in Data Transformation
  • Prevention of Imputation Bias
  • Avoiding Distortions in Nearest Neighbor Imputation
  • Visual Proof
• Column Stratification with Cross-Validation
  • Cross-Validation and Stratification
• Model Calibration
  • Goal of Calibration
  • Calibration Curve
  • Brier Score
  • Platt Scaling
  • Isotonic Regression
  • Example: Calibration in Logistic Regression
  • Summary
• Using Imputation and Scaling in Pipeline Steps for Model Preprocessing
  • When Is Imputation and Feature Scaling in pipeline_steps Beneficial?
  • Models Not Benefiting From Imputation and Scaling in pipeline_steps
  • Why Doesn’t XGBoost Require Imputation and Scaling in pipeline_steps?
• Caveats in Imbalanced Learning
  • Bias from Class Distribution
  • Threshold-Dependent Predictions
  • Limitations of Accuracy
    • Integration and Practical Considerations
  • Impact of Resampling Techniques
    • SMOTE: A Mathematical Illustration
    • Example of Synthetic Sample Creation
    • Mitigating the Caveats
• Threshold Tuning Considerations
  • The Precision-Recall Tradeoff
  • Threshold Optimization
• ElasticNet Regularization
  • Important Considerations
• CatBoost Training Parameters
  • Example: Tuning hyperparameters for CatBoost

About Model Tuner

• GitHub Repository
• Acknowledgements
• Citing Model Tuner
• Changelog
  • Version 0.0.28b (Beta)
  • Version 0.0.27b (Beta)
  • Version 0.0.26b (Beta)
  • Version 0.0.25a
  • Version 0.0.24a
  • Version 0.0.23a
  • Version 0.0.22a
  • Version 0.0.21a
  • Version 0.0.20a
  • Version 0.0.19a
  • Version 0.0.18a
  • Version 0.0.17a
  • Version 0.0.16a
  • Version 0.0.15a
  • Version 0.0.014a
  • Version 0.0.013a
  • Version 0.0.012a
  • Version 0.0.011a
  • Version 0.0.010a
  • Version 0.0.09a
  • Version 0.0.08a
  • Version 0.0.07a
  • Version 0.0.06a
  • Version 0.0.05a
  • Version 0.0.02a
• References