eval_eq_odds {fairmetrics} | R Documentation |
Examine Equalized Odds of a Predictive Model
Description
This function evaluates whether a predictive model satisfies the Equalized Odds criterion by comparing both False Negative Rates (FNR) and False Positive Rates (FPR) across two groups defined by a binary sensitive attribute. It reports the rate for each group, their differences, ratios, and bootstrap-based confidence regions. A Bonferroni-corrected union test is used to test whether the model violates the Equalized Odds criterion.
Usage
eval_eq_odds(
data,
outcome,
group,
probs,
cutoff = 0.5,
confint = TRUE,
bootstraps = 2500,
alpha = 0.05,
digits = 2,
message = TRUE
)
Arguments
data |
A data frame containing the true binary outcomes, predicted probabilities, and sensitive group membership. |
outcome |
A string specifying the name of the binary outcome variable in
|
group |
A string specifying the name of the binary sensitive attribute variable (e.g., race, gender) used to define the comparison groups. |
probs |
A string specifying the name of the variable containing predicted probabilities or risk scores. |
cutoff |
A numeric value used to threshold predicted probabilities into binary predictions; defaults to 0.5. |
confint |
Whether to compute 95% confidence interval, default is TRUE. |
bootstraps |
An integer specifying the number of bootstrap resamples for constructing confidence intervals; vdefaults to 2500. |
alpha |
Significance level for the (1 - |
digits |
Number of decimal places to round numeric results; defaults to 2. |
message |
Logical; if TRUE (default), prints a textual summary of the
fairness evaluation. Only works if |
Value
A data frame summarizing group disparities in both FNR and FPR with the following columns:
-
Metric
: The reported metrics ("FNR; FPR"). -
Group1
: Estimated FNR and FPR for the first group. -
Group2
: Estimated FNR and FPR for the second group. -
Difference
: Differences in FNR and FPR, computed as Group1 - Group2. -
95% CR
: Bonferroni-adjusted confidence regions for the differences. -
Ratio
: Ratios in FNR and FPR, computed as Group1 / Group2. -
95% CR
: Bonferroni-adjusted confidence regions for the ratios.
Examples
library(fairmetrics)
library(dplyr)
library(magrittr)
library(randomForest)
data("mimic_preprocessed")
set.seed(123)
train_data <- mimic_preprocessed %>%
dplyr::filter(dplyr::row_number() <= 700)
# Fit a random forest model
rf_model <- randomForest::randomForest(factor(day_28_flg) ~ .,
data = train_data, ntree = 1000
)
# Test the model on the remaining data
test_data <- mimic_preprocessed %>%
dplyr::mutate(gender = ifelse(gender_num == 1, "Male", "Female")) %>%
dplyr::filter(dplyr::row_number() > 700)
test_data$pred <- predict(rf_model, newdata = test_data, type = "prob")[, 2]
# Fairness evaluation
# We will use sex as the sensitive attribute and day_28_flg as the outcome.
# We choose threshold = 0.41 so that the overall FPR is around 5%.
# Evaluate Equalized Odds
eval_eq_odds(
data = test_data,
outcome = "day_28_flg",
group = "gender",
probs = "pred",
cutoff = 0.41
)