eval_bs_parity {fairmetrics} | R Documentation |
Examine Brier Score Parity of a Model
Description
This function evaluates Brier Score Parity, a fairness measure that checks whether the Brier score (a measure of the calibration of probabilistic predictions) is similar across different groups. Brier score parity ensures that the model's predicted probabilities are equally well calibrated across subpopulations.
Usage
eval_bs_parity(
data,
outcome,
group,
probs,
confint = TRUE,
alpha = 0.05,
bootstraps = 2500,
digits = 2,
message = TRUE
)
Arguments
data |
Data frame containing the outcome, predicted outcome, and sensitive attribute |
outcome |
Name of the outcome variable |
group |
Name of the sensitive attribute |
probs |
Predicted probabilities |
confint |
Logical indicating whether to calculate confidence intervals |
alpha |
The 1 - significance level for the confidence interval, default is 0.05 |
bootstraps |
Number of bootstraps to use for confidence intervals |
digits |
Number of digits to round the results to, default is 2 |
message |
Logical; if TRUE (default), prints a textual summary of the
fairness evaluation. Only works if |
Value
A list containing the following elements:
Brier Score for Group 1
Brier Score for Group 2
Difference in Brier Score
Ratio in Brier Score If confidence intervals are computed (
confint = TRUE
):A vector of length 2 containing the lower and upper bounds of the 95% confidence interval for the difference in Brier Score
A vector of length 2 containing the lower and upper bounds of the 95% confidence interval for the ratio in Brier Score
See Also
eval_acc_parity
, eval_cond_acc_equality
, eval_pos_pred_parity
, eval_neg_pred_parity
Examples
library(fairmetrics)
library(dplyr)
library(magrittr)
library(randomForest)
data("mimic_preprocessed")
set.seed(123)
train_data <- mimic_preprocessed %>%
dplyr::filter(dplyr::row_number() <= 700)
# Fit a random forest model
rf_model <- randomForest::randomForest(factor(day_28_flg) ~ ., data = train_data, ntree = 1000)
# Test the model on the remaining data
test_data <- mimic_preprocessed %>%
dplyr::mutate(gender = ifelse(gender_num == 1, "Male", "Female")) %>%
dplyr::filter(dplyr::row_number() > 700)
test_data$pred <- predict(rf_model, newdata = test_data, type = "prob")[, 2]
# Fairness evaluation
# We will use sex as the sensitive attribute and day_28_flg as the outcome.
# Evaluate Brier Score Parity
eval_bs_parity(
data = test_data,
outcome = "day_28_flg",
group = "gender",
probs = "pred"
)