cac_rud {irtQ} | R Documentation |
Classification Accuracy and Consistency Based on Rudner's (2001, 2005) Approach
Description
This function computes classification accuracy and consistency indices using the method proposed by Rudner in 2001 and 2005. This function supports both scenarios: when the empirical ability distribution of the population is available, and when individual ability estimates are used.
Usage
cac_rud(cutscore, theta = NULL, se, weights = NULL)
Arguments
cutscore |
A numeric vector specifying the cut scores for classification. Cut scores are the points that separate different performance categories (e.g., pass vs. fail, or different grades). |
theta |
A numeric vector of ability estimates. Ability estimates (theta
values) are the individual proficiency estimates obtained from the IRT
model. The theta parameter is optional and can be |
se |
A numeric vector of the same length as |
weights |
An optional two-column data frame or matrix where the first column is the quadrature points (nodes) and the second column is the corresponding weights. This is typically used in quadrature-based IRT analysis. |
Details
This function first validates the input arguments. If both theta
and weights
are NULL
, the function will stop and return an error message.
Either theta
or weights
must be specified. In addition, se
must be
provided and must match the length of theta
or the number of quadrature
points in weights
.
It then computes the probability that an examinee with a given ability is
classified into each performance level using the normal distribution function
centered at each theta
(or quadrature point) with standard deviation se
.
These probabilities are used to calculate conditional classification accuracy
(the probability of being correctly classified) and conditional classification
consistency (the probability of being consistently classified upon repeated
testing) for each ability value.
Finally, the function computes marginal classification accuracy and consistency across all examinees by aggregating the conditional indices with the associated weights.
Value
A list containing the following elements:
confusion: A confusion matrix showing the cross table between true and expected levels.
marginal: A data frame showing the marginal classification accuracy and consistency indices.
conditional: A data frame showing the conditional classification accuracy and consistency indices.
prob.level: A data frame showing the probability of being assigned to each level category.
cutscore: A numeric vector showing the cut scores used in the analysis.
Author(s)
Hwanggyu Lim hglim83@gmail.com
References
Rudner, L. M. (2001). Computing the expected proportions of misclassified examinees. Practical Assessment, Research, and Evaluation, 7(1), 14.
Rudner, L. M. (2005). Expected classification accuracy. Practical Assessment, Research, and Evaluation, 10(1), 13.
See Also
gen.weight()
, est_score()
, cac_lee()
Examples
## -------------------------------------------
# 1. Using the empirical ability distribution
## -------------------------------------------
# Import the "-prm.txt" output file from flexMIRT
flex_prm <- system.file("extdata", "flexmirt_sample-prm.txt", package = "irtQ")
# Read item parameter estimates and convert them into item metadata
x <- bring.flexmirt(file = flex_prm, "par")$Group1$full_df
# Define cut scores on the theta scale
cutscore <- c(-2, -0.5, 0.8)
# Create quadrature points and corresponding weights
node <- seq(-4, 4, 0.25)
weights <- gen.weight(dist = "norm", mu = 0, sigma = 1, theta = node)
# Compute conditional standard errors across quadrature points
tif <- info(x = x, theta = node, D = 1, tif = TRUE)$tif
se <- 1 / sqrt(tif)
# Compute classification accuracy and consistency
cac_1 <- cac_rud(cutscore = cutscore, se = se, weights = weights)
print(cac_1)
## -----------------------------------------
# 2. Using individual ability estimates
## -----------------------------------------
# Generate true abilities from N(0, 1)
set.seed(12)
theta <- rnorm(n = 1000, mean = 0, sd = 1)
# Simulate item response data
data <- simdat(x = x, theta = theta, D = 1)
# Estimate ability and standard errors using ML estimation
est_theta <- est_score(
x = x, data = data, D = 1, method = "ML",
range = c(-4, 4), se = TRUE
)
theta_hat <- est_theta$est.theta
se <- est_theta$se.theta
# Compute classification accuracy and consistency
cac_2 <- cac_rud(cutscore = cutscore, theta = theta_hat, se = se)
print(cac_2)