cac_lee {irtQ} | R Documentation |
Classification Accuracy and Consistency Using Lee's (2010) Approach
Description
This function computes classification accuracy and consistency indices for complex assessments based on the method proposed by Lee (2010). This function supports both dichotomous and polytomous item response theory (IRT) models.
Usage
cac_lee(x, cutscore, theta = NULL, weights = NULL, D = 1, cut.obs = TRUE)
Arguments
x |
A data frame containing item metadata (e.g., item parameters, number
of categories, IRT model types, etc.). See |
cutscore |
A numeric vector specifying the cut scores for classification. Cut scores are the points that separate different performance categories (e.g., pass vs. fail, or different grades). |
theta |
A numeric vector of ability estimates. Ability estimates (theta
values) are the individual proficiency estimates obtained from the IRT
model. The theta parameter is optional and can be |
weights |
An optional two-column data frame or matrix where the first column is the quadrature points (nodes) and the second column is the corresponding weights. This is typically used in quadrature-based IRT analysis. |
D |
A scaling constant used in IRT models to make the logistic function closely approximate the normal ogive function. A value of 1.7 is commonly used for this purpose. Default is 1. |
cut.obs |
Logical. If |
Details
This function first validates the input arguments. If both theta
and weights
are NULL
, the function will stop and return an error message. Either theta
(a vector of ability estimates) or weights
(a quadrature-based weight matrix)
must be specified.
If cut.obs = FALSE
, the provided cut scores are assumed to be on the theta (ability)
scale, and they are internally converted to the observed summed score scale using the
test characteristic curve (TCC). This transformation allows classification to be carried
out on the summed score metric, even if theta-based cut points are provided.
When weights
are provided (D method), the function uses the Lord-Wingersky recursive
algorithm to compute the conditional distribution of observed total scores at each node.
These conditional distributions are used to compute:
the probability of being classified into each performance level,
conditional classification accuracy (probability of correct classification), and
conditional classification consistency (probability of being assigned to the same level upon repeated testing).
When theta
values are provided instead (P method), the same logic applies, but using
an empirical distribution of examinees instead of quadrature-based integration.
In this case, uniform weights are assigned to all examinees.
Finally, marginal classification accuracy and consistency are computed as weighted averages of the conditional statistics across the ability distribution.
Value
A list containing the following elements:
confusion: A confusion matrix showing the cross table between true and expected levels.
marginal: A data frame showing the marginal classification accuracy and consistency indices.
conditional: A data frame showing the conditional classification accuracy and consistency indices.
prob.level: A data frame showing the probability of being assigned to each level category.
cutscore: A numeric vector showing the cut scores used in the analysis.
Author(s)
Hwanggyu Lim hglim83@gmail.com
References
Lee, W. C. (2010). Classification consistency and accuracy for complex assessments using item response theory. Journal of Educational Measurement, 47(1), 1-17.
See Also
gen.weight()
, est_score()
, cac_rud()
Examples
## --------------------------------------------------------------------------
## 1. When the empirical ability distribution of the population is available
## (D method)
## --------------------------------------------------------------------------
# Import the "-prm.txt" output file from flexMIRT
flex_prm <- system.file("extdata", "flexmirt_sample-prm.txt", package = "irtQ")
# Read item parameter data and convert it to item metadata
x <- bring.flexmirt(file = flex_prm, "par")$Group1$full_df
# Set the cut scores on the observed summed score scale
cutscore <- c(10, 20, 30, 50)
# Create a data frame containing the quadrature points and corresponding weights
node <- seq(-4, 4, 0.25)
weights <- gen.weight(dist = "norm", mu = 0, sigma = 1, theta = node)
# Calculate classification accuracy and consistency
cac_1 <- cac_lee(x = x, cutscore = cutscore, weights = weights, D = 1)
print(cac_1)
## -------------------------------------------------------------
## 2. When individual ability estimates are available (P method)
## -------------------------------------------------------------
# Randomly draw true ability values from N(0, 1)
set.seed(12)
theta <- rnorm(n = 1000, mean = 0, sd = 1)
# Simulate item response data
data <- simdat(x = x, theta = theta, D = 1)
# Estimate ability parameters using maximum likelihood (ML)
est_th <- est_score(
x = x, data = data, D = 1, method = "ML",
range = c(-4, 4), se = FALSE
)$est.theta
# Calculate classification accuracy and consistency
cac_2 <- cac_lee(x = x, cutscore = cutscore, theta = est_th, D = 1)
print(cac_2)
## ---------------------------------------------------------
## 3. When individual ability estimates are available,
## but cut scores are specified on the IRT theta scale
## ---------------------------------------------------------
# Set the cut scores on the theta scale
cutscore <- c(-2, -0.4, 0.2, 1.0)
# Calculate classification accuracy and consistency
cac_3 <- cac_lee(
x = x, cutscore = cutscore, theta = est_th, D = 1,
cut.obs = FALSE
)
print(cac_3)