reval_mst {irtQ} | R Documentation |
Recursion-based MST evaluation method
Description
This function evaluates the measurement precision and bias in Multistage-adaptive Test (MST) panels using a recursion-based evaluation method introduced by Lim et al. (2020). This function computes conditional biases and standard errors of measurement (CSEMs) across a range of IRT ability levels, facilitating efficient and accurate MST panel assessments without extensive simulations.
Usage
reval_mst(
x,
D = 1,
route_map,
module,
cut_score,
theta = seq(-5, 5, 1),
intpol = TRUE,
range.tcc = c(-7, 7),
tol = 1e-04
)
Arguments
x |
A data frame containing the metadata for the item bank, which
includes important information for each item such as the number of score
categories and the IRT model applied. This metadata is essential for
evaluating the MST panel, with items selected based on the specifications
in the |
D |
A scaling constant used in IRT models to make the logistic function closely approximate the normal ogive function. A value of 1.7 is commonly used for this purpose. Default is 1. |
route_map |
A binary square matrix that defines the MST structure,
illustrating transitions between modules and stages. This concept and
structure are inspired by the |
module |
A binary matrix that maps items from the item bank specified in
|
cut_score |
A list defining cut scores for routing test takers through
MST stages. Each list element is a vector of cut scores for advancing
participants to subsequent stage modules. In a 1-3-3 MST configuration, for
example, |
theta |
A vector of ability levels (theta) at which the MST panel's
performance is assessed. This allows for the evaluation of measurement
precision and bias across a continuum of ability levels. The default range
is |
intpol |
A logical value to enable linear interpolation in the inverse
test characteristic curve (TCC) scoring, facilitating ability estimate
approximation for observed sum scores not directly obtainable from the TCC,
such as those below the sum of item guessing parameters. Default is TRUE,
applying interpolation to bridge gaps in the TCC. Refer to
|
range.tcc |
A vector to define the range of ability estimates for inverse TCC scoring, expressed as the two numeric values for lower and upper bounds. Default is to c(-7, 7). |
tol |
A numeric value of the convergent tolerance for the inverse TCC scoring. For the inverse TCC scoring, the bisection method is used for optimization. Default is 1e-4. |
Details
The reval_mst()
function evaluates an MST panel by
implementing a recursion-based method to assess measurement precision
across IRT ability levels. This approach, detailed in Lim et al. (2020),
enables the computation of conditional biases and CSEMs efficiently,
bypassing the need for extensive simulations traditionally required for MST
evaluation.
The module
argument, used in conjunction with the item bank metadata x
,
systematically organizes items into modules for MST panel evaluation. Each
row of x
corresponds to an item, detailing its characteristics like score
categories and IRT model. The module
matrix, structured with the same
number of rows as x
and columns representing modules, indicates item
assignments with 1s. This precise mapping enables the reval_mst()
function to evaluate the MST panel's performance by analyzing how items
within each module contribute to measurement precision and bias, reflecting
the tailored progression logic inherent in MST designs.
The route_map
argument is essential for defining the MST's structure by
indicating possible transitions between modules. Similar to the
transMatrix()
in the mstR package (Magis et al., 2017), route_map
is a binary matrix that outlines which module transitions are possible
within an MST design. Each "1" in the matrix represents a feasible
transition from one module (row) to another (column), effectively mapping
the flow of test takers through the MST based on their performance. For
instance, a "1" at the intersection of row i and column j indicates the
possibility for test takers to progress from the module corresponding to
row i directly to the module denoted by column j. This structure allows
reval_mst()
to simulate and evaluate the dynamic routing of test
takers through various stages and modules of the MST panel.
To further detail the cut_score
argument with an illustration: In a 1-3-3
MST configuration, the list cut_score = list(c(-0.5, 0.5), c(-0.6, 0.6))
operates as a decision guide at each stage. Initially, all test takers
start in the first module. Upon completion, their scores determine their
next stage module: scores below -0.5 route to the first module of the next
stage, between -0.5 and 0.5 to the second, and above 0.5 to the third. This
pattern allows for dynamic adaptation, tailoring the test path to
individual performance levels.
Value
This function returns a list of seven internal objects. The four objects are:
panel.info |
A list of several sub-objects containing detailed information about the MST panel configuration, including:
|
item.by.mod |
A list where each entry represents a module in the MST panel, detailing the item metadata within that module. Each module's metadata includes item IDs, the number of categories, the IRT model used (model), and the item parameters (e.g., par.1, par.2, par.3). |
item.by.path |
A list containing item metadata arranged according to the paths through the MST structure. This detailed breakdown allows for an analysis of item characteristics along specific MST paths. Each list entry corresponds to a testing stage and path, providing item metadata. This structure facilitates the examination of how items function within the context of each unique path through the MST. |
eq.theta |
Estimated ability levels (
|
cdist.by.mod |
A list where each entry contains the conditional distributions of the observed scores for each module given the ability levels. |
jdist.by.path |
Joint distributions of observed scores for different paths
at each stage in a MST panel. The example below outlines the organization of
|
eval.tb |
A table summarizing the measurement precision of the MST panel. It contains the true
ability levels ( |
Author(s)
Hwanggyu Lim hglim83@gmail.com
References
Magis, D., Yan, D., & Von Davier, A. A. (2017). Computerized adaptive and multistage testing with R: Using packages catR and mstR. Springer.
Lim, H., Davey, T., & Wells, C. S. (2020). A recursion-based analytical approach to evaluate the performance of MST. Journal of Educational Measurement, 58(2), 154-178.
See Also
Examples
## ------------------------------------------------------------------------------
# Evaluation of a 1-3-3 MST panel using simMST data.
# This simulation dataset was utilized in Lim et al.'s (2020) simulation study.
# Details:
# (a) Panel configuration: 1-3-3 MST panel
# (b) Test length: 24 items (each module contains 8 items across all stages)
# (c) IRT model: 3-parameter logistic model (3PLM)
## ------------------------------------------------------------------------------
# Load the necessary library
library(dplyr)
library(tidyr)
library(ggplot2)
# Import item bank metadata
x <- simMST$item_bank
# Import module information
module <- simMST$module
# Import routing map
route_map <- simMST$route_map
# Import cut scores for routing to subsequent modules
cut_score <- simMST$cut_score
# Import ability levels (theta) for evaluating measurement precision
theta <- simMST$theta
# Evaluate MST panel using the reval_mst() function
eval <-
reval_mst(x,
D = 1.702, route_map = route_map, module = module,
cut_score = cut_score, theta = theta, range.tcc = c(-5, 5)
)
# Review evaluation results
# The evaluation result table below includes conditional biases and
# standard errors of measurement (CSEMs) across ability levels
print(eval$eval.tb)
# Generate plots for biases and CSEMs
p_eval <-
eval$eval.tb %>%
dplyr::select(theta, bias, csem) %>%
tidyr::pivot_longer(
cols = c(bias, csem),
names_to = "criterion", values_to = "value"
) %>%
ggplot2::ggplot(mapping = ggplot2::aes(x = theta, y = value)) +
ggplot2::geom_point(mapping = ggplot2::aes(shape = criterion), size = 3) +
ggplot2::geom_line(
mapping = ggplot2::aes(
color = criterion,
linetype = criterion
),
linewidth = 1.5
) +
ggplot2::labs(x = expression(theta), y = NULL) +
ggplot2::theme_classic() +
ggplot2::theme_bw() +
ggplot2::theme(legend.key.width = unit(1.5, "cm"))
print(p_eval)