Pretest item calibration in multistage adaptive testing

The purpose of this study was to compare calibration and linking methods for placing pretest item parameter estimates on the item pool scale in MST designs. The panel design was 1-3 with 40 items in total. The module difficulty and distribution of information across stages were also not manipulated. Cut scores for routing were determined by AMI. Provisional θ was estimated by EAP with the prior distribution set to N(0,1). Manipulated conditions related to MST administration were module length (20-20 and 10-30), the placement of pretest items (ES and EI), and number of pretest items (12-pre and 20-pre). Under the ES model, all pretest items were administered in a separate module. With the EI model, pretest items were distributed across modules. Manipulated conditions related to linking were sample size set to 1500 and 3000, and examinee distribution parameters set to N(.0,1), N(.5,1), N(-.5,1), and N(.5,.64). These conditions were fully crossed and resulted in 64 study conditions. The IRT model was 3PL, and calibration methods were separate calibration (SC) and fixed calibration (FC) with three parallel approaches under each (FC-1 and SC-1; FC-2 and SC-2; and FC-3 and SC-3). Linking under FC was implemented by fixing operational item parameters. Linking under SC was realized by applying the STF (Stocking & Lord, 1983). For both FC and SC, the first approach used only operational items in the routing module (R) to link pretest items. The second approach also used only operational R items for linking, but in addition the operational items in 2E, 2M and 2D were freely estimated. The third approach used operational items in R, 2E, 2M and 2D to link pretest items. Evaluation criteria were the recovery of ICCs, and a, b and c parameters for the pretest items. Among the calibration approaches, the best performing method was the third approach (i.e., FC-3 and SC-3) consistent across all study conditions. For all three approaches, SC outperformed FC in all study conditions. For the ICC and b parameters, N(0,1), N(.5,1) and N(-.5,1) demonstrated similar recovery. Among the distributions, item recovery was the worst under N(.5,.64). The a recovery was the best and worst when the distribution parameters were N(.5,1) and N(-.5,1). The c recovery was the best and worst when the distribution parameters were N(-.5,1) and N(.5,.64), respectively. For all FC and SC approaches, increasing the sample size from 1500 to 3000 improved the recovery of generating parameters. Between 12-pre and 20-pre, there was not a consistent finding indicating which condition was better at item parameter recovery. Depending on other factors, the smaller or larger number of pretest items was better at parameter recovery. Under both the ES and EI model, item parameter recovery obtained from the 20-20 and 10-30 panels were similar. Recovery of the ICC, a, b, and c parameters was better under the ES model than the EI. At the item level, there were no differences for the pretest items administered in the routing module. For the pretest items administered at the second stage modules, however, the differences between the EI and ES models were conditional on the ability distribution