The Feasibility of National Inference Under the NSCAW IV L-State Sample Design

The purpose of this Feasibility Analysis Study (FAS) was to evaluate methods for producing valid national estimates under the National Survey of Child and Adolescent Well-being (NSCAW) IV sample design. The NSCAW IV design is a stratified two-stage, unequal probability design that restricts the sample to within L purposively selected states, where L is a number between 8 and 12. (L = 10 was assumed for most the analyses in this document). The design is fully consistent with the NSCAW IV’s primary objective (i.e., to provide detailed, precise state-level estimates for the L key states selected for the survey). However, an important secondary objective is to provide for valid national estimates, even though the sample design itself is not nationally representative. The FAS was conducted to determine whether the NSCAW IV L-state design could achieve this secondary objective. FAS confirmed that national estimates are indeed feasible. This report provides a summary of the FAS results and offers detailed, statistical evidence as to the NSCAW IV L-state design’s ability to produce valid national inferences. State estimates can be produced from the NSCAW IV design by applying classical survey estimation techniques; however, national estimation requires nonprobability or so-called quasi-random estimation and modeling techniques. The FAS investigated two such estimation approaches: quasi-randomization weighting (QRW) and superpopulation modeling (SPM). Data from prior NSCAW cohorts—NSCAW I and II—were analyzed to simulate the NSCAW IV design. Fortunately, the NSCAW I and II designs are quite similar to what is proposed for NSCAW IV in that they both include a purposive selection of states—L = 8, in total. The key difference between their designs and NSCAW IV’s design is that the two prior surveys also included a random sample the “remainder” stratum (i.e., the combination of states that were not purposively selected). This remainder stratum provided for the unbiased national estimation in NSCAW I and II without the needed for nonprobability estimation approaches. However, the remainder stratum was initially ignored in application of the QRW and SPM to NSCAW I and II states to simulate the NSCAW IV design. Later in the analysis, the remainder stratum provided a mechanism to assessing the biases in the QRW and SPM national estimates. This was critical in order for the FAS analysis speculate on the potential bias risks for NSCAW IV national estimates using the sample estimation approaches. As the results in the report convincingly show, both the QRW and SPM approaches are capable of producing valid national estimates from the L-state design envisioned for NSCAW IV. QRW is recommended for the estimating the means of the vast majority of questionnaire characteristics; however, in situations where QRW may fail to produce accurate results, the SPM approach can then be implemented to produce more accurate results. One drawback of SPM is that it requires more effort and a higher level of statistical expertise to implement. The research reported in this document initially addresses the question of whether the QRW method can be feasibly applied to the NSCAW IV data to obtain valid national inferences based on a small number (8 ≤ L ≤ 12) of purposively selected sample of states. Variations of QRW were applied to the NSCAW I eight-state sample and then, using the combination of the state and remainder stratum samples, the quality of the QRW national estimates was evaluated by comparing them to the full sample, unbiased national estimates. A number of key quality indicators were analyzed for national estimates related to 26 NSCAW characteristics—23 variables that overlap with the NCANDS data and three Child Behavior Check List (CBCL) characteristics from the survey questionnaire. Four types of estimates based solely on the eight-state data were computed: (1) unweighted estimates, (2) estimates based on the unadjusted NSCAW weights, (3) estimates derived from Method 1 weighting, and (4) estimates derived from Method 2 weighting. Estimates (1) and (2) were computed primarily to gain insight around the degree to which approaches (3) and (4) affect the bias of the unweighted and unadjusted eight-state estimates. Finally, the NSCAW I full sample estimates were computed, which provided the “gold-standard” estimates used to evaluate the quality of estimates (1) through (4) for national inference. Because the NSCAW IV design will use three domains—the eight NSCAW I domains were collapsed to create the three NSCAW IV domains and the quality analysis was repeated for these “ad hoc” domains. The NSCAW I QRW analyses demonstrated the success of these methods for extend inference from the NSCAW IV purposively selected state sample to the national level. Although some bias could remain in the estimates after weighting, the biases were significantly reduced from their unweighted counterparts for the vast majority of estimates. For substantive characteristics such as the CBCL variables, the biases in the QRW estimates were slightly larger but still acceptable for many research purposes. The results for the NSCAW I analysis were essentially confirmed using NSCAW II data: both data sets produced very similar results.SPM offers an alternative to QRW for creating national estimates from the L-state design. Because it incorporates specific covariates that are the most highly correlated with the outcome variable, it can achieve much greater estimation accuracy than is possible with the more general QRW methodology. A key distinction between SPM and the QRW is that, although the latter can be applied to any outcome variable, SPM must be tailored to each specific outcome variable of interest; i.e., each questionnaire variable represents a dependent variable in its own customized model. Nevertheless, for questionnaire variables where the QRW estimates have, or are suspected to have, unacceptably large biases, SPM offers a less biased alternative and can be used to produce higher quality inferences. The SPM analyses used the eight-state strata to project national estimates and then evaluated those estimates by comparing them to the corresponding national sample estimates that served as the gold standard. The results showed that, at least for the variables considered from NSCAW I and II, SPM produces excellent results, with very small biases in most instances. Based on these results, we believe the SPM approach is quite feasible for the NSCAW IV L-state design. Thus, its use is recommended for NSCAW IV, particularly in situations where QRW estimates are shown or suspected to be unacceptably biased.