Variable selection procedures under collinearity (multicollinearity)

Variable selection is an important area of statistical modeling, which is still an active area of research. In this study, we investigated the performance of four variable selection procedures: backward elimination, Bayesian, least absolute shrinkage selection operator (lasso), and ridge selection operator under collinearity. Specifically, we focused on the variable selection accuracy, prediction accuracy, and their ability to mitigate the impact of collinearity on variable selection and predictive accuracy. What we found is that sample size, and the strength of the regression coefficient impacted the selection accuracy specifically in the standard linear regression simulation, where the simulation results showed that as the sample size decreased, the selection accuracy decreased specifically for methods such as backward elimination, Bayesian, and the lasso, while that of the ridge selection operator the variable selection accuracy increased. In the high dimensional simulation, the lasso outperformed the other variable selection methods, indicating its unique features to select sparse models even when the number of parameters exceeds the sample size. However, the estimates were extremely biased for all methods investigated in the high dimensional regression simulation. Though the computational times were not reported, the ridge selection operator was the slowest, especially for large sample sizes or number of predictors. This indicates its unfeasibility in the practical world where sample sizes or number of predictors are large since it is too computationally costly. In terms of prediction, overall, Bayesian g-prior variable selection and the lasso-fitted models performed better than those from the other variable selection methods. In the last section we discuss the limitations and provide recommendations on using these methods.