A Heuristically Penalized Framework for Asymptotic Ridge Estimation

Abstract The presence of multicollinearity occurs frequently in high-dimensional datasets. Multicollinearity leads to unstable estimates because it tends to increase the variance of regression coefficients. With the advancement of computers, much more complex high-dimensional datasets are being produced daily, which shows the severe need for the development of current approaches. In this paper, we extend the concept of the ridge approach through its asymptotic cases by proposing a heuristic combined algorithm based on the asymptotic elastic-net penalty function. Two new theorems supported the combined algorithm. The first theorem illustrates the new asymptotic property of the penalty function with respect to the classical one. The second theorem shows the difference in model accuracy between the classical penalized function and the asymptotic one. As a result, based on these new theorems, a bunch of heuristic ridges are introduced that we call asymptotic ridges. The asymptotic ridge is produced in two types: the models based on the individual points of the defined sequence and the models based on the average points of the defined sequence. It is found that, in comparison to the classical ridge, the asymptotic approach frequently results in improved regularization. As an asymptotic ridge uses dynamic hyperparameters, the mean square errors of the fitted model are often smaller than a classical penalty function. Herein, the performances of the introduced approaches are reported on simulated and real datasets. The datasets, include a comprehensive analysis of the behavior of the generalized cross-validation function.