Honey, I Shrunk the Irrelevant Effects! Simple and Fast Approximate Bayesian Regularization

Statistical models are becoming increasingly complex with more parameters to explain complex dependency structures among larger sets of variables. Regularization techniques (such as penalized regression) are ideal to identify the most important parameters by shrinking negligible effects to zero. The resulting regularized solutions are parsimonious and often show good predictive performance. Currently however regularization techniques have mainly been developed for standard modeling designs even though regularization techniques are also very useful for more complex modeling designs. Moreover, even though Bayesian regularization algorithms are competitive (and sometimes superior) to their classical counterpart, classical regularization techniques (such as the lasso) are still most common in applied research. To address these shortcomings, the current paper presents a fast and flexible approximate Bayesian regularization procedure. A Gaussian approximation is used for the integrated likelihood of the (large) set of parameters which is then combined with a Bayesian shrinkage prior to obtain a parsimonious solution with many (approximately) zero estimates. The method is implemented in the R package ‘shrinkem’. The general applicability of the methodology is illustrated in various applications including linear regression models, relational event models, mediation models, factor analytic models, and Gaussian graphical models.