A geometrical interpretation of collinearity: a natural way to justify ridge regression and its anomalies

Abstract

Justifying ridge regression from a geometrical perspective is one of the main contributions of this paper. To the best of our knowledge, this question has not been treated previously. This paper shows that ridge regression is a particular case of the raising procedures that provides greater flexibility by transforming the matrix X associated with the model. Thus, the raising procedures, based on a geometrical idea of the vectorial space associated with the columns of matrix X, lead naturally to ridge regression and justify the presence of the well-known constant k on the main diagonal of matrix X^t X. This paper also analyzes and compares different alternatives to raising with respect to collinearity mitigation. The results are illustrated with an empirical application.