A Connection Between Pattern Classification by Machine Learning and Statistical Inference With the General Linear Model

Abstract

A connection between the general linear model (GLM) with frequentist statistical testing and machine learning (MLE) inference is derived and illustrated. Initially, the estimation of GLM parameters is expressed as a Linear Regression Model (LRM) of an indicator matrix; that is, in terms of the inverse problem of regressing the observations. Both approaches, i.e. GLM and LRM, apply to different domains, the observation and the label domains, and are linked by a normalization value in the least-squares solution. Subsequently, we derive a more refined predictive statistical test: the linear Support Vector Machine (SVM), that maximizes the class margin of separation within a permutation analysis. This MLE-based inference employs a residual score and associated upper bound to compute a better estimation of the actual (real) error. Experimental results demonstrate how parameter estimations derived from each model result in different classification performance in the equivalent inverse problem. Moreover, using real data, the MLE-based inference including model-free estimators demonstrates an efficient trade-off between type I errors and statistical power.