Comparing the Predictive Powers of Alternative Multiple Regression Models

Mean squared error of prediction is used as the criterion for determining which of two multiple regression models (not necessarily nested) is more predictive. We show that an unrestricted (or true) model with t parameters should be chosen over a restricted (or misspecified) model with m parameters if (P t 2−P m 2)>(1−P t 2)(t−m)/n, where P t 2 and P m 2 are the population coefficients of determination of the unrestricted and restricted models, respectively, and n is the sample size. The left-hand side of the above inequality represents the squared bias in prediction by using the restricted model, and the right-hand side gives the reduction in variance of prediction error by using the restricted model. Thus, model choice amounts to the classical statistical tradeoff of bias against variance. In practical applications, we recommend that P 2 be estimated by adjusted R 2. Our recommendation is equivalent to performing the F-test for model comparison, and using a critical value of 2−(m/n); that is, if F>2−(m/n), the unrestricted model is recommended; otherwise, the restricted model is recommended.