Linear Regression
Least squares, its geometry, and regularized regression.
Fitting a straight line (or plane) to data by least squares: the formulas, the geometry behind them, what goes wrong when assumptions fail, and the regularized and logistic variants.
Simple linear regression
Model – Observe pairs and assume depends linearly on one input plus random noise with mean zero. The intercept and slope are unknown.
Least squares – Choose to minimize the residual sum of squares (RSS), the total squared vertical distance from the points to the line. Each is a residual.
Notation – With means , define the sums of squares and cross-products
Sample correlation – , a number in measuring how close the points are to a line.
Estimates – Setting the derivatives of RSS to zero gives
where are the sample standard deviations. The fitted line always passes through .
Residual error – At the optimum,
and is an unbiased estimate of the noise variance.
Regressing on gives slope , which is not unless the points lie exactly on a line.
Multiple regression (ordinary least squares)
Model – With inputs, stack the data into a vector and a design matrix whose row holds the inputs for observation (include a column of ones for an intercept):
Ordinary least squares (OLS) – Minimize the squared length of the residual vector, . Setting the gradient to zero gives the normal equations , so when the columns of are linearly independent
Numerically, solve with a QR or singular value decomposition rather than inverting , which squares the sensitivity to rounding error.
Hat matrix – The fitted values are with
projects any vector onto the column space of , the set of all vectors of the form .
Residual maker – projects onto the directions perpendicular to that space, and produces the residuals: .
Properties – and are both symmetric () and idempotent (: projecting twice changes nothing). Also and , so the residuals depend only on the noise:
, so the residuals are uncorrelated with every input. Least squares is the orthogonal projection of onto the space the inputs can reach.
Variance of the estimate – If the noise has variance , , with estimated by .
Gauss–Markov theorem – If the errors have mean zero, constant variance and are uncorrelated, OLS has the smallest variance among all unbiased estimators that are linear in (it is the "best linear unbiased estimator", BLUE). If the errors are also normal, OLS equals the maximum likelihood estimate.
Goodness of fit
Decomposition of variation – With an intercept, the total sum of squares splits into an explained part and the residual part: .
Coefficient of determination – The fraction of variation explained:
For simple linear regression, .
Adjusted – never decreases when an input is added, even a useless one. The adjusted version penalizes the number of inputs :
When the assumptions fail
| Problem | What goes wrong | Usual fix |
|---|---|---|
| Omitted variable that is correlated with an included input | is biased | include it, or use instrumental variables |
| Highly correlated inputs (multicollinearity) | is unstable, with large variance | ridge regression, or drop or combine inputs |
| Noise variance changes with (heteroscedasticity) | standard errors are wrong | robust (White) standard errors, weighted least squares |
| Errors correlated over time | standard errors are wrong | Newey–West standard errors, generalized least squares |
| Noise in the inputs themselves | is shrunk toward zero (attenuation) | instrumental variables, errors-in-variables models |
Regularization
Adding a penalty on the size of trades a little bias for lower variance, with strength set by .
| Method | Minimizes | Effect |
|---|---|---|
| Ridge | closed form ; shrinks all coefficients and stabilizes correlated inputs | |
| Lasso | no closed form; sets some coefficients exactly to zero | |
| Elastic net | both penalties | sparse, but keeps groups of correlated inputs together |
Here and .
Logistic regression
Model – For a yes/no outcome , model the probability of "yes" by passing a linear score through the sigmoid , which maps any number into :
Fitting – Maximize the likelihood, equivalently minimize the negative log-likelihood, which is convex (bowl-shaped, so it has a single minimum). Its gradient is
There is no closed form, so it is solved with gradient descent or Newton's method (iteratively reweighted least squares).
If a line separates the two classes perfectly, the coefficients grow without bound unless a penalty is added.