Linear regression

DATASCI 415: Statistical Learning and Data Mining

University of Michigan

including slides by Gareth James, Daniela Witten, Trevor Hastie, Rob Tibshirani, Jonathan Taylor

Linear regression

  1. simple and multiple linear regression

  2. variable selection

  3. qualitative features and interactions

  4. (ordinary) least squares as maximum likelihood

Least squares as maximum likelihood

Assume

  1. $(X_i,Y_i)$ are independent
  2. $Y_i\mid X_i\sim N(\beta_0+\beta^\top X_i,\sigma^2)$

The likelihood function

$$\textstyle L(\beta_0,\beta)\triangleq\prod_{i=1}^n\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{1}{2\sigma^2}(Y_i-\beta_0-\beta^\top X_i)^2\right)$$

ouputs the probability of observing the training data $\{(X_i,Y_i)\}_{i=1}^n$ for the input parameters.

Least squares as maximum likelihood

Idea: find the parameters that maximize the probability of observing the training data:

$$\textstyle(\hat{\beta}_0,\hat{\beta}_1)\gets\argmax_{\beta_0,\beta_1}L(\beta_0,\beta_1);$$

i.e. find the parameters so that training data is most probable.

In practice, it is often more convenient to find the parameters that (equivalently) minimize the negative log-likelihood:

$$\textstyle(\hat{\beta}_0,\hat{\beta}_1) = \argmin_{\beta_0,\beta_1}-\log L(\beta_0,\beta_1).$$