STATS 413: Lecture 3

Finding the Least Squares Regression Line

We again consider distances between observed values, $y_i$, and predicted values, ${\beta}_0 + {\beta}_1x_{i1}+...+{\beta}_px_{ip}$ for different choices of $\beta = (\beta_0,\beta_1,...,\beta_p)$

Finding the Least Squares Regression Line

R produces the solution using the lm command.

lm(y~x1+x2+...+xp)

Python offers multiple approaches. Using scikit-learn:

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X, y)  # X is n×p matrix of predictors

or using statsmodels for statistical output similar to R:

import statsmodels.api as sm
X_with_const = sm.add_constant(X)  # adds intercept
model = sm.OLS(y, X_with_const).fit()

Additional Quantities

• Fitted values for the $n$ observations in our data set

  \[ \hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_{i1} + \dots + \hat{\beta}_p x_{ip}\;\; (i=1,\dots,n) \]

• Fitted model (for forming predictions at other values of ${x}$ beyond those in our data).

  \[ \hat{f}({x}^*) = \hat {\beta}_0 + \hat{\beta}_1 x^*_{1} + \dots + \hat{\beta}_p x^*_{p} \\ \]

• Residuals:

  \[ e_i = y_i - \hat{y}_i \;\; (i=1,\dots,n) \]

Additional Quantities

Note that

General Objective in Regression

We begin by postulating a generative model, which describes how our observed data $({x}_1, y_1), \dots, ({x}_n, y_n)$ came to be.

• Relate the observed data to parameters of our population of interest.

• Through using the data, we try to infer the values of the parameters for the population of interest.

General Objective in Regression

General form for this generative model states that, for $i=1,\dots,n$,

where

• $f(x_i) \triangleq \textbf{E}(y_i \mid x_i)$ is the (unknown) regression function (ie the signal) relating expected response to predictors

• $\varepsilon_i$ are random noise terms such that $\textbf{E}(\varepsilon_i\mid x_i)=0$.

Objectives

• estimate $f$ using observed data, resulting in estimated function $\hat{f}$

• estimate the conditional distribution $Y_i\mid {x}_i$

The Weaker Linear Model, in Words

The model on the previous slide makes the following important assumptions:

1. Linearity: the true relationship between the conditional expectation of $y$ and the predictors is linear. That is, $\textbf{E}(y_i \mid x_i)$ is truly $\beta_0 + \beta_1x_{i,1} + ... \beta_px_{i,p}$, as opposed to some nonlinear function of $x_i$

2. Homoskedasticity: the error terms have the same variance, i.e. $\text{var}(\varepsilon_i\mid x_i) = \sigma^2_\varepsilon$ for all individuals.

• If the variability of the noise instead changes as a function of the covariates, we call the noise heteroskedastic

3. Uncorrelated noise: $\text{Cov}(\varepsilon_i, \varepsilon_j\mid x_i, x_j) = 0$ for $i\neq j$.

The Weaker Linear Model, in Words

Note: for now, we don't assume that $\varepsilon_i$ follow a particular distribution (for instance, normal). The model will be strengthened later when discussing statistical inference.

Fixed-$x$

Under this regression model, we view the values of $(x_1\dots,x_n)$ as fixed quantities

• Could be truly fixed (for example, if we run an experiment and get to choose $x$)

• More generally, we proceed by conditioning upon the observed values of $x_1,\dots,x_n$ in the sample at hand.

• The outcomes are still random, due to their dependence on $\{\varepsilon_i\}$.

• Regression models the conditional distribution of $Y$ given $X=x$.

Fixed-$x$

Throughout the semester, we'll be conditioning on the covariates used in the regression model when making probabilistic statements

• Often this conditioning will not be overtly reflected in notation

• For instance, moving forwards I will typically write $\textbf{E}(\varepsilon_i)$ instead of $\textbf{E}(\varepsilon_i\mid x_i)$.