STATS 413: Lecture 12

Recap: Diagnostics for the Stronger Linear Model

The stronger linear model has the following core components:

1. Linearity: At any $x_i$, $\text{E}(y \mid x_i) = \beta_0 + \beta_1x_{i1} + ... + \beta_px_{ip} = x_i^\top\beta$

2. Homoskedasticity: For any $x_i$, $\text{var}(\varepsilon_i\mid x_i) = \sigma^2_\varepsilon$ for all $i$.

3. Normality: $\varepsilon_i$ are IID normally distributed

Recap: Diagnostics for the Stronger Linear Model

We discussed the use of diagnostic plots to assess each of these assumptions.

• What plot(s) do we use to assess linearity? What are we hoping to see in these plots?

• What plot(s) do we use to assess homoskedasticity? What are we hoping to see in these plots?

• What plot(s) do we use to assess normality? What are we hoping to see in these plots?

Today: Finding Unusual Data Points

In addition to our suite of regression diagnostics assessing linearity, homoskedasticity, and normality, we'll introduce additional diagnostics to help us flag potentially unusual / important observations in our data set

• Outliers - observations do not behave like the bulk of our data points

  • Seem to represent a departure from the trends observed in the majority of our observations.

Today: Finding Unusual Data Points

• Influential points - observations which unduly/substantially affect the prediction equation.

  • Not all observations have the same impact on the OLS solution $\hat{\beta}$.
  • What makes a point matter more or less?

A data point could be none, one, or both of these.

Outliers

Informally, an outlier is an observation that differs substantially from the other points in the data set.

• Might be due to misrecording or typo when inputing the data, or due to a failed experiment

• Might also represent a rare, but possible, outcome in our population of interest (might be an exciting discovery!)

• Could represent that something has gone awry (for instance, credit card fraud in the distribution of one's monthly expenditures).

Outliers

Informally, an outlier is an observation that differs substantially from the other points in the data set.

• Might be due to lurking variables (for instance, if a professional athlete is included in an investigation of exercise habits).

• Including an additional predictor variable can also reveal outliers that weren't apparent before.

Self-Influence / Leverage

Individual observations vary in the influence they have over the fitted values, and over the estimated regression line.

• From the previous slide, we see evidence that whether a not a data point's predictor value is "extreme" plays a key role here

We define the leverage, or self-influence, of the $i$th observation (with covariate $x_i$) as

where $h_{i,i}$ is the $i$th diagonal entry of the hat matrix,

Self-Influence / Leverage

• Leverage thus depends not only on $x_i$, but on the predictor variables for the other $n-1$ individuals in the data set.

Intuition for Leverage

Fitted value:

$h_{i,i}$ can be thought of as the weight placed on $y_i$ when forming $\hat{y}_i$. $\sum_{j\neq i} h_{i,j}y_j$ is the contribution from the other $n-1$ observations.

Consider the extreme case $h_{i,i}=1$:

• What happens to $\hat{y}_i$?

• What happens to $e_i$?

• Under the weaker linear model, what happens to $\text{var}(\hat{y}_i\mid x_i)$?

• What happens to $\text{var}(e_i\mid x_i)$?

Intuition for Leverage

Under the weaker linear model...

The larger $h_{i,i}$...

• The smaller the variability in the $i$th residual

• Less variability $\Rightarrow$ $y_i$ tends to be closer to $\hat{y}_i$.

• The more $\hat{y}_i$ behaves like $y_i$.

• The closer $\hat{y}_i$'s variability is to that of $y_i$ itself.

Leverage and "Outliers in X"

Through a bit of linear algebra (no need to derive), we can derive an alternative form for $h_{i,i}$ that helps understand what values of $x_i$ have high leverage.

• Let $H_0$ be the "hat" matrix from a regression including only the intercept column.

• Let $X_{\backslash 0}$ be our design matrix without the intercept column (so it has $p$ columns, one for each predictor, and $n$ rows)

• Let $x_{i \backslash 0} = (x_{i1},...,{x}_{ip})$ be the predictor variables for the $i$th individual, with the intercept entry excluded

• Let $\bar{x}_{\backslash 0} = (\bar{x}_1,...,\bar{x}_p)$ be the means of the predictor variables in our data

Leverage and "Outliers in X"

Second term is always non-negative

• If $x_i = \bar{x}$, second term cancels out, left with $\frac1n$.

• We see that $x_i = \bar{x}$ results in the smallest possible self-influence / leverage, $\frac1n$.

• As $x_i$ moves farther away from $\bar{x}$, the leverage increases

• Points with high leverage are outliers with respect to the distribution of the predictor variables

Leverage and "Outliers in X"

Uses for Leverage

Leverages $h_{i,i}$ provide a scalar/numeric representation of the "self-influence" of a point

• How much does the observation $y_i$ determine the fitted value $\hat{y}_i$?

We see it is a function of the predictor variables alone, as it is based on the hat matrix.

Uses for Leverage

Leverages help us answer the following

• "At what points $x_i$ would an outlier in my responses $y_i$ have the potential to dramatically alter my prediction equation?"

• Low leverage - outlying $y_i$ unlikely to impact our prediction equation
• High leverage - outlying $y_i$ can dramatically impact the prediction equation

Rule of thumb: Leverages greater than $\frac{2(p+1)}{n}$ are considered high.

A Test for Outliers

Formally, we are testing

• Null hypothesis: For all observations $a=1,...,n$, $y_a = x_a^\top\beta + \varepsilon_a$, $\varepsilon_a\overset{\text{iid}}{\sim} N(0, \sigma^2_\varepsilon)$

• Alternative hypothesis: This model holds for all observations except for the $i$th individual, who comes from a different, unspecified, model.

A Test for Outliers

Under the null hypothesis, we have that

and that $y_i - \hat{y}_{(i)}$ is normally distributed.

[Does this remind you of the results underpinning our prediction intervals? It should...]

Externally Studentized Residuals

It turns out that, with $\hat{\sigma}_{\varepsilon(i)}$ the RMSE from a regression excluding the $i$th individual,

Externally Studentized Residuals

These $t_i$ are called externally studentized residuals. Interestingly, $t_i$ can also be expressed as

where $r_i$ are the standardized residuals we introduced previously.

• Implication: we can actually compute $t_i$ without computing a regression that excludes the $i$th individual.

Multiple Hypothesis Tests

For any particular $i$,

• We could then compute $|t_i|$ and compare it to $t_{1-\alpha/2, (n-p-2)}$.

• Reject if $|t_i|$ exceeds this value

• For any particular $i$, this controls the Type I error rate (the probability of falsely rejecting a true null hypothesis) at $\alpha$.

Multiple Hypothesis Tests

Suppose I repeated this process for all $n$ observations in my study. For each observation, I try to determine whether that individual observation is an outlier.

• What do you think the probability of committing at least one Type I error will be?

• $\alpha$? Larger than $\alpha$?

What to Do with Outliers?

In general, it is not good practice to remove outliers unless we have a compelling reason that the point isn't representative of the population of interest

• Good reason: data entry error, individual was included in data set who truly should have been excluded

What to Do with Outliers?

In general, it is not good practice to remove outliers unless we have a compelling reason that the point isn't representative of the population of interest

• Bad reasons

  • Slope coefficients become more significant when excluded
  • Data looks more linear when observation excluded
  • $R^2$ increases when observation excluded
  • ...

Outliers can indicate that your linear model doesn't apply to the entirety of the population under study, that there might be a lurking variable, that you might need a more complex model...

Cook's Distance

The Cook's distance $D_i$ of an observation $i$ is the sum of the squared differences between predicted values in regressions fit including $i$ and excluding $i$, normalized by $(p+1)\hat{\sigma}_\varepsilon^2$

where $\hat{\beta}_{(i)}$ are the regression coefficients from a regression excluding observation $i$.

Cook's Distance

Equivalently, Cook's distance can be expressed as

from whence we see that $D_i$ combines the impact of

• Large residual (in terms of standardized residual $r_i$)
• Large leverage ($h_{i,i}$)

Rule of thumb: $D_i > 1$ indicates an influential point

Residuals vs Leverage

Here's a normal looking residuals vs leverage plot. Nothing concerning here...

Residuals vs Leverage

In our simulated data sets: