STATS 413: Lecture 10

The Variability of $\hat{\mu}_{y\mid \tilde{x}}$

We see that $\text{se}(\hat{\mu}_{y\mid \tilde{x}}) = \hat{\sigma}_\varepsilon \sqrt{\tilde{x}^\top(X^\top X)^{-1}\tilde{x}}$ depends on the value of $\tilde{x}$.

• What values of $\tilde{x}$ yield smaller variability? Larger variability?

The answer:

• Smallest variability: $\tilde{x} = (1, \bar{x}_1,\ldots,\bar{x}_p)$, the mean of the predictor variables
• Variability increases as $\tilde{x}$ moves farther from the mean.

Accords with intuition

• More information about what's occurring near the center of my data
• Less information in the extremes

The variability of $\hat{\mu}_{y\mid \tilde{x}}$

Through a bit of linear algebra (no need to derive), we can get an alternative form that makes this clear

• Let $H_0$ be the "hat" matrix from a regression including only the intercept column.

• Let $X_1$ be our design matrix without the intercept column (so it has $p$ columns, one for each predictor, and $n$ rows)

• Let $\tilde{a}_1 = (\tilde{x}_1,\ldots,\tilde{x}_p)^\top$ be the point of interest, with the intercept entry excluded

• Let $\bar{a}_1 = (\bar{x}_1,\ldots,\bar{x}_p)^\top$ be the means of the predictor variables in our data

The variability of $\hat{\mu}_{y\mid \tilde{x}}$

Second term under square root is always non-negative

• If $\tilde{a}_1 = \bar{a}_1$, second term cancels out, left with $\hat{\sigma}_\varepsilon/\sqrt{n}$

OLS Prediction Eq Passes Through the Mean

A consequence of the normal equations is that the OLS solution always passes through the point $(\bar{x}_1,\bar{x}_2,\ldots,\bar{x}_p, \bar{y})$.

• Normal equations: $X^\top(X\hat{\beta}) = X^\top y$

The first normal equation takes the form:

OLS Prediction Eq Passes Through the Mean

Dividing by $n$...

So, when setting $\tilde{x} = (1, \bar{x}_1,\ldots,\bar{x}_p)$:

• Estimate of $E(Y\mid x = \tilde{x})$ is $\hat{\mu}_{y\mid \tilde{x}} = \bar{y}$

• Standard error of estimate is $\hat{\sigma}_{\varepsilon}/\sqrt{n}$

Motivating Prediction Intervals

Motivation: We know our predicted values $\hat{y}$ are just compromises, best guesses, linear approximations....

• There's still left over variability that we can't account for

• In our data: $R^2$ won't equal 1; residuals $e_i$ won't equal zero.

• If we just give someone a prediction, we give them no sense of how much uncertainty is left over!

Motivating Prediction Intervals

• Predicting sales will be 600 with an uncertainty range of $[580, 620]$ is much different than predicting sales will be 600 with a range of $[400, 800]$

• Why not form an interval around my prediction that has a promise of catching future observations $y^*$ at a particular $\tilde{x}$ value 95% of the time?

• These will be called prediction intervals

Recap: Inference for Conditional Expectations

Suppose $y_1,...,y_n$ are generated from the (stronger) linear model:

For any particular value for the predictors $\tilde{x} = (1, \tilde{x}_1,...,\tilde{x}_p)^\top$:

After running OLS regression and obtaining my estimate $\hat{\beta}$:

Recap: Inference for Conditional Expectations

A $100(1-\alpha)\%$ $t$-based confidence interval for ${\mu}_{y\mid \tilde{x}}$ is:

Recap: Inference for Conditional Expectations

A $100(1-\alpha)\%$ $t$-based confidence interval for ${\mu}_{y\mid \tilde{x}}$ is:

Is the above confidence interval appropriate for quantifying the uncertainty in a future observation?

Consider the case of $p=0$: regression with only an intercept,

• From intro stats: does a confidence interval of the form $\bar{y} \pm t_{1-\alpha/2, n-1} \hat{\sigma}_\varepsilon/\sqrt{n}$ quantify uncertainty in a future observation?

• Take $n\rightarrow \infty$. What happens to the interval width?

• Is this appropriate behavior for future observations?

An Ideal Prediction Interval

Suppose the stronger linear model holds, and that we are interested in forming a prediction interval for a future observation $y^*$ with covariate values $\tilde{x}$.

What's the distribution of

An Ideal Prediction Interval

The following would be a valid $100(1-\alpha)\%$ prediction interval for a future observation at $\tilde{x}$.

where $z_{1-\alpha/2}$ is a quantile from the standard normal distribution.

Towards an Actionable Prediction Interval

The following would be a valid prediction interval (in the sense that the probability that $y^*$ falls within this interval is $100(1-\alpha)\%$)

While valid, we can't actually use this interval in practice:

• We don't know $\beta$, crucial for the center of this interval

• We don't know $\sigma_\varepsilon$, crucial for the spread

Towards an Actionable Prediction Interval

We'll now try to replace these components by sample-based estimators. This will change

1. Center of the interval

2. Width of the interval

3. Distribution used to form the interval

Center of Prediction Intervals

Suppose the stronger linear model holds, and that we are interested in forming a prediction interval for a future observation $y^*$ with covariate values $\tilde{x}$.

The first $n$ observations represent my observed data, which I use to form coefficient estimates $\hat{\beta} = (\hat{\beta}_0,...,\hat{\beta}_p)^\top$, and my estimated prediction equation

Center of Prediction Intervals

In words, the expected value for a future response at $\tilde{x}$ equals the expected value of my sample-based prediction equation when evaluated at $\tilde{x}$, $\hat{\mu}_{y\mid \tilde{x}}$

• Both have expectation $\tilde{x}^\top\beta$

Center of Prediction Intervals

Take-away:

• Can't center prediction interval at $\beta_0 + \beta_1\tilde{x}_1 + ... + \beta_p\tilde{x}_p$

• Instead, center prediction interval at $\hat{\beta}_0 + \hat{\beta}_1\tilde{x}_1 + ... + \hat{\beta}_p\tilde{x}_p = \hat{\mu}_{y\mid \tilde{x}}$

The Variance for Prediction Intervals

When centering our prediction intervals at $\hat{\mu}_{y\mid \tilde{x}}$, we need to take two sources of variability into account:

• Variance of $y^*$ (future observation)

• Variance of $\hat{\mu}_{y\mid \tilde{x}}$ (center of the interval)

Observe that $y^*$ and $\hat{\mu}_{y\mid \tilde{x}}$ are independent of one another

• $\hat{\mu}_{y\mid \tilde{x}} = \tilde{x}^\top\hat{\beta}$ is computed using our observed data, with outcomes $y_1,...,y_n$

• $y^*$ is a future observation, independent of $y_1,...,y_n$, which was not used in estimating $\hat{\beta}$

The Variance for Prediction Intervals

Using the independence of $y^*$ and $\hat{\mu}_{y\mid \tilde{x}}$ and our results on linear combinations of slopes, we can derive a variance to help determine the width of our prediction intervals

Estimating $\sigma^2_\varepsilon$

Under the stronger linear model, when predicting for a future observation at $\tilde{x}$

We are almost there, but we generally can't make use of this result because we don't know $\sigma_{\varepsilon}$.

Estimating $\sigma^2_\varepsilon$

Why is Homoskedasticity Important?

We've talked about $\sigma_\varepsilon$ as the standard deviation of the $\varepsilon_i$ terms. It has an alternate interpretation crucial for the construction of prediction intervals:

Why is Homoskedasticity Important?

Homoskedasticity allows us to pool information across residuals (formed at different $x$ values, $x_1,...,x_n$) to calculate an estimate of the standard deviation of $y$ at any point $\tilde{x}$.

Forming the Prediction Interval

Consider the random variable

Because this statistic follows a $t_{n-p-1}$ distribution, it follows that

Forming the Prediction Interval

Re-arranging, we find that with probability $(1-\alpha)$, a future observation $y^*$ with covariates $\tilde{x}$ falls within the following interval:

Approximate Prediction Intervals

An approximation to the prediction intervals produced by R (appropriate when $n$ is large) is given by

Approximation ignores

1. Randomness in $\hat{\sigma}_{\varepsilon}$

2. Randomness in $\hat{\beta}_0 + \hat{\beta}_1\tilde{x}_1+...+\hat{\beta}_p\tilde{x}_p$

Approximate Prediction Intervals

When $n$ is large, $\hat{\sigma}_{\varepsilon}$ closely approximates $\sigma_\varepsilon$, and randomness in the center of interval is small relative to $\sigma_\varepsilon$

• No reason to use these if you have the data in R. Use the predict function instead...

• If all you have is the output from lm, this would be a reasonable approximation.

Prediction vs Confidence Intervals

We've recently introduced (1) prediction intervals for future observations $y^*$ at a point $\tilde{x}$; (2) and confidence intervals for $E(y\mid \tilde{x}) = \mu_{y\mid \tilde{x}}$

Prediction Interval for $y^*$

Confidence Interval for $\mu_{y\mid \tilde{x}}$

Same center, but different width. Prediction intervals are wider (often much wider) than confidence intervals.

Prediction vs Confidence Intervals

• Variance of $\hat{\mu}_{y\mid \tilde{x}}$, ${\sigma}_\varepsilon^2\tilde{x}^\top(X^\top X)^{-1}\tilde{x}$, tends to zero as $n\rightarrow \infty$

• Variance used for prediction intervals, ${\sigma}_\varepsilon^2(1+\tilde{x}^\top(X^\top X)^{-1}\tilde{x})$, tends to ${\sigma}_\varepsilon^2$ as $n\rightarrow \infty$.

• Even if I knew $\beta$ (and hence $\mu_{y\mid \tilde{x}}$), I wouldn't be able to perfectly predict future observations.

Prediction vs Confidence Intervals

The two types of intervals address different questions. Here's an example in the context of our father/son data set

Prediction Interval for $y^*$

• We are told that a father who is 76 inches tall has a son. Find an interval that characterizes the uncertainty in the height of an individual son.

Confidence Interval for $\mu_{y\mid \tilde{x}}$

• We are interested in the average height for the population of sons whose fathers are 76 inches tall. Find an interval that characterizes our uncertainty in our estimate of the average height of all sons whose fathers are 76 inches tall.