Leverage

• leverage is the amount of influence an observation has on the estimation of $\hat{\beta }$
• Mathematically, we can define this as the diagonal elements of the hat matrix.

  What is the hat matrix?

• $X(X^{T}X{)}^{-1}{X}^T$

Leverage

• leverage is the amount of influence an observation has on the estimation of $\hat{\beta }$
• Mathematically, we can define this as the diagonal elements of the hat matrix.

$$h_i=H_{ii}{h}_i=X(X^{T}X{)}^{-1}{X}_{ii}^T$$

Leverage

• Recall that the variance of the residuals is

$$\text{var}\left(e_i\right)={\sigma }^2(1-h_i)$$

• so large leverage points will pull the fit towards $y_i$

Leverage

• The leverage, $h_i$, will always be between 0 and 1

$$\begin{array}{cc}{h}_i& =\sum _j{H}_{ij}{H}_{ji}\end{array}$$

Leverage

• The leverage, $h_i$, will always be between 0 and 1

$$\begin{array}{cc}{h}_i& =\sum _j{H}_{ij}{H}_{ji}\\ & =\sum _j{H}_{ij}^2\end{array}$$

Leverage

• The leverage, $h_i$, will always be between 0 and 1

$$\begin{array}{cc}{h}_i& =\sum _j{H}_{ij}{H}_{ji}\\ & =\sum _j{H}_{ij}^2\\ & =H_{ii}^2+\sum _{j\ne i}{H}_{ij}^2\end{array}$$

Leverage

• The leverage, $h_i$, will always be between 0 and 1

$$\begin{array}{cc}{h}_i& =\sum _j{H}_{ij}{H}_{ji}\\ & =\sum _j{H}_{ij}^2\\ & =H_{ii}^2+\sum _{j\ne i}{H}_{ij}^2\\ & =h_i^2+\sum _{j\ne i}{H}_{ij}^2\end{array}$$

Leverage

• The leverage, $h_i$, will always be between 0 and 1

$$\begin{array}{cc}{h}_i& =\sum _j{H}_{ij}{H}_{ji}\\ & =\sum _j{H}_{ij}^2\\ & =H_{ii}^2+\sum _{j\ne i}{H}_{ij}^2\\ & =h_i^2+\sum _{j\ne i}{H}_{ij}^2\end{array}$$

• This means that $h_i$ must be larger than $h_i^2$, implying that $h_i$ will always be between 0 and 1 ✅

Leverage

• The ${\sum }_i{h}_i=p+1$ (remember when we calculated the trace of H?)
• This means an average value for $h_i$ is $(p+1)/n$
• 👍 A rule of thumb leverages greater than $2(p+1)/n$ should get an extra look

Standardized residuals

• We can use the leverages to standardize the residuals
• Instead of plotting the residuals, $e$, we can plot the standardized residuals

$$r_i=\frac{e}{\hat{\sigma }\sqrt{1-h_i}}$$

• 👍 A rule of thumb for standardized residuals: those greater than 4 would be very unusual and should get an extra look

Doing it in R

• It is good to understand how to calculate these standardized residuals by hand, but there is an R function that does this for you (rstandard())
• There is also an R function to calculate the leverage (hatvalues())

Standardized residuals

mod <- lm(mpg ~ disp, data = mtcars)
d <- data.frame(
  standardized_resid = rstandard(mod),
  fit = fitted(mod)
)
ggplot(d, aes(fit, standardized_resid)) +
  geom_point() +geom_hline(yintercept = 0) + 
  labs(y = "Standardized Residual")

Outliers

• An outlier is a point that doesn't fit the current model well

Outliers

• An outlier is a point that doesn't fit the current model well

• This first plot, see a point that is definitely an outlier but it doesn't have much leverage or influence over the fit

Outliers

• An outlier is a point that doesn't fit the current model well

• This second plot, see a point that has a large leverage but is not an outlier and doesn't have much influence over the fit

Outliers

• An outlier is a point that doesn't fit the current model well

• In the third plot, the point is both an outlier and very influential. Not only is the residual for this point large, but it inflates the residuals for the other points

Outliers

• To detect points like this third example, it can be prudent to exclude the point and recompute the estimates to get ${\hat{\beta }}_{\left(i\right)}$ and ${\hat{\sigma }}_{\left(i\right)}^2$

$${\hat{y}}_{\left(i\right)}=x_i^T{\hat{\beta }}_{\left(i\right)}$$

• If ${\hat{y}}_{\left(i\right)}-y_i$ is large, then observation $i$ is an outlier.

Studentized residuals

$$t_i=\frac{y_i-{\hat{y}}_{\left(i\right)}}{{\hat{\sigma }}_{\left(i\right)}(1+x_i^T(X_{\left(i\right)}^T{X}_{\left(i\right)}{)}^{-1}{x}_i{)}^{1/2}}$$ The have a $t$ distribution with $(n-1)-(p+1)=n-p-2$ degrees of freedom if the model is correct and $ϵN(0,{\sigma }^{2}I)$.

• There is an easier way to compute these using the studentized residuals!

$$t_i=r_i{\left(\frac{n-p-2}{n-p-1-r_i^2}\right)}^{1/2}$$

Studentized residuals

It is good to understand how to calculate these studentized residuals by hand, but there is an R function that does this for you (rstudent())

Influential points

• A common measure to determine influential points is Cook's D

$$D_i=\frac{(\hat{y}-{\hat{y}}_{\left(i\right)}{)}^T(\hat{y}-{\hat{y}}_{\left(i\right)})}{(p+1){\hat{\sigma }}^2}$$

• $(\hat{y}-{\hat{y}}_{\left(i\right)})$ is the change in the fit after leaving observation $i$ out.
• This can be calculated using:

$$\frac{1}{p+1}{r}_i^2\frac{h_i}{1-h_i}$$

• 👍 A rule of thumb is to give observations with Cook's Distance > $4/n$ an extra look

Cook's Distance

Cook's Distance

It is good to understand how to calculate these by hand, but there is an R function that does this for you (cooks.distance())