Gauss Markov Theorem

Gauss Markov TheoremPart 2Dr. D’Agostino McGowan1 / 9

  
  Dr. Lucy D'Agostino McGowan

Gauss Markov TheoremLeast Squares is the Best Linear Unbiased Estimator (BLUE)
2 / 9

Gauss Markov Theorem

Least Squares is the Best Linear Unbiased Estimator (BLUE)

What does this mean?

2 / 9

Gauss Markov Theorem

Least Squares is the Best Linear Unbiased Estimator (BLUE)

What does this mean?

⬜ Best: has the smallest variance

2 / 9

Gauss Markov Theorem

Least Squares is the Best Linear Unbiased Estimator (BLUE)

What does this mean?

⬜ Best: has the smallest variance
✅ Linear: it is linear in the observed output variables

2 / 9

Gauss Markov Theorem

Least Squares is the Best Linear Unbiased Estimator (BLUE)

What does this mean?

⬜ Best: has the smallest variance
✅ Linear: it is linear in the observed output variables
✅ Unbiased: it is unbiased

2 / 9

Gauss Markov Theorem

Least Squares is the Best Linear Unbiased Estimator (BLUE)

What does this mean?

⬜ Best: has the smallest variance
✅ Linear: it is linear in the observed output variables
✅ Unbiased: it is unbiased
✅ Estimator

2 / 9

Gauss Markov Theorem

Least Squares is the Best Linear Unbiased Estimator (BLUE)

What does this mean?

⬜ Best: has the smallest variance
✅ Linear: it is linear in the observed output variables
✅ Unbiased: it is unbiased
✅ Estimator

Let's prove best now.

2 / 9

What do we need to do?

☝️ Come up with another linear unbiased estimator of $β$ , let's call it $\tilde{β}$ ✌️ Show that this estimator has a variance that is no smaller than $var (\hat{β} | X)$

What is the $var (\hat{β} | X)$ ?

3 / 9

  
  Dr. Lucy D'Agostino McGowan

☝️ A linear, unbiased estimator of ββ~β=Cyβ~=Cy
4 / 9

☝️ A linear, unbiased estimator of $β$

$\tilde{β} = C y$

Why $C y$ ? This is linear.

4 / 9

☝️ A linear, unbiased estimator of $β$

$\tilde{β} = C y$

Why $C y$ ? This is linear.

$C = (X^{T} X)^{- 1} X^{T} + D$

4 / 9

☝️ A linear, unbiased estimator of $β$

When is $\tilde{β}$ unbiased?

5 / 9

☝️ A linear, unbiased estimator of $β$

When is $\tilde{β}$ unbiased?

Show that $E [C y | X] = β$

5 / 9

☝️ A linear, unbiased estimator of $β$

When is $\tilde{β}$ unbiased?

Show that $E [C y | X] = β$ $E [C y | X] = E [((X^{T} X)^{- 1} X^{T} + D) (X β + ϵ) | X]$

5 / 9

☝️ A linear, unbiased estimator of $β$

When is $\tilde{β}$ unbiased?

Show that $E [C y | X] = β$ $E [C y | X] = E [((X^{T} X)^{- 1} X^{T} + D) (X β + ϵ) | X]$

`Try It`

Solve for $E [C y | X]$ .

02:00

5 / 9

☝️ A linear, unbiased estimator of $β$

When is $\tilde{β}$ unbiased?

$\begin{aligned} E [\tilde{β} | X] & = E [((X^{T} X)^{- 1} X^{T} + D) (X β + ϵ) | X] \\ = D X β + β \end{aligned}$

6 / 9

☝️ A linear, unbiased estimator of $β$

When is $\tilde{β}$ unbiased?

$\begin{aligned} E [\tilde{β} | X] & = E [((X^{T} X)^{- 1} X^{T} + D) (X β + ϵ) | X] \\ = D X β + β \end{aligned}$

We need $E [C y | X] = β$

For $\tilde{β}$ to be unbiased $D X$ must be 0.

6 / 9

Now let's calculate the variance

For our estimate $\hat{β}$ to be best what needs to be true?

7 / 9

Now let's calculate the variance

For our estimate $\hat{β}$ to be best what needs to be true?

$var (\hat{β} | X) < var (\tilde{β} | X)$

7 / 9

Now let's calculate the variance

For our estimate $\hat{β}$ to be best what needs to be true?

$var (\hat{β} | X) < var (\tilde{β} | X)$ $var (\tilde{β} | X) = var (C y | X)$

7 / 9

Now let's calculate the variance

For our estimate $\hat{β}$ to be best what needs to be true?

$var (\hat{β} | X) < var (\tilde{β} | X)$ $var (\tilde{β} | X) = var (C y | X)$

What is constant?

7 / 9

Now let's calculate the variance

For our estimate $\hat{β}$ to be best what needs to be true?

$var (\hat{β} | X) < var (\tilde{β} | X)$

$\begin{aligned} var (\tilde{β} | X) & = var (C y | X) \\ = C var (y | X) C^{T} \end{aligned}$

`Try It`

Finish solving for $v a r [C y | X]$ . Remember $D X = 0$ for this to be unbiased.

03:00

8 / 9

Now let's calculate the variance

For our estimate $\hat{β}$ to be best what needs to be true?

$var (\hat{β} | X) < var (\tilde{β} | X)$

$\begin{aligned} var (\tilde{β} | X) & = var (C y | X) \\ = C var (y | X) C^{T} \\ = var (\hat{β}) + σ^{2} D D^{T} \end{aligned}$

9 / 9

Now let's calculate the variance

For our estimate $\hat{β}$ to be best what needs to be true?

$var (\hat{β} | X) < var (\tilde{β} | X)$

$\begin{aligned} var (\tilde{β} | X) & = var (C y | X) \\ = C var (y | X) C^{T} \\ = var (\hat{β}) + σ^{2} D D^{T} \end{aligned}$

${D D}^{T}$ is a positive semidefinite matrix, therefore

$var (\hat{β})$ is always $< var (\tilde{β})$

9 / 9

Help

Keyboard shortcuts

↑, ←, Pg Up, k

Go to previous slide

↓, →, Pg Dn, Space, j

Go to next slide

Home

Go to first slide

End

Go to last slide

Number + Return

Go to specific slide

b / m / f

Toggle blackout / mirrored / fullscreen mode

Clone slideshow

Toggle presenter mode

Restart the presentation timer

?, h

Toggle this help