2.4 — OLS: Goodness of Fit and Bias — Class Content

Problem Set 2

Problem Set 2 is due by Tuesday September 21. Please see the instructions for more information on how to submit your assignment (there are multiple ways!).

Deriving the OLS Estimators

The population linear regression model is:

$$Y_i={\beta }_0+{\beta }_1{X}_i+u_i$$

The errors $\left(u_i\right)$ are unobserved, but for candidate values of $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$, we can obtain an estimate of the residual. Algebraically, the error is:

$$\widehat{u_i}=Y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{X}_i$$

Recall our goal is to find $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$ that minimizes the sum of squared errors (SSE):

$$SSE=\sum _{i=1}^n{\widehat{u_i}}^2$$

So our minimization problem is:

$$\underset{\widehat{{\beta }_0},\widehat{{\beta }_1}}{min}\sum _{i=1}^n(Y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{X}_i{)}^2$$

Using calculus, we take the partial derivatives and set it equal to 0 to find a minimum. The first order conditions are:

$$\begin{array}{cc}\frac{\partial SSE}{\partial \widehat{{\beta }_0}}& =-2\sum _{i=1}^n(Y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{X}_i)=0\\ \frac{\partial SSE}{\partial \widehat{{\beta }_1}}& =-2\sum _{i=1}^n(Y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{X}_i)X_i=0\end{array}$$

Finding $\widehat{{\beta }_0}$

Working with the first FOC, divide both sides by $-2$:

$$\sum _{i=1}^n(Y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{X}_i)=0$$

Then expand the summation across all terms and divide by $n$:

$$\underset{\overline{Y}}{\underset{}{\frac{1}{n}\sum _{i=1}^n{Y}_i}}-\underset{\widehat{{\beta }_0}}{\underset{}{\frac{1}{n}\sum _{i=1}^n\widehat{{\beta }_0}}}-\underset{\widehat{{\beta }_1}\overline{X}}{\underset{}{\frac{1}{n}\sum _{i=1}^n\widehat{{\beta }_1}{X}_i}}=0$$

Note the first term is $\overline{Y}$, the second is $\widehat{{\beta }_0}$, the third is $\widehat{{\beta }_1}\overline{X}$.²

So we can rewrite as: $$\overline{Y}-\widehat{{\beta }_0}-{\beta }_1=0$$

Rearranging:

$$\widehat{{\beta }_0}=\overline{Y}-\overline{X}{\beta }_1$$

Finding $\widehat{{\beta }_1}$

To find $\widehat{{\beta }_1}$, take the second FOC and divide by $-2$:

$$\sum _{i=1}^n(Y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{X}_i)X_i=0$$

From the formula for $\widehat{{\beta }_0}$, substitute in for $\widehat{{\beta }_0}$:

$$\sum _{i=1}^n(Y_i-[\overline{Y}-\widehat{{\beta }_1}\overline{X}]-\widehat{{\beta }_1}{X}_i)X_i=0$$

Combining similar terms:

$$\sum _{i=1}^n([Y_i-\overline{Y}]-[X_i-\overline{X}]\widehat{{\beta }_1})X_i=0$$

Distribute $X_i$ and expand terms into the subtraction of two sums (and pull out $\widehat{{\beta }_1}$ as a constant in the second sum:

$$\sum _{i=1}^n[Y_i-\overline{Y}]X_i-\widehat{{\beta }_1}\sum _{i=1}^n[X_i-\overline{X}]X_i=0$$

Move the second term to the righthand side:

$$\sum _{i=1}^n[Y_i-\overline{Y}]X_i=\widehat{{\beta }_1}\sum _{i=1}^n[X_i-\overline{X}]X_i$$

Divide to keep just $\widehat{{\beta }_1}$ on the right:

$$\frac{\sum _{i=1}^n[Y_i-\overline{Y}]X_i}{\sum _{i=1}^n[X_i-\overline{X}]X_i}=\widehat{{\beta }_1}$$

Note that from the rules about summation operators:

$$\sum _{i=1}^n[Y_i-\overline{Y}]X_i=\sum _{i=1}^n(Y_i-\overline{Y})(X_i-\overline{X})$$

and:

$$\sum _{i=1}^n[X_i-\overline{X}]X_i=\sum _{i=1}^n(X_i-\overline{X})(X_i-\overline{X})=\sum _{i=1}^n(X_i-\overline{X}{)}^2$$

Plug in these two facts:

$$\frac{\sum _{i=1}^n(Y_i-\overline{Y})(X_i-\overline{X})}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}=\widehat{{\beta }_1}$$

Algebraic Properties of OLS Estimators

The OLS residuals $\hat{u}$ and predicted values $\hat{Y}$ are chosen by the minimization problem to satisfy:

1. The expected value (average) error is 0: $$E\left(u_i\right)=\frac{1}{n}\sum _{i=1}^n\widehat{u_i}=0$$

2. The covariance between $X$ and the errors is 0: $${\hat{\sigma }}_{X,u}=0$$

Note the first two properties imply strict exogeneity. That is, this is only a valid model if $X$ and $u$ are not correlated.

3. The expected predicted value of $Y$ is equal to the expected value of $Y$: $$\overline{\hat{Y}}=\frac{1}{n}\sum _{i=1}^n\widehat{Y_i}=\overline{Y}$$

4. Total sum of squares is equal to the explained sum of squares plus sum of squared errors: $$\begin{array}{cc}TSS& =ESS+SSE\\ \sum _{i=1}^n(Y_i-\overline{Y}{)}^2& =\sum _{i=1}^n(\widehat{Y_i}-\overline{Y}{)}^2+\sum _{i=1}^n{u}^2\end{array}$$

Recall $R^2$ is $\frac{ESS}{TSS}$ or $1-SSE$

5. The regression line passes through the point $(\overline{X},\overline{Y})$, i.e. the mean of $X$ and the mean of $Y$.

Bias in $\widehat{{\beta }_1}$

Begin with the formula we derived for $\widehat{{\beta }_1}$:

$$\widehat{{\beta }_1}=\frac{\sum _{i=1}^n(Y_i-\overline{Y})(X_i-\overline{X})}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}$$

Recall from Rule 6 of summations, we can rewrite the numerator as

$$\begin{array}{cc}=& \sum _{i=1}^n(Y_i-\overline{Y})(X_i-\overline{X})\\ =& \sum _{i=1}^n{Y}_i(X_i-\overline{X})\end{array}$$

$$\widehat{{\beta }_1}=\frac{\sum _{i=1}^n{Y}_i(X_i-\overline{X})}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}$$

We know the true population relationship is expressed as:

$$Y_i={\beta }_0+{\beta }_1{X}_i+u_i$$

Substituting this in for $Y_i$ in equation 2:

$$\widehat{{\beta }_1}=\frac{\sum _{i=1}^n({\beta }_0+{\beta }_1{X}_i+u_i)(X_i-\overline{X})}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}$$ Breaking apart the sums in the numerator:

$$\widehat{{\beta }_1}=\frac{\sum _{i=1}^n{\beta }_0(X_i-\overline{X})+\sum _{i=1}^n{\beta }_1{X}_i(X_i-\overline{X})+\sum _{i=1}^n{u}_i(X_i-\overline{X})}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}$$

We can simplify equation 4 using Rules 4 and 5 of summations

1. The first term in the numerator $\left[\sum _{i=1}^n{\beta }_0(X_i-\overline{X})\right]$ has the constant ${\beta }_0$, which can be pulled out of the summation. This gives us the summation of deviations, which add up to 0 as per Rule 4:

$$\begin{array}{cc}\sum _{i=1}^n{\beta }_0(X_i-\overline{X})& ={\beta }_0\sum _{i=1}^n(X_i-\overline{X})\\ & ={\beta }_0\left(0\right)\\ & =0\end{array}$$

2. The second term in the numerator $\left[\sum _{i=1}^n{\beta }_1{X}_i(X_i-\overline{X})\right]$ has the constant ${\beta }_1$, which can be pulled out of the summation. Additionally, Rule 5 tells us $\sum _{i=1}^n{X}_i(X_i-\overline{X})=\sum _{i=1}^n(X_i-\overline{X}{)}^2$:

$$\begin{array}{cc}\sum _{i=1}^n{\beta }_1{X}_1(X_i-\overline{X})& ={\beta }_1\sum _{i=1}^n{X}_i(X_i-\overline{X})\\ & ={\beta }_1\sum _{i=1}^n(X_i-\overline{X}{)}^2\end{array}$$

When placed back in the context of being the numerator of a fraction, we can see this term simplifies to just ${\beta }_1$:

$$\begin{array}{cc}\frac{{\beta }_1\sum _{i=1}^n(X_i-\overline{X}{)}^2}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}& =\frac{{\beta }_1}{1}\times \frac{\sum _{i=1}^n(X_i-\overline{X}{)}^2}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}\\ & ={\beta }_1\end{array}$$

Thus, we are left with:

$$\widehat{{\beta }_1}={\beta }_1+\frac{\sum _{i=1}^n{u}_i(X_i-\overline{X})}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}$$

Now, take the expectation of both sides:

$$E\left[\widehat{{\beta }_1}\right]=E[{\beta }_1+\frac{\sum _{i=1}^n{u}_i(X_i-\overline{X})}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}]$$

We can break this up, using properties of expectations. First, recall $E[a+b]=E\left[a\right]+E\left[b\right]$, so we can break apart the two terms.

$$E\left[\widehat{{\beta }_1}\right]=E\left[{\beta }_1\right]+E\left[\frac{\sum _{i=1}^n{u}_i(X_i-\overline{X})}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}\right]$$

Second, the true population value of ${\beta }_1$ is a constant, so $E\left[{\beta }_1\right]={\beta }_1$.

Third, since we assume $X$ is also “fixed” and not random, the variance of $X$, $\sum _{i=1}^n(X_i-\overline{X})$, in the denominator, is just a constant, and can be brought outside the expectation.

$$E\left[\widehat{{\beta }_1}\right]={\beta }_1+\frac{E\left[\sum _{i=1}^n{u}_i(X_i-\overline{X})\right]}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}$$

Thus, the properties of the equation are primarily driven by the expectation $E[\sum _{i=1}^n{u}_i(X_i-\overline{X})]$. We now turn to this term.

Use the property of summation operators to expand the numerator term:

$$\begin{array}{cc}\widehat{{\beta }_1}& ={\beta }_1+\frac{\sum _{i=1}^n{u}_i(X_i-\overline{X})}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}\\ \widehat{{\beta }_1}& ={\beta }_1+\frac{\sum _{i=1}^n(u_i-\overline{u})(X_i-\overline{X})}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}\end{array}$$

Now divide the numerator and denominator of the second term by $\frac{1}{n}$. Realize this gives us the covariance between $X$ and $u$ in the numerator and variance of $X$ in the denominator, based on their respective definitions.

$$\begin{array}{cc}\widehat{{\beta }_1}& ={\beta }_1+\frac{\frac{1}{n}\sum _{i=1}^n(u_i-\overline{u})(X_i-\overline{X})}{\frac{1}{n}\sum _{i=1}^n(X_i-\overline{X}{)}^2}\\ \widehat{{\beta }_1}& ={\beta }_1+\frac{cov(X,u)}{var\left(X\right)}\\ \widehat{{\beta }_1}& ={\beta }_1+\frac{s_{X,u}}{s_X^2}\end{array}$$

By the Zero Conditional Mean assumption of OLS, $s_{X,u}=0$.

Alternatively, we can express the bias in terms of correlation instead of covariance:

$$E\left[\widehat{{\beta }_1}\right]={\beta }_1+\frac{cov(X,u)}{var\left(X\right)}$$

From the definition of correlation:

$$\begin{array}{cc}cor(X,u)& =\frac{cov(X,u)}{s_X{s}_u}\\ cor(X,u)s_X{s}_u& =cov(X,u)\end{array}$$

Plugging this in:

$$\begin{array}{cc}E\left[\widehat{{\beta }_1}\right]& ={\beta }_1+\frac{cov(X,u)}{var\left(X\right)}\\ E\left[\widehat{{\beta }_1}\right]& ={\beta }_1+\frac{[cor(X,u)s_x{s}_u]}{s_X^2}\\ E\left[\widehat{{\beta }_1}\right]& ={\beta }_1+\frac{cor(X,u)s_u}{s_X}\\ E\left[\widehat{{\beta }_1}\right]& ={\beta }_1+cor(X,u)\frac{s_u}{s_X}\end{array}$$

Proof of the Unbiasedness of $\widehat{{\beta }_1}$

Begin with equation:³

$$\widehat{{\beta }_1}=\frac{\sum Y_i{X}_i}{\sum X_i^2}$$

Substitute for $Y_i$:

$$\widehat{{\beta }_1}=\frac{\sum ({\beta }_1{X}_i+u_i)X_i}{\sum X_i^2}$$

Distribute $X_i$ in the numerator:

$$\widehat{{\beta }_1}=\frac{\sum {\beta }_1{X}_i^2+u_i{X}_i}{\sum X_i^2}$$

Separate the sum into additive pieces:

$$\widehat{{\beta }_1}=\frac{\sum {\beta }_1{X}_i^2}{\sum X_i^2}+\frac{u_i{X}_i}{\sum X_i^2}$$

${\beta }_1$ is constant, so we can pull it out of the first sum:

$$\widehat{{\beta }_1}={\beta }_1\frac{\sum X_i^2}{\sum X_i^2}+\frac{u_i{X}_i}{\sum X_i^2}$$

Simplifying the first term, we are left with:

$$\widehat{{\beta }_1}={\beta }_1+\frac{u_i{X}_i}{\sum X_i^2}$$

Now if we take expectations of both sides:

$$E\left[\widehat{{\beta }_1}\right]=E\left[{\beta }_1\right]+E\left[\frac{u_i{X}_i}{\sum X_i^2}\right]$$

${\beta }_1$ is a constant, so the expectation of ${\beta }_1$ is itself.

$$E\left[\widehat{{\beta }_1}\right]={\beta }_1+E[\frac{u_i{X}_i}{\sum X_i^2}]$$

Using the properties of expectations, we can pull out $\frac{1}{\sum X_i^2}$ as a constant:

$$E\left[\widehat{{\beta }_1}\right]={\beta }_1+\frac{1}{\sum X_i^2}E[\sum u_i{X}_i]$$

Again using the properties of expectations, we can put the expectation inside the summation operator (the expectation of a sum is the sum of expectations):

$$E\left[\widehat{{\beta }_1}\right]={\beta }_1+\frac{1}{\sum X_i^2}\sum E\left[u_i{X}_i\right]$$

Under the exogeneity condition, the correlation between $X_i$ and $u_i$ is 0.

$$E\left[\widehat{{\beta }_1}\right]={\beta }_1$$