class: center, middle, inverse, title-slide # 3.4 — Multivariate OLS Estimators ## ECON 480 • Econometrics • Fall 2021 ### Ryan Safner<br> Assistant Professor of Economics <br> <a href="mailto:safner@hood.edu"><i class="fa fa-paper-plane fa-fw"></i>safner@hood.edu</a> <br> <a href="https://github.com/ryansafner/metricsF21"><i class="fa fa-github fa-fw"></i>ryansafner/metricsF21</a><br> <a href="https://metricsF21.classes.ryansafner.com"> <i class="fa fa-globe fa-fw"></i>metricsF21.classes.ryansafner.com</a><br> --- class: inverse # Outline ### [The Multivariate OLS Estimators](#3) ### [The Expected Value of `\(\hat{\beta_j}\)`: Bias](#10) ### [Precision of `\(\hat{\beta_j}\)`](#41) ### [A Summary of Multivariate OLS Estimator Properties](#80) ### [Updated Measures of Fit](#83) --- class: inverse, center, middle # The Multivariate OLS Estimators --- # The Multivariate OLS Estimators - By analogy, we still focus on the .hi[ordinary least squares (OLS) estimators] of the unknown population parameters `\(\beta_0, \beta_1, \beta_2, \cdots, \beta_k\)` which solves: `$$\min_{\hat{\beta_0}, \hat{\beta_1}, \hat{\beta_2}, \cdots, \hat{\beta_k}} \sum^n_{i=1}\left[\underbrace{Y_i-\underbrace{(\hat{\beta_0}+\hat{\beta_1}X_{1i}+\hat{\beta_2}X_{2i}+\cdots+ \hat{\beta_k}X_{ki})}_{\color{gray}{\hat{Y}_i}}}_{\color{gray}{u_i}}\right]^2$$` - Again, OLS estimators are chosen to .hi-purple[minimize] the .hi[sum of squared errors (SSE)] - i.e. sum of squared distances between actual values of `\(Y_i\)` and predicted values `\(\hat{Y_i}\)` --- # The Multivariate OLS Estimators: FYI .smallest[ .content-box-red[ .red[**Math FYI]**: in linear algebra terms, a regression model with `\(n\)` observations of `\(k\)` independent variables: `$$\mathbf{Y} = \mathbf{X \beta}+\mathbf{u}$$` `$$\underbrace{\begin{pmatrix} y_1\\ y_2\\ \vdots \\ y_n\\ \end{pmatrix}}_{\mathbf{Y}_{(n \times 1)}} = \underbrace{\begin{pmatrix} x_{1,1} & x_{1,2} & \cdots & x_{1,n}\\ x_{2,1} & x_{2,2} & \cdots & x_{2,n}\\ \vdots & \vdots & \ddots & \vdots\\ x_{k,1} & x_{k,2} & \cdots & x_{k,n}\\ \end{pmatrix}}_{\mathbf{X}_{(n \times k)}} \underbrace{\begin{pmatrix} \beta_1\\ \beta_2\\ \vdots \\ \beta_k \\ \end{pmatrix}}_{\mathbf{\beta}_{(k \times 1)}} + \underbrace{\begin{pmatrix} u_1\\ u_2\\ \vdots \\ u_n \\ \end{pmatrix}}_{\mathbf{u}_{(n \times 1)}}$$` ] ] -- .smallest[ - The OLS estimator for `\(\beta\)` is `\(\hat{\beta}=(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{Y}\)` 😱 ] -- .smallest[ - Appreciate that I am saving you from such sorrow 🤖 ] --- # The Sampling Distribution of `\(\hat{\beta_j}\)` .pull-left[ .smallest[ - For *any* individual `\(\beta_j\)`, it has a sampling distribution: `$$\hat{\beta_j} \sim N \left(E[\hat{\beta_j}], \;se(\hat{\beta_j})\right)$$` - We want to know its sampling distribution’s: - .hi-purple[Center]: `\(\color{#6A5ACD}{E[\hat{\beta_j}]}\)`; what is the *expected value* of our estimator? - .hi-purple[Spread]: `\(\color{#6A5ACD}{se(\hat{\beta_j})}\)`; how *precise* or *uncertain* is our estimator? ] ] .pull-right[ <img src="3.4-slides_files/figure-html/unnamed-chunk-1-1.png" width="504" /> ] --- # The Sampling Distribution of `\(\hat{\beta_j}\)` .pull-left[ .smallest[ - For *any* individual `\(\beta_j\)`, it has a sampling distribution: `$$\hat{\beta_j} \sim N \left(E[\hat{\beta_j}], \;se(\hat{\beta_j})\right)$$` - We want to know its sampling distribution’s: - .hi-purple[Center]: `\(\color{#6A5ACD}{E[\hat{\beta_j}]}\)`; what is the *expected value* of our estimator? - .hi-purple[Spread]: `\(\color{#6A5ACD}{se(\hat{\beta_j})}\)`; how *precise* or *uncertain* is our estimator? ] ] .pull-right[ .center[  ] ] --- class: inverse, center, middle # The Expected Value of `\(\hat{\beta_j}\)`: Bias --- # Exogeneity and Unbiasedness - As before, `\(E[\hat{\beta_j}]=\beta_j\)` when `\(X_j\)` is .hi-purple[exogenous] (i.e. `\(cor(X_j, u)=0\)`) -- - We know the true `\(E[\hat{\beta_j}]=\beta_j+\underbrace{cor(X_j,u)\frac{\sigma_u}{\sigma_{X_j}}}_{\text{O.V. Bias}}\)` -- - If `\(X_j\)` is .hi[endogenous] (i.e. `\(cor(X_j, u)\neq 0\)`), contains **omitted variable bias** -- - We can now try to *quantify* the omitted variable bias --- # Measuring Omitted Variable Bias I - Suppose the .hi-green[_true_ population model] of a relationship is: `$$\color{#047806}{Y_i=\beta_0+\beta_1 X_{1i}+\beta_2 X_{2i}+u_i}$$` - What happens when we run a regression and **omit** `\(X_{2i}\)`? -- - Suppose we estimate the following .hi-blue[omitted regression] of just `\(Y_i\)` on `\(X_{1i}\)` (omitting `\(X_{2i})\)`:^{.magenta[†]} `$$\color{#0047AB}{Y_i=\alpha_0+\alpha_1 X_{1i}+\nu_i}$$` .footnote[^{.magenta[†]} Note: I am using `\\(\alpha\\)`'s and `\\(\nu_i\\)` only to denote these are different estimates than the .hi-green[true] model `\\(\beta\\)`'s and `\\(u_i\\)`] --- # Measuring Omitted Variable Bias II - .hi-turquoise[**Key Question**:] are `\(X_{1i}\)` and `\(X_{2i}\)` correlated? -- - Run an .hi-purple[auxiliary regression] of `\(X_{2i}\)` on `\(X_{1i}\)` to see:^{.magenta[†]} `$$\color{#6A5ACD}{X_{2i}=\delta_0+\delta_1 X_{1i}+\tau_i}$$` -- - If `\(\color{#6A5ACD}{\delta_1}=0\)`, then `\(X_{1i}\)` and `\(X_{2i}\)` are *not* linearly related - If `\(|\color{#6A5ACD}{\delta_1}|\)` is very big, then `\(X_{1i}\)` and `\(X_{2i}\)` are strongly linearly related .footnote[^{.magenta[†]} Note: I am using `\\(\delta\\)`'s and `\\(\tau\\)` to differentiate estimates for this model.] --- # Measuring Omitted Variable Bias III .smallest[ - Now substitute our .hi-purple[auxiliary regression] between `\(X_{2i}\)` and `\(X_{1i}\)` into the .hi-green[*true* model]: - We know `\(\color{#6A5ACD}{X_{2i}=\delta_0+\delta_1 X_{1i}+\tau_i}\)` `$$\begin{align*} Y_i&=\beta_0+\beta_1 X_{1i}+\beta_2 \color{#6A5ACD}{X_{2i}}+u_i \\ \end{align*}$$` ] --- # Measuring Omitted Variable Bias III .smallest[ - Now substitute our .hi-purple[auxiliary regression] between `\(X_{2i}\)` and `\(X_{1i}\)` into the .hi-green[*true* model]: - We know `\(\color{#6A5ACD}{X_{2i}=\delta_0+\delta_1 X_{1i}+\tau_i}\)` `$$\begin{align*} Y_i&=\beta_0+\beta_1 X_{1i}+\beta_2 \color{#6A5ACD}{X_{2i}}+u_i \\ Y_i&=\beta_0+\beta_1 X_{1i}+\beta_2 \color{#6A5ACD}{\big(\delta_0+\delta_1 X_{1i}+\tau_i \big)}+u_i \\ \end{align*}$$` ] --- # Measuring Omitted Variable Bias III .smallest[ - Now substitute our .hi-purple[auxiliary regression] between `\(X_{2i}\)` and `\(X_{1i}\)` into the .hi-green[*true* model]: - We know `\(\color{#6A5ACD}{X_{2i}=\delta_0+\delta_1 X_{1i}+\tau_i}\)` `$$\begin{align*} Y_i&=\beta_0+\beta_1 X_{1i}+\beta_2 \color{#6A5ACD}{X_{2i}}+u_i \\ Y_i&=\beta_0+\beta_1 X_{1i}+\beta_2 \color{#6A5ACD}{\big(\delta_0+\delta_1 X_{1i}+\tau_i \big)}+u_i \\ Y_i&=(\beta_0+\beta_2 \color{#6A5ACD}{\delta_0})+(\beta_1+\beta_2 \color{#6A5ACD}{\delta_1})\color{#6A5ACD}{X_{1i}}+(\beta_2 \color{#6A5ACD}{\tau_i}+u_i)\\ \end{align*}$$` ] --- # Measuring Omitted Variable Bias III .smallest[ - Now substitute our .hi-purple[auxiliary regression] between `\(X_{2i}\)` and `\(X_{1i}\)` into the .hi-green[*true* model]: - We know `\(\color{#6A5ACD}{X_{2i}=\delta_0+\delta_1 X_{1i}+\tau_i}\)` `$$\begin{align*} Y_i&=\beta_0+\beta_1 X_{1i}+\beta_2 \color{#6A5ACD}{X_{2i}}+u_i \\ Y_i&=\beta_0+\beta_1 X_{1i}+\beta_2 \color{#6A5ACD}{\big(\delta_0+\delta_1 X_{1i}+\tau_i \big)}+u_i \\ Y_i&=(\underbrace{\beta_0+\beta_2 \color{#6A5ACD}{\delta_0}}_{\color{#0047AB}{\alpha_0}})+(\underbrace{\beta_1+\beta_2 \color{#6A5ACD}{\delta_1}}_{\color{#0047AB}{\alpha_1}})\color{#6A5ACD}{X_{1i}}+(\underbrace{\beta_2 \color{#6A5ACD}{\tau_i}+u_i}_{\color{#0047AB}{\nu_i}})\\ \end{align*}$$` - Now relabel each of the three terms as the OLS estimates `\((\alpha\)`'s) and error `\((\nu_i)\)` from the .hi-blue[omitted regression], so we again have: `$$\color{#0047AB}{Y_i=\alpha_0+\alpha_1X_{1i}+\nu_i}$$` ] -- .smallest[ - Crucially, this means that our OLS estimate for `\(X_{1i}\)` in the .hi-purple[omitted regression] is: `$$\color{#0047AB}{\alpha_1}=\beta_1+\beta_2 \color{#6A5ACD}{\delta_1}$$` ] --- # Measuring Omitted Variable Bias IV .smallest[ .center[ `\(\color{#0047AB}{\alpha_1}= \,\)`.green[`\\(\beta_1\\)`] `\(+\)` .red[`\\(\beta_2\\)`].purple[`\\(\delta_1\\)`] ] - The .hi-blue[Omitted Regression] OLS estimate for `\\(X_{1i}\\)`, `\\((\color{#0047AB}{\alpha_1})\\)` picks up *both*: ] -- .smallest[ 1) .green[The true effect of `\\(X_{1}\\)` on `\\(Y_i\\)`: `\\((\beta_1)\\)`] ] -- .smallest[ 2) .red[The true effect of `\\(X_{2}\\)` on `\\(Y_i\\)`: `\\((\beta_2)\\)`] - As pulled through .purple[the relationship between `\\(X_1\\)` and `\\(X_2\\)`: `\\((\delta_1)\\)`] ] -- .smallest[ - Recall our conditions for omitted variable bias from some variable `\(Z_i\)`: ] -- .smallest[ 1) `\(\mathbf{Z_i}\)` **must be a determinant of `\(Y_i\)`** `\(\implies\)` .red[`\\(\beta_2 \neq 0\\)`] ] -- .smallest[ 2) `\(\mathbf{Z_i}\)` **must be correlated with `\(X_i\)`** `\(\implies\)` .purple[`\\(\delta_1 \neq 0\\)`] ] -- .smallest[ - Otherwise, if `\(Z_i\)` does not fit these conditions, `\(\alpha_1=\beta_1\)` and the .hi-purple[omitted regression] is *unbiased*! ] --- # Measuring OVB in Our Class Size Example I - The .hi-green[“True” Regression] `\((Y_i\)` on `\(X_{1i}\)` and `\(X_{2i})\)` `$$\color{#047806}{\widehat{\text{Test Score}_i}=686.03-1.10\text{ STR}_i-0.65\text{ %EL}_i}$$` .center[ .quitesmall[ <div data-pagedtable="false"> <script data-pagedtable-source type="application/json"> {"columns":[{"label":["term"],"name":[1],"type":["chr"],"align":["left"]},{"label":["estimate"],"name":[2],"type":["dbl"],"align":["right"]},{"label":["std.error"],"name":[3],"type":["dbl"],"align":["right"]},{"label":["statistic"],"name":[4],"type":["dbl"],"align":["right"]},{"label":["p.value"],"name":[5],"type":["dbl"],"align":["right"]}],"data":[{"1":"(Intercept)","2":"686.0322487","3":"7.41131248","4":"92.565554","5":"3.871501e-280"},{"1":"str","2":"-1.1012959","3":"0.38027832","4":"-2.896026","5":"3.978056e-03"},{"1":"el_pct","2":"-0.6497768","3":"0.03934255","4":"-16.515879","5":"1.657506e-47"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}} </script> </div> ] ] --- # Measuring OVB in Our Class Size Example II - The .hi-blue[“Omitted” Regression] `\((Y_{i}\)` on just `\(X_{1i})\)` `$$\color{#0047AB}{\widehat{\text{Test Score}_i}=698.93-2.28\text{ STR}_i}$$` .center[ .quitesmall[ <div data-pagedtable="false"> <script data-pagedtable-source type="application/json"> {"columns":[{"label":["term"],"name":[1],"type":["chr"],"align":["left"]},{"label":["estimate"],"name":[2],"type":["dbl"],"align":["right"]},{"label":["std.error"],"name":[3],"type":["dbl"],"align":["right"]},{"label":["statistic"],"name":[4],"type":["dbl"],"align":["right"]},{"label":["p.value"],"name":[5],"type":["dbl"],"align":["right"]}],"data":[{"1":"(Intercept)","2":"698.932952","3":"9.4674914","4":"73.824514","5":"6.569925e-242"},{"1":"str","2":"-2.279808","3":"0.4798256","4":"-4.751327","5":"2.783307e-06"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}} </script> </div> ] ] --- # Measuring OVB in Our Class Size Example III - The .hi-purple[“Auxiliary” Regression] `\((X_{2i}\)` on `\(X_{1i})\)` `$$\color{#6A5ACD}{\widehat{\text{%EL}_i}=-19.85+1.81\text{ STR}_i}$$` .center[ .quitesmall[ <div data-pagedtable="false"> <script data-pagedtable-source type="application/json"> {"columns":[{"label":["term"],"name":[1],"type":["chr"],"align":["left"]},{"label":["estimate"],"name":[2],"type":["dbl"],"align":["right"]},{"label":["std.error"],"name":[3],"type":["dbl"],"align":["right"]},{"label":["statistic"],"name":[4],"type":["dbl"],"align":["right"]},{"label":["p.value"],"name":[5],"type":["dbl"],"align":["right"]}],"data":[{"1":"(Intercept)","2":"-19.854055","3":"9.1626044","4":"-2.166857","5":"0.0308099863"},{"1":"str","2":"1.813719","3":"0.4643735","4":"3.905733","5":"0.0001095165"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}} </script> </div> ] ] --- # Measuring OVB in Our Class Size Example IV .pull-left[ .center[ .smallest[ .hi-green[“True” Regression] `$$\widehat{\text{Test Score}_i}=686.03-1.10\text{ STR}_i-0.65\text{ %EL}$$` .hi-blue[“Omitted” Regression] `\(\widehat{\text{Test Score}_i}=698.93\color{#0047AB}{-2.28}\text{ STR}_i\)` .hi-purple[“Auxiliary” Regression] `$$\widehat{\text{%EL}_i}=-19.85+1.81\text{ STR}_i$$` ] ] ] .pull-right[ .smallest[ - Omitted Regression `\(\alpha_1\)` on STR is .blue[-2.28] ] ] --- # Measuring OVB in Our Class Size Example IV .pull-left[ .center[ .smallest[ .hi-green[“True” Regression] `$$\widehat{\text{Test Score}_i}=686.03 \color{#047806}{-1.10}\text{ STR}_i-0.65\text{ %EL}$$` .hi-blue[“Omitted” Regression] `\(\widehat{\text{Test Score}_i}=698.93\color{#0047AB}{-2.28} \text{ STR}_i\)` .hi-purple[“Auxiliary” Regression] `\(\widehat{\text{%EL}_i}=-19.85+1.81\text{ STR}_i\)` ] ] ] .pull-right[ .smallest[ - Omitted Regression `\(\alpha_1\)` on STR is .blue[-2.28] .center[ `$$\color{#0047AB}{\alpha_1}=\color{#047806}{\beta_1}+\color{#D7250E}{\beta_2} \color{#6A5ACD}{\delta_1}$$` ] - .green[The true effect of STR on Test Score: -1.10] ] ] --- # Measuring OVB in Our Class Size Example IV .pull-left[ .center[ .smallest[ .hi-green[“True” Regression] `$$\widehat{\text{Test Score}_i}=686.03 \color{#047806}{-1.10}\text{ STR}_i\color{#D7250E}{-0.65}\text{ %EL}$$` .hi-blue[“Omitted” Regression] `\(\widehat{\text{Test Score}_i}=698.93\color{#0047AB}{-2.28} \text{ STR}_i\)` .hi-purple[“Auxiliary” Regression] `\(\widehat{\text{%EL}_i}=-19.85+1.81\text{ STR}_i\)` ] ] ] .pull-right[ .smallest[ - Omitted Regression `\(\alpha_1\)` on STR is .blue[-2.28] .center[ `$$\color{#0047AB}{\alpha_1}=\color{#047806}{\beta_1}+\color{#D7250E}{\beta_2} \color{#6A5ACD}{\delta_1}$$` ] - .green[The true effect of STR on Test Score: -1.10] - .red[The true effect of %EL on Test Score: -0.65] ] ] --- # Measuring OVB in Our Class Size Example IV .pull-left[ .center[ .smallest[ .hi-green[“True” Regression] `$$\widehat{\text{Test Score}_i}=686.03 \color{#047806}{-1.10}\text{ STR}_i\color{#D7250E}{-0.65}\text{ %EL}$$` .hi-blue[“Omitted” Regression] `\(\widehat{\text{Test Score}_i}=698.93\color{#0047AB}{-2.28} \text{ STR}_i\)` .hi-purple[“Auxiliary” Regression] `\(\widehat{\text{%EL}_i}=-19.85+\color{#6A5ACD}{1.81}\text{ STR}_i\)` ] ] ] .pull-right[ .smallest[ - Omitted Regression `\(\alpha_1\)` on STR is .blue[-2.28] .center[ `$$\color{#0047AB}{\alpha_1}=\color{#047806}{\beta_1}+\color{#D7250E}{\beta_2} \color{#6A5ACD}{\delta_1}$$` ] - .green[The true effect of STR on Test Score: -1.10] - .red[The true effect of %EL on Test Score: -0.65] - .purple[The relationship between STR and %EL: 1.81] ] ] --- # Measuring OVB in Our Class Size Example IV .pull-left[ .center[ .smallest[ .hi-green[“True” Regression] `$$\widehat{\text{Test Score}_i}=686.03 \color{#047806}{-1.10}\text{ STR}_i\color{#D7250E}{-0.65}\text{ %EL}$$` .hi-blue[“Omitted” Regression] `\(\widehat{\text{Test Score}_i}=698.93\color{#0047AB}{-2.28} \text{ STR}_i\)` .hi-purple[“Auxiliary” Regression] `\(\widehat{\text{%EL}_i}=-19.85+\color{#6A5ACD}{1.81}\text{ STR}_i\)` ] ] ] .pull-right[ .smallest[ - Omitted Regression `\(\alpha_1\)` on STR is .blue[-2.28] .center[ `$$\color{#0047AB}{\alpha_1}=\color{#047806}{\beta_1}+\color{#D7250E}{\beta_2} \color{#6A5ACD}{\delta_1}$$` ] - .green[The true effect of STR on Test Score: -1.10] - .red[The true effect of %EL on Test Score: -0.65] - .purple[The relationship between STR and %EL: 1.81] - So, for the .hi-blue[omitted regression]: .center[ `$$\color{#0047AB}{-2.28}=\color{#047806}{-1.10}+\color{#D7250E}{(-0.65)} \color{#6A5ACD}{(1.81)}$$` ] ] ] --- # Measuring OVB in Our Class Size Example IV .pull-left[ .center[ .smallest[ .hi-green[“True” Regression] `$$\widehat{\text{Test Score}_i}=686.03 \color{#047806}{-1.10}\text{ STR}_i\color{#D7250E}{-0.65}\text{ %EL}$$` .hi-blue[“Omitted” Regression] `\(\widehat{\text{Test Score}_i}=698.93\color{#0047AB}{-2.28} \text{ STR}_i\)` .hi-purple[“Auxiliary” Regression] `\(\widehat{\text{%EL}_i}=-19.85+\color{#6A5ACD}{1.81}\text{ STR}_i\)` ] ] ] .pull-right[ .smallest[ - Omitted Regression `\(\alpha_1\)` on STR is .blue[-2.28] .center[ `$$\color{#0047AB}{\alpha_1}=\color{#047806}{\beta_1}+\color{#D7250E}{\beta_2} \color{#6A5ACD}{\delta_1}$$` ] - .green[The true effect of STR on Test Score: -1.10] - .red[The true effect of %EL on Test Score: -0.65] - .purple[The relationship between STR and %EL: 1.81] - So, for the .hi-blue[omitted regression]: .center[ `$$\color{#0047AB}{-2.28}=\color{#047806}{-1.10}+\underbrace{\color{#D7250E}{(-0.65)} \color{#6A5ACD}{(1.81)}}_{O.V.Bias=\mathbf{-1.18}}$$` ] ] ] --- class: inverse, center, middle # Precision of `\(\hat{\beta_j}\)` --- # Precision of `\(\hat{\beta_j}\)` I .pull-left[ - `\(\sigma_{\hat{\beta_j}}\)`; how **precise** are our estimates? - <span class="hi">Variance `\(\sigma^2_{\hat{\beta_j}}\)`</span> or <span class="hi">standard error `\(\sigma_{\hat{\beta_j}}\)`</span> ] .pull-right[ <img src="3.4-slides_files/figure-html/unnamed-chunk-5-1.png" width="504" /> ] --- # Precision of `\(\hat{\beta_j}\)` II .pull-left[ `$$var(\hat{\beta_j})=\underbrace{\color{#6A5ACD}{\frac{1}{1-R^2_j}}}_{\color{#6A5ACD}{VIF}} \times \frac{(SER)^2}{n \times var(X)}$$` `$$se(\hat{\beta_j})=\sqrt{var(\hat{\beta_1})}$$` ] .pull-right[ .smallest[ - Variation in `\(\hat{\beta_j}\)` is affected by **four** things now^{.magenta[†]}: 1. .hi-purple[Goodness of fit of the model (SER)] - Larger `\(SER\)` `\(\rightarrow\)` larger `\(var(\hat{\beta_j})\)` 2. .hi-purple[Sample size, *n*] - Larger `\(n\)` `\(\rightarrow\)` smaller `\(var(\hat{\beta_j})\)` 3. .hi-purple[Variance of X] - Larger `\(var(X)\)` `\(\rightarrow\)` smaller `\(var(\hat{\beta_j})\)` 4. .hi-purple[Variance Inflation Factor] `\(\color{#6A5ACD}{\frac{1}{(1-R^2_j)}}\)` - Larger `\(VIF\)`, larger `\(var(\hat{\beta_j})\)` - **This is the only new effect** ] ] .footnote[^{.magenta[†]} See [Class 2.5](/content/2.5-content) for a reminder of variation with just one X variable.] --- # VIF and Multicollinearity I - Two *independent* (X) variables are .hi[multicollinear]: `$$cor(X_j, X_l) \neq 0 \quad \forall j \neq l$$` -- - .hi-purple[Multicollinearity between X variables does *not bias* OLS estimates] - Remember, we pulled another variable out of `\(u\)` into the regression - If it were omitted, then it *would* cause omitted variable bias! -- - .hi-purple[Multicollinearity does *increase the variance* of each estimate] by `$$VIF=\frac{1}{(1-R^2_j)}$$` --- # VIF and Multicollinearity II .smallest[ `$$VIF=\frac{1}{(1-R^2_j)}$$` - `\(R^2_j\)` is the `\(R^2\)` from an .hi-blue[auxiliary regression] of `\(X_j\)` on all other regressors `\((X\)`’s) ] -- .smallest[ .bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[ .green[**Example**]: Suppose we have a regression with three regressors `\((k=3)\)`: `$$Y_i=\beta_0+\beta_1X_{1i}+\beta_2X_{2i}+\beta_3X_{3i}$$` ] ] -- .smallest[ - There will be three different `\(R^2_j\)`'s, one for each regressor: `$$\begin{align*} R^2_1 \text{ for } X_{1i}&=\gamma+\gamma X_{2i} + \gamma X_{3i} \\ R^2_2 \text{ for } X_{2i}&=\zeta_0+\zeta_1 X_{1i} + \zeta_2 X_{3i} \\ R^2_3 \text{ for } X_{3i}&=\eta_0+\eta_1 X_{1i} + \eta_2 X_{2i} \\ \end{align*}$$` ] --- # VIF and Multicollinearity III .smallest[ `$$VIF=\frac{1}{(1-R^2_j)}$$` - `\(R^2_j\)` is the `\(R^2\)` from an .hi-blue[auxiliary regression] of `\(X_j\)` on all other regressors `\((X\)`'s) - The `\(R_j^2\)` tells us .hi-purple[how much *other* regressors explain regressor `\\(X_j\\)`] - .hi-turquoise[Key Takeaway]: If other `\(X\)` variables explain `\(X_j\)` well (high `\(R^2_J\)`), it will be harder to tell how *cleanly* `\(X_j \rightarrow Y_i\)`, and so `\(var(\hat{\beta_j})\)` will be higher ] --- # VIF and Multicollinearity IV - Common to calculate the .hi[Variance Inflation Factor (VIF)] for each regressor: `$$VIF=\frac{1}{(1-R^2_j)}$$` - VIF quantifies the factor (scalar) by which `\(var(\hat{\beta_j})\)` increases because of multicollinearity - e.g. VIF of 2, 3, etc. `\(\implies\)` variance increases by 2x, 3x, etc. -- - Baseline: `\(R^2_j=0\)` `\(\implies\)` *no* multicollinearity `\(\implies VIF = 1\)` (no inflation) -- - Larger `\(R^2_j\)` `\(\implies\)` larger VIF - Rule of thumb: `\(VIF>10\)` is problematic --- # VIF and Multicollinearity V .pull-left[ .smallest[ .code50[ ```r # scatterplot of X2 on X1 ggplot(data=CASchool, aes(x=str,y=el_pct))+ geom_point(color="blue")+ geom_smooth(method="lm", color="red")+ scale_y_continuous(labels=function(x){paste0(x,"%")})+ labs(x = expression(paste("Student to Teacher Ratio, ", X[1])), y = expression(paste("Percentage of ESL Students, ", X[2])), title = "Multicollinearity Between Our Independent Variables")+ ggthemes::theme_pander(base_family = "Fira Sans Condensed", base_size=16) ``` ```r # Make a correlation table CASchool %>% select(testscr, str, el_pct) %>% cor() ``` ``` ## testscr str el_pct ## testscr 1.0000000 -0.2263628 -0.6441237 ## str -0.2263628 1.0000000 0.1876424 ## el_pct -0.6441237 0.1876424 1.0000000 ``` - Cor(STR, %EL) = -0.644 ] ] ] .pull-right[ <img src="3.4-slides_files/figure-html/unnamed-chunk-7-1.png" width="504" /> ] --- # VIF and Multicollinearity in R I ```r # our multivariate regression elreg <- lm(testscr ~ str + el_pct, data = CASchool) # use the "car" package for VIF function library("car") # syntax: vif(lm.object) vif(elreg) ``` ``` ## str el_pct ## 1.036495 1.036495 ``` -- .smaller[ - `\(var(\hat{\beta_1})\)` on `str` increases by **1.036** times (3.6%) due to multicollinearity with `el_pct` - `\(var(\hat{\beta_2})\)` on `el_pct` increases by **1.036** times (3.6%) due to multicollinearity with `str` ] --- # VIF and Multicollinearity in R II - Let's calculate VIF manually to see where it comes from: -- .smallest[ .code60[ ```r # run auxiliary regression of x2 on x1 auxreg <- lm(el_pct ~ str, data = CASchool) # use broom package's tidy() command (cleaner) library(broom) # load broom tidy(auxreg) # look at reg output ``` <div data-pagedtable="false"> <script data-pagedtable-source type="application/json"> {"columns":[{"label":["term"],"name":[1],"type":["chr"],"align":["left"]},{"label":["estimate"],"name":[2],"type":["dbl"],"align":["right"]},{"label":["std.error"],"name":[3],"type":["dbl"],"align":["right"]},{"label":["statistic"],"name":[4],"type":["dbl"],"align":["right"]},{"label":["p.value"],"name":[5],"type":["dbl"],"align":["right"]}],"data":[{"1":"(Intercept)","2":"-19.854055","3":"9.1626044","4":"-2.166857","5":"0.0308099863"},{"1":"str","2":"1.813719","3":"0.4643735","4":"3.905733","5":"0.0001095165"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}} </script> </div> ] ] --- # VIF and Multicollinearity in R III .smallest[ .code60[ ```r glance(auxreg) # look at aux reg stats for R^2 ``` <div data-pagedtable="false"> <script data-pagedtable-source type="application/json"> {"columns":[{"label":["r.squared"],"name":[1],"type":["dbl"],"align":["right"]},{"label":["adj.r.squared"],"name":[2],"type":["dbl"],"align":["right"]},{"label":["sigma"],"name":[3],"type":["dbl"],"align":["right"]},{"label":["statistic"],"name":[4],"type":["dbl"],"align":["right"]},{"label":["p.value"],"name":[5],"type":["dbl"],"align":["right"]},{"label":["df"],"name":[6],"type":["dbl"],"align":["right"]},{"label":["logLik"],"name":[7],"type":["dbl"],"align":["right"]},{"label":["AIC"],"name":[8],"type":["dbl"],"align":["right"]},{"label":["BIC"],"name":[9],"type":["dbl"],"align":["right"]},{"label":["deviance"],"name":[10],"type":["dbl"],"align":["right"]},{"label":["df.residual"],"name":[11],"type":["int"],"align":["right"]},{"label":["nobs"],"name":[12],"type":["int"],"align":["right"]}],"data":[{"1":"0.03520966","2":"0.03290155","3":"17.98259","4":"15.25475","5":"0.0001095165","6":"1","7":"-1808.502","8":"3623.003","9":"3635.124","10":"135170.2","11":"418","12":"420"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}} </script> </div> ```r # extract our R-squared from aux regression (R_j^2) aux_r_sq <- glance(auxreg) %>% select(r.squared) aux_r_sq # look at it ``` <div data-pagedtable="false"> <script data-pagedtable-source type="application/json"> {"columns":[{"label":["r.squared"],"name":[1],"type":["dbl"],"align":["right"]}],"data":[{"1":"0.03520966"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}} </script> </div> ] ] --- # VIF and Multicollinearity in R IV ```r # calculate VIF manually our_vif <- 1 / (1 - aux_r_sq) # VIF formula our_vif ``` <div data-pagedtable="false"> <script data-pagedtable-source type="application/json"> {"columns":[{"label":["r.squared"],"name":[1],"type":["dbl"],"align":["right"]}],"data":[{"1":"1.036495"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}} </script> </div> - Again, multicollinearity between the two `\(X\)` variables inflates the variance on each by 1.036 times --- # VIF and Multicollinearity: Another Example I .bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[ .green[**Example**:] What about district expenditures per student? ] ```r CASchool %>% select(testscr, str, expn_stu) %>% cor() ``` ``` ## testscr str expn_stu ## testscr 1.0000000 -0.2263628 0.1912728 ## str -0.2263628 1.0000000 -0.6199821 ## expn_stu 0.1912728 -0.6199821 1.0000000 ``` --- # VIF and Multicollinearity: Another Example II .pull-left[ .code60[ ```r ggplot(data=CASchool, aes(x=str,y=expn_stu))+ geom_point(color="blue")+ geom_smooth(method="lm", color="red")+ scale_y_continuous(labels = scales::dollar)+ labs(x = "Student to Teacher Ratio", y = "Expenditures per Student")+ ggthemes::theme_pander(base_family = "Fira Sans Condensed", base_size=14) ``` ] ] .pull-right[ <img src="3.4-slides_files/figure-html/unnamed-chunk-13-1.png" width="504" /> ] --- # VIF and Multicollinearity: Another Example III .pull-left[ 1. `\(cor(\text{Test score, expn})\neq0\)` 2. `\(cor(\text{STR, expn})\neq 0\)` ] .pull-right[ <!-- --> ] --- # VIF and Multicollinearity: Another Example III .pull-left[ 1. `\(cor(\text{Test score, expn})\neq0\)` 2. `\(cor(\text{STR, expn})\neq 0\)` - Omitting `\(expn\)` will **bias** `\(\hat{\beta_1}\)` on STR ] .pull-right[ <!-- --> ] --- # VIF and Multicollinearity: Another Example III .pull-left[ 1. `\(cor(\text{Test score, expn})\neq0\)` 2. `\(cor(\text{STR, expn})\neq 0\)` - Omitting `\(expn\)` will **bias** `\(\hat{\beta_1}\)` on STR - *Including* `\(expn\)` will *not* bias `\(\hat{\beta_1}\)` on STR, but *will* make it less precise (higher variance) ] .pull-right[ <!-- --> ] --- # VIF and Multicollinearity: Another Example III .pull-left[ - Data tells us little about the effect of a change in `\(STR\)` holding `\(expn\)` constant - Hard to know what happens to test scores when high `\(STR\)` AND high `\(expn\)` and vice versa (*they rarely happen simultaneously*)! ] .pull-right[ <img src="3.4-slides_files/figure-html/unnamed-chunk-17-1.png" width="504" /> ] --- # VIF and Multicollinearity: Another Example IV .pull-left[ .quitesmall[ .code60[ ```r expreg <- lm(testscr ~ str + expn_stu, data = CASchool) expreg %>% tidy() ``` <div data-pagedtable="false"> <script data-pagedtable-source type="application/json"> {"columns":[{"label":["term"],"name":[1],"type":["chr"],"align":["left"]},{"label":["estimate"],"name":[2],"type":["dbl"],"align":["right"]},{"label":["std.error"],"name":[3],"type":["dbl"],"align":["right"]},{"label":["statistic"],"name":[4],"type":["dbl"],"align":["right"]},{"label":["p.value"],"name":[5],"type":["dbl"],"align":["right"]}],"data":[{"1":"(Intercept)","2":"675.577173851","3":"19.562221636","4":"34.534788","5":"2.244554e-124"},{"1":"str","2":"-1.763215599","3":"0.610913641","4":"-2.886195","5":"4.101913e-03"},{"1":"expn_stu","2":"0.002486571","3":"0.001823105","4":"1.363921","5":"1.733281e-01"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}} </script> </div> ] ] ] -- .pull-right[ .code60[ ```r expreg %>% vif() ``` ``` ## str expn_stu ## 1.624373 1.624373 ``` ] - Including `expn_stu` increases variance of `\\(\hat{\beta_1}\\)` and `\\(\hat{\beta_2}\\)` by 1.62x (62%) ] --- # Multicollinearity Increases Variance .pull-left[ .code50[ ```r library(huxtable) huxreg("Model 1" = school_reg, "Model 2" = expreg, coefs = c("Intercept" = "(Intercept)", "Class Size" = "str", "Expenditures per Student" = "expn_stu"), statistics = c("N" = "nobs", "R-Squared" = "r.squared", "SER" = "sigma"), number_format = 2) ``` ] - We can see `\\(SE(\hat{\beta_1})\\)` on `str` increases from 0.48 to 0.61 when we add `expn_stu` ] .pull-right[ .quitesmall[ <table class="huxtable" style="border-collapse: collapse; border: 0px; margin-bottom: 2em; margin-top: 2em; ; margin-left: auto; margin-right: auto; " id="tab:unnamed-chunk-20"> <col><col><col><tr> <th style="vertical-align: top; text-align: center; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0pt 0pt; padding: 6pt 6pt 6pt 6pt; font-weight: normal;"></th><th style="vertical-align: top; text-align: center; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0.4pt 0pt; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">Model 1</th><th style="vertical-align: top; text-align: center; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0.4pt 0pt; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">Model 2</th></tr> <tr> <th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">Intercept</th><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0pt 0pt; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">698.93 ***</td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0pt 0pt; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">675.58 ***</td></tr> <tr> <th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;"></th><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(9.47)   </td><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(19.56)   </td></tr> <tr> <th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">Class Size</th><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">-2.28 ***</td><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">-1.76 ** </td></tr> <tr> <th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;"></th><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(0.48)   </td><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(0.61)   </td></tr> <tr> <th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">Expenditures per Student</th><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">       </td><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">0.00    </td></tr> <tr> <th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;"></th><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.4pt 0pt; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">       </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.4pt 0pt; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">(0.00)   </td></tr> <tr> <th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">N</th><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0pt 0pt; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">420       </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0.4pt 0pt 0pt 0pt; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">420       </td></tr> <tr> <th style="vertical-align: top; text-align: left; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">R-Squared</th><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">0.05    </td><td style="vertical-align: top; text-align: right; white-space: normal; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">0.06    </td></tr> <tr> <th style="vertical-align: top; text-align: left; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.8pt 0pt; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">SER</th><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.8pt 0pt; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">18.58    </td><td style="vertical-align: top; text-align: right; white-space: normal; border-style: solid solid solid solid; border-width: 0pt 0pt 0.8pt 0pt; padding: 6pt 6pt 6pt 6pt; font-weight: normal;">18.56    </td></tr> <tr> <th colspan="3" style="vertical-align: top; text-align: left; white-space: normal; border-style: solid solid solid solid; border-width: 0.8pt 0pt 0pt 0pt; padding: 6pt 6pt 6pt 6pt; font-weight: normal;"> *** p < 0.001; ** p < 0.01; * p < 0.05.</th></tr> </table> ] ] --- # Perfect Multicollinearity - .hi[*Perfect* multicollinearity] is when a regressor is an exact linear function of (an)other regressor(s) -- `$$\widehat{Sales} = \hat{\beta_0}+\hat{\beta_1}\text{Temperature (C)} + \hat{\beta_2}\text{Temperature (F)}$$` -- `$$\text{Temperature (F)}=32+1.8*\text{Temperature (C)}$$` -- - `\(cor(\text{temperature (F), temperature (C)})=1\)` -- - `\(R^2_j=1\)` is implying `\(VIF=\frac{1}{1-1}\)` and `\(var(\hat{\beta_j})=0\)`! -- - .hi-purple[This is fatal for a regression] - A logical impossiblity, **always caused by human error** --- # Perfect Multicollinearity: Example .bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[ .green[**Example**:] `$$\widehat{TestScore_i} = \hat{\beta_0}+\hat{\beta_1}STR_i +\hat{\beta_2}\%EL+\hat{\beta_3}\%EF$$` ] - `\(\%EL\)`: the percentage of students learning English - `\(\%EF\)`: the percentage of students fluent in English - `\(\%EF=100-\%EL\)` - `\(|cor(\%EF, \%EL)|=1\)` --- # Perfect Multicollinearity Example II ```r # generate %EF variable from %EL CASchool_ex <- CASchool %>% mutate(ef_pct = 100 - el_pct) # get correlation between %EL and %EF CASchool_ex %>% summarize(cor = cor(ef_pct, el_pct)) ``` ``` ## # A tibble: 1 × 1 ## cor ## <dbl> ## 1 -1 ``` --- # Perfect Multicollinearity Example III .pull-left[ .code60[ ```r ggplot(data = CASchool_ex)+ aes(x = el_pct, y = ef_pct)+ geom_point(color = "blue")+ scale_x_continuous(labels = scales::percent_format(scale = 1))+ scale_y_continuous(labels = scales::percent_format(scale = 1))+ labs(x = "Percent of ESL Students", y = "Percent of Non-ESL Students")+ ggthemes::theme_pander(base_family = "Fira Sans Condensed", base_size=16) ``` ] ] .pull-right[ <img src="3.4-slides_files/figure-html/unnamed-chunk-22-1.png" width="504" /> ] --- # Perfect Multicollinearity Example IV .pull-left[ .quitesmall[ .code60[ ```r mcreg <- lm(testscr ~ str + el_pct + ef_pct, data = CASchool_ex) summary(mcreg) ``` ``` ## ## Call: ## lm(formula = testscr ~ str + el_pct + ef_pct, data = CASchool_ex) ## ## Residuals: ## Min 1Q Median 3Q Max ## -48.845 -10.240 -0.308 9.815 43.461 ## ## Coefficients: (1 not defined because of singularities) ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 686.03225 7.41131 92.566 < 2e-16 *** ## str -1.10130 0.38028 -2.896 0.00398 ** ## el_pct -0.64978 0.03934 -16.516 < 2e-16 *** ## ef_pct NA NA NA NA ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 14.46 on 417 degrees of freedom ## Multiple R-squared: 0.4264, Adjusted R-squared: 0.4237 ## F-statistic: 155 on 2 and 417 DF, p-value: < 2.2e-16 ``` ] ] ] .pull-right[ .quitesmall[ .code60[ ```r mcreg %>% tidy() ``` ``` ## # A tibble: 4 × 5 ## term estimate std.error statistic p.value ## <chr> <dbl> <dbl> <dbl> <dbl> ## 1 (Intercept) 686. 7.41 92.6 3.87e-280 ## 2 str -1.10 0.380 -2.90 3.98e- 3 ## 3 el_pct -0.650 0.0393 -16.5 1.66e- 47 ## 4 ef_pct NA NA NA NA ``` ] ] ] - Note `R` *drops* one of the multicollinear regressors (`ef_pct`) if you include both 🤡 --- class: inverse, center, middle # A Summary of Multivariate OLS Estimator Properties --- # A Summary of Multivariate OLS Estimator Properties .smallest[ - `\(\hat{\beta_j}\)` on `\(X_j\)` is biased only if there is an omitted variable `\((Z)\)` such that: 1. `\(cor(Y,Z)\neq 0\)` 2. `\(cor(X_j,Z)\neq 0\)` - If `\(Z\)` is *included* and `\(X_j\)` is collinear with `\(Z\)`, this does *not* cause a bias - `\(var[\hat{\beta_j}]\)` and `\(se[\hat{\beta_j}]\)` measure precision (or uncertainty) of estimate: ] -- .smallest[ `$$var[\hat{\beta_j}]=\frac{1}{(1-R^2_j)}*\frac{SER^2}{n \times var[X_j]}$$` - VIF from multicollinearity: `\(\frac{1}{(1-R^2_j)}\)` - `\(R_j^2\)` for auxiliary regression of `\(X_j\)` on all other `\(X\)`'s - mutlicollinearity does not bias `\(\hat{\beta_j}\)` but raises its variance - *perfect* multicollinearity if `\(X\)`'s are linear function of others ] --- class: inverse, center, middle # Updated Measures of Fit --- # (Updated) Measures of Fit - Again, how well does a linear model fit the data? - How much variation in `\(Y_i\)` is “explained” by variation in the model `\((\hat{Y_i})\)`? -- `$$\begin{align*} Y_i&=\hat{Y_i}+\hat{u_i}\\ \hat{u_i}&= Y_i-\hat{Y_i}\\ \end{align*}$$` --- # (Updated) Measures of Fit: SER - Again, the .hi[Standard errror of the regression (SER)] estimates the standard error of `\(u\)` `$$SER=\frac{SSE}{n-\mathbf{k}-1}$$` - A measure of the spread of the observations around the regression line (in units of `\(Y\)`), the average "size" of the residual - .hi-purple[Only new change:] divided by `\(n-\color{#6A5ACD}{k}-1\)` due to use of `\(k+1\)` degrees of freedom to first estimate `\(\beta_0\)` and then all of the other `\(\beta\)`'s for the `\(k\)` number of regressors^{.magenta[†]} .footnote[^{.magenta[†]} Again, because your textbook defines *k* as including the constant, the denominator would be *n-k* instead of *n-k-1*.] --- # (Updated) Measures of Fit: `\(R^2\)` `$$\begin{align*} R^2&=\frac{ESS}{TSS}\\ &=1-\frac{SSE}{TSS}\\ &=(r_{X,Y})^2 \\ \end{align*}$$` - Again, `\(R^2\)` is fraction of total variation in `\(Y_i\)` (“total sum of squares”) that is explained by variation in predicted values `\((\hat{Y_i}\)`, i.e. our model (“explained sum of squares”) `$$\frac{var(\hat{Y})}{var(Y)}$$` --- # Visualizing `\(R^2\)` .pull-left[ .quitesmall[ - **Total Variation in Y**: Areas .hi-red[A] + D + E + G `$$TSS = \sum^n_{i=1}(Y_i-\bar{Y})^2$$` - .hi-purple[Variation in Y explained by X1 and X2]: Areas D + E + G `$$ESS = \sum^n_{i=1}(\hat{Y_i}-\bar{Y})^2$$` - .hi-red[Unexplained variation in Y]: .hi-red[Area A] `$$SSE = \sum^n_{i=1}(\hat{u_i})^2$$` [Compare with one X variable](https://metricsf21.classes.ryansafner.com/slides/2.4-slides#17) ] ] .pull-right[ .smallest[ `$$R^2 = \frac{ESS}{TSS} = \frac{D+E+G}{\color{red}{A}+D+E+G}$$` ] <!-- --> ] --- # Visualizing `\(R^2\)` .pull-left[ .quitesmall[ .code60[ ```r # make a function to calc. sum of sq. devs sum_sq <- function(x){sum((x - mean(x))^2)} # find total sum of squares TSS <- elreg %>% augment() %>% summarize(TSS = sum_sq(testscr)) # find explained sum of squares ESS <- elreg %>% augment() %>% summarize(TSS = sum_sq(.fitted)) # look at them and divide to get R^2 tribble( ~ESS, ~TSS, ~R_sq, ESS, TSS, ESS/TSS ) %>% knitr::kable() ``` |ESS |TSS |R_sq | |:-------|:--------|:---------| |64864.3 |152109.6 |0.4264314 | ] ] ] .pull-right[ .smallest[ `$$R^2 = \frac{ESS}{TSS} = \frac{D+E+G}{\color{red}{A}+D+E+G}=0.426$$` ] <!-- --> ] --- # (Updated) Measures of Fit: Adjusted `\(\bar{R}^2\)` .smallest[ - Problem: `\(R^2\)` **mechanically** increases *every* time a new variable is added (it reduces SSE!) - Think in the diagram: more area of `\(Y\)` covered by more `\(X\)` variables! - This does **not** mean adding a variable *improves the fit of the model* per se, `\(R^2\)` gets .hi-turquoise[inflated] ] -- .smallest[ - We correct for this effect with the .hi[adjusted `\\(\bar{R}^2\\)`] which penalizes adding new variables: `$$\bar{R}^2 = 1- \underbrace{\frac{n-1}{n-k-1}}_{penalty} \times \frac{SSE}{TSS}$$` - In the end, recall `\(R^2\)` .hi-turquoise[was never that useful]^{.magenta[†]}, so don't worry about the formula - Large sample sizes `\((n)\)` make `\(R^2\)` and `\(\bar{R}^2\)` very close ] .source[^{.magenta[†]} ...for measuring causal effects (our goal). It *is* useful if you care about prediction [instead](https://metricsf21.classes.ryansafner.com/slides/3.1-slides#3)!] --- # In R (base) .pull-left[ .quitesmall[ ``` ## ## Call: ## lm(formula = testscr ~ str + el_pct, data = CASchool) ## ## Residuals: ## Min 1Q Median 3Q Max ## -48.845 -10.240 -0.308 9.815 43.461 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 686.03225 7.41131 92.566 < 2e-16 *** ## str -1.10130 0.38028 -2.896 0.00398 ** ## el_pct -0.64978 0.03934 -16.516 < 2e-16 *** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 14.46 on 417 degrees of freedom ## Multiple R-squared: 0.4264, Adjusted R-squared: 0.4237 ## F-statistic: 155 on 2 and 417 DF, p-value: < 2.2e-16 ``` ] ] .pull-right[ .smallest[ - Base `\(R^2\)` (`R` calls it “`Multiple R-squared`”) went up - `Adjusted R-squared` went down ] ] --- # In R (broom) .pull-left[ .quitesmall[ ```r elreg %>% glance() ``` ``` ## # A tibble: 1 × 12 ## r.squared adj.r.squared sigma statistic p.value df logLik AIC BIC ## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> ## 1 0.426 0.424 14.5 155. 4.62e-51 2 -1717. 3441. 3457. ## # … with 3 more variables: deviance <dbl>, df.residual <int>, nobs <int> ``` ] ] --- # Coefficient Plots .pull-left[ - The `modelsummary` package has a great command `modelplot()` for quickly making coefficient plots - [Learn more](https://vincentarelbundock.github.io/modelsummary/articles/modelplot.html) .code60[ ```r library(modelsummary) modelplot(elreg, # our regression object coef_omit = 'Intercept') # don't show intercept ``` ] ] .pull-right[ <!-- --> ] --- # Modelsummary .pull-left[ .smallest[ - The `modelsummary` package also is a good alternative to `huxtable` for making regression tables (that's growing on me): - [Learn more](https://vincentarelbundock.github.io/modelsummary/articles/modelsummary.html) ] .code40[ ```r modelsummary(models = list("Base Model" = school_reg, "Multivariate Model" = elreg), fmt = 2, # round to 2 decimals output = "html", coef_rename = c("(Intercept)" = "Constant", "str" = "STR", "el_pct" = "% ESL Students"), gof_map = list( list("raw" = "nobs", "clean" = "N", "fmt" = 0), list("raw" = "r.squared", "clean" = "R<sup>2</sup>", "fmt" = 2), list("raw" = "adj.r.squared", "clean" = "Adj. R<sup>2</sup>", "fmt" = 2), list("raw" = "sigma", "clean" = "SER", "fmt" = 2) ), escape = FALSE, stars = TRUE ) ``` ] ] .pull-right[ .regtable[ .smallest[ <table style="NAborder-bottom: 0; width: auto !important; margin-left: auto; margin-right: auto;" class="table"> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:center;"> Base Model </th> <th style="text-align:center;"> Multivariate Model </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> Constant </td> <td style="text-align:center;"> 698.93*** </td> <td style="text-align:center;"> 686.03*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (9.47) </td> <td style="text-align:center;"> (7.41) </td> </tr> <tr> <td style="text-align:left;"> STR </td> <td style="text-align:center;"> -2.28*** </td> <td style="text-align:center;"> -1.10** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (0.48) </td> <td style="text-align:center;"> (0.38) </td> </tr> <tr> <td style="text-align:left;"> % ESL Students </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> -0.65*** </td> </tr> <tr> <td style="text-align:left;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> </td> <td style="text-align:center;box-shadow: 0px 1px"> (0.04) </td> </tr> <tr> <td style="text-align:left;"> N </td> <td style="text-align:center;"> 420 </td> <td style="text-align:center;"> 420 </td> </tr> <tr> <td style="text-align:left;"> R² </td> <td style="text-align:center;"> 0.05 </td> <td style="text-align:center;"> 0.43 </td> </tr> <tr> <td style="text-align:left;"> Adj. R² </td> <td style="text-align:center;"> 0.05 </td> <td style="text-align:center;"> 0.42 </td> </tr> <tr> <td style="text-align:left;"> SER </td> <td style="text-align:center;"> 18.58 </td> <td style="text-align:center;"> 14.46 </td> </tr> </tbody> <tfoot><tr><td style="padding: 0; " colspan="100%">  + p &lt; 0.1, * p &lt; 0.05, ** p &lt; 0.01, *** p &lt; 0.001</td></tr></tfoot> </table> ] ] ]