Ch3: Multiple Regression Analysis: Estimation

TOC

Necessity for Multiple Regression Model Estimation and Interpretation 1. Esitmation 2. Interpretation Properties of OLS Estimators 1. Goodness of Fit 2. Expected Value and Variance Gauss-Markov Theorem A Few Practical Issues 1. Excluding a relevant variable 2. Including irrelevant variables 3. Multicollinearity

Necessity for Multiple Regression Model

Descriptive analysis: sometimes we want to estimate the conditional mean of on multiple variables.

Causal estimation: we know that something other than may affect , so we explicitly control them.

Forecasting: we want to use more variables to better predict .

Estimation and Interpretation

1. Esitmation

Zero conditional mean assumption

Thus,

and

1.1 Sample Analogue

1.2 Ordianry Least Squares

define the fitted value: , thus, the residual is

The sum of residual squares

using OLS, we derive

The first order conditions are

📌

Above two methods to estimate coefficients are mathematically equivalent. We can solve them by hand or simply using STATA: reg y x1 x2 x3

1.3 Partialling Out Method

can be obtained by:

Regress on other independent variables (including the constant), obtain the residual .

Regress on (including the constant). The resulting slope coefficient is .

💡

- The residual are the part of that is uncorrelated with other . It is after the effects of other have been partialled out, or netted out. - Thus, measures the sample relationship between and after other has been partialled out.

1.4 Frisch-Watgh-Lovell Theorem

Regress on other independent variables (including the constant), obtain the residual .

Regress on other independent variables (including the constant), obtain the residual .

Regress on . The resulting slope coefficient is .

Note that when we use F-W-L theorem, in the final step, wheter there is a intercept in the model does not affect since

2. Interpretation

Causal Perspective

thus, , when is held fixed, .

Therefor, the coefficient of represents holding fixed other factors, the change in when

increases by on unit.

Properties of OLS Estimators

1. Goodness of Fit

Similar to the simple linear regression model, we can prove that:

define .

Adjusted

For the same model, we can reduce by adding more independent variables, for example

This implies that adding more independent variables will never reduce the .

Use adjusted instead:

is the size of sample, represents the number of independent variales (excluding the constant).

In this way, we add penalty when adding more independent variables.

2. Expected Value and Variance

2.1 Expected Value

Under some assumptions, the OLS estimator can have the desired feature — Unbiasedness

MLR.1 Linear in Parameters:

Coefficients are linear similar to the restriction of SLR.1.

MLR.2 Random Sampling:

Data () in the sample are iid (following the population model).

MLR.3 No Perfect Collinearity

Any independer variable should not be an exact linear combination of the other independent variables.

MLR.3 is different from SLR.3. However, it is related to the assumption that there is variation in . If has no variation, then it is perfectly correlated with the constant.

MLR.4 Zero Contional Mean —

Under MLR.1 - MLR.4

Proof

2.2 Expected Value

Add another assumption:

MLR.5 Homoskedasticity —

☝🏻

MLR.1 - MLR.5 are collectively known as the Gauss-Markov assumptions

Under the Gauss-Markov assumptions,

where is the total sample variation in , and is the R-squared from regressing on all other independent variables (including the intercept).

Proof

Estimating

is unknown which should be estimated. The unbiased estimator of is :

Degree of freedom is because we have restrictions on the OLS residuls (there are equations we used when derive the coefficients).

Under MLR.1 - MLR.5 .

Standard Deviation and Standard Error of

Gauss-Markov Theorem

🔥

Under Assumptions MLR.1 through MLR.5, are the best linear unbiased estimators(BLUE) of respectively.

Linear:

is a linear estimator of iff it can be expressed as a linear function of the data on the dependent variable, i.e.

Best:

for the current theorem, best is defined as having the smallest variance. Given two unbiased estimators, it is reasonable to prefer the one with the smallest variance.

A Few Practical Issues

1. Excluding a relevant variable

Underspecifying the model

Compare the simple regression model and multiple regression model

Regress on . The estimated model is

Regress on and . The estimated model is

Therefore, the relationship between coefficients in the tow models is , where is the slope coefficient regressing on .

Proof

So if we excluding a relevant variable (which is called omitted variable) Then there will be omitted variable bias. Assume and we only regress on . Then . Thus

2. Including irrelevant variables

Overspecifying the model

I t means that one (or more) of the independent variables is included in the model even though it has no partial effect on in the population. (That is, its population coeffcient is zero.)

Mathematically, we say is irrelevant if

which means .

📌

Including will not affect the unbiasedness of other estimators, but will increase their variances.

3. Multicollinearity

High (but not perfect) correlation between two or more independent variables

Since , when increases, also increases. If , then