Ch4: Multiple Regression Analysis: Inference

TOC

Classical Linear Regression Model Hypothesis Testing 1. Terminologies of hypothesis testing:2. Testing Against One-Sided Alternatives 3. Testing Against Two-sided Alternatives 4. p-Values 5. Economic Significance v.s. Statistical Significance Confidence Interval Hypothesis Testing about Multiple Parameters 1. Testing Single Linear Combination of Parameters 2. Testing Multiple Linear Restrictions: F Test 3. Tesing General Linear Restrictions

Classical Linear Regression Model

We have known that under the Gauss-Markov assumptions, the mean and variance of OLS estimators are

To derive the full distribution of , we need more assumptions.

🔥

MLR.6 Normality

The population error is independent of the explanatory variables and is normally distributed with zero mean and variance :

MLR.6 is much stronger than the previous assumptions and implies MLR.4 and MLR.5

MLR.1 - MLR.6 are called the classical linear model (CLM) assumptions. Models under these six assumptions are called classical linear model. It can be shown that under the CLM assumptions, the OLS estimators are the minimum variance unbiased estimators.

A succinct way to summarize the population assumptions of the CLM is

📌

In reality, MLR.6 is often hard to be satisfied. But the nonnormality of the error is not a serious problem with large smaple sizes.

Since

where . So conditional on (which means can be viewed as constant), is a linear combination of and thus follows the normal distribution. Since we have already known the and of , thus

since , where where is unkown. So we use

💡

Under the CLM assumptions,

where is the number of unknown parameters in the population model . And is the degree of freedom

t distribution

t distribution has one parameter: degree of freedom .

The PDF of the t distribution is similar to the normal distribution, but more dispersed and there are more areas in the tail.

When the degree of freedom gets larger, the t distribution becomes closer to the normal distribution

The degree of freedom of is . So when size is large enough, we can approximate the distribution of it as normal.

Hypothesis Testing

🎯

When we know (or assume) the distribution of the OLS estimators, we can do hypothesis tests about the parameters in population regression model.

Hypothesis testing helps formalize the idea of “unlikeliness”.

1. Terminologies of hypothesis testing:

Test statistic:

Depend on the data.

We know its distribution under the null hypothesis.

Reject the null hypothesis when t-stat falls in the rejection region.

Type I error:

significance level := = (Rejecting | is true).

Type II error:

(Not rejecting | is false)

Trade-off between Type I error and Type II error:

It is often the case that when we reduce the type I error, we increase the type II error

Rejection region:

when deciding it, classical hypothesis testing requires that

initially specify a significance level (quantifying the tolerance for Type I error).
minimize the probability of a Type II error.

2. Testing Against One-Sided Alternatives

consider a test statistic:

When is true, the t-stat is:

define , which is called t-ratio of

👉🏻

Rejection rule: , where is the 95th (, is the significance level) percentile when is true (i.e. 95th percentile of ).

Use STATA to find the critical value: display invt(N-k-1, 1-)

Similarly, for the test :

reject when , where is the -th percentile of

3. Testing Against Two-sided Alternatives

Fix a significance level . That is, decide our level of “tolerance” for typr I error.

Find the critical value associated with . For , is the -th percentile of

Reject if

When a specific alternative is not stated, it is usually considered to be two-sided.

If is rejected in favor of at the 5% level, we say that “ is statistically significant or statistically different from zero, at the 5% level”.

4. p-Values

Given the observed value of t statistic, the smallest significance level at which the would be rejected is called p-value.

p-value represents the probability of observing a value as extreme as under .

For , the p-value is , where denotes the random variable following

📌

Significance level and critical value have a one-one mapping relationship. For the test above, two methods below are equivalent: 1. Fix a significance level , calculate the critical value , then reject if . 2. Fix a significance level , calculate the p-value, reject if .

5. Economic Significance v.s. Statistical Significance

The statistical significance of a variable is determined entirely by the size of , whereas the economic significance (or practical significance) of a variable is related to the size (and sign) of .

We often care about both of them 🧐

Confidence Interval

Besides calculating a point estimate, we can also calculate an interval estimate, which provides a range of likely values fot the population parameter.

We can construct a confidence level depending on . We call it a confidence interval.

where (critical value) is the percentile in

Interpretation of the Confidence Interval:

The upper and lower bound of aconfidence interval are random variables.

The true parameter is fixed and unknown. For every sample, we can calculate a confidence interval. The meaning of a 95% confidence interval: if we sample repeatedly many times, then the true will appear in 95% of the confidence intervals.

Unfortunately, for the single sample that we use to construct the CI, we do not know whether is actually contained in the interval.

We want to catch the butterfly () with our net (CI). The butterfly is fixed while we move our net.

Two-tailed hypothesis test using confidence interval

Fix a significance level, calculate the critical value and then construct the confidence interval.

Reject if 0 is not inside the CI.

💡

for the test , the rejection rules: 1. 2. 3. 0 is not inside the CI

Hypothesis Testing about Multiple Parameters

1. Testing Single Linear Combination of Parameters

Method 1:

construct the t-stat for

🚧

Difficuluty: calculate , which equals to , where is the estimate of

Method 2:

estimate a new regression model:

Define . So , thus

construct a new variable, , and then regress on and .

Test in the new regression, whether the coefficient for is 0.

2. Testing Multiple Linear Restrictions: F Test

test

Consider the restricted model:

If is true, the two models are the same, thus whether we include and into the model, the sum of squared residuals should not change much.

However, if is false. It means that at leat one of is nonzero and the sum of queard residuals should fall when we include thest new variables.

is the number of linear restrictions, which is the difference in degrees of freedom in the restricted model versus the unrestricted model.

is the sum of squared residuals from the restricted model and is the sum of squared residuals from the unrestricted model.

Since is no smaller thatn , the F-stat is always nonnegative.

⇒ the distribution with degrees of freedom in the numerator and degrees of freedom in the denominator.

Reject if

For testing, the p-value is defined as , where is an F random variable with degrees of freedom, and F is the actual value of the test statistic.

p-value is the probability of observing a value of F at least as large as we did, given that the null hypothesis is true. Reject if .

version of the F-stat

🚧

version of F-stat can be used when the dependent variable of the unrestricted and restricted models are the same.

3. Tesing General Linear Restrictions

Example

We cannot use version of F-stat in this test because the dependent variables of the two models are different.

In general, use F test to test multiple linear hypothesis. More generally, use the SSR form of F-stat when there are different dependent variables in restricted model and unrestricted model.

🔑

Using STATA:

reg y x1 x2 x3 test (x1=0) (x2=0) .Then we get the F-stat and p-value of the test.