Announcements

• Grades for Homework 2 will be posted this week.

  • Review the Answer Key on the course website (posted Mon/Tue after submission).

  • Everyone did pretty well, but remember that answers need to match submitted code.

• Midterm is in class (week of Oct. 16th):

  • Practice quizz (not graded, but mandatory) for proctored exams (HonorLock).

  • There will be a review session Thur/Fri before the midterm (poll).

Last week

Today

What did we see last week?

• Limitations in RCTs:

  • Generalizability

  • Breaking SUTVA: Spillover effects and General Equilibrium Effects.

• Introduced Observational Studies:

  • We need to control by confouders: Conditional Ignorability Assumption.

  • How? E.g. Regression, Matching.

Identification strategies (designs) we have seen so far...

Randomized Controlled trials (RCTs)

• Treatment assignment is randomized

• Ignorability assumption holds by design: Groups are comparable in obs. and unobs. characteristics.

• Analysis? (i) Check balance and (ii) difference in means.

Identification strategies (designs) we have seen so far...

Selection on Observables (Matching, Regressions with covariates):

• Treatment assignment is not randomized

• Conditional independence assumption holds if we can control for all confounders (assumes all confounders are observed)

  • After adjusting for covariates, assignment to treatment is as good as random (Is this a credible assumption?).

• Analysis? (i) Compare balance before matching, (ii) compare balance after matching, and (iii) difference in means for the matched sample.

Finding "RCTs" in the wild

• Given that we can't run RCTs for everything, the next best thing is finding a source of random variation that, for all practical purposes, would work as an RCT
  
  

Natural Experiments

You, as a researcher, did not assign units to treatment levels

1. Random: Assignment to an intervention is random (e.g. lottery)

2. As if random: Assignment to an intervention is not random, but it's not correlated with potential outcomes.

Context matters!

Examples of natural experiments

• Oregon Health experiment: Lotteries for Medicaid expansion.

• Vietnam Draft: Impact of military service/education (GI Bill) on earnings.

• Lottery winners: Impact of unearned income on labor earnings.

We can analyze these cases just like an RCT

What do we do if we have something like a natural experiment but both our groups are not necessarily balanced?

Raising the minimum wage

What happens if we raise the minimum wage

Economic theory says there should be fewer jobs

New Jersey in 1992

$4.25 → $5.05

The setup

Before vs After

Avg. # of jobs per fast food restaurant in NJ

New Jersey_before = 20.44

New Jersey_after = 21.03

∆ = 0.59

Is this a causal effect?

Treatment vs Control

Avg. # of jobs per fast food restaurant

Pennsylvania_after = 21.17

New Jersey_after = 21.03

∆ = -0.14

Is this a causal effect?

Problems

Difference-in-Differences

The idea of a DD analysis is to take the within-unit growth...

∆ (post − pre) = within-unit growth

Difference-in-Differences

... and the across-group growth...

∆ (treatment − control) = across-group growth

Difference-in-Differences

... and combine them!

∆_{within units} − ∆_{across groups} =
Difference-in-differences =
causal effect!

Coming back to New Jersey

How does it look in a plot?

... And the real plot!

Difference-in-Differences in practice

• There's no need to manually estimate all group means..

We can use regressions!

• If the two dimensions for our DD are time and treatment:

$$Y_i={\beta }_0+{\beta }_{1}Trea{t}_i+{\beta }_{2}Pos{t}_i+{\beta }_{3}Trea{t}_i\times Pos{t}_i+{\epsilon }_i$$ where $Treat=1$ for the treatment group, and $Post=1$ for the after period.

Can you identify the different coefficients?

Difference-in-Differences in practice

• There's no need to manually estimate all group means..

We can use regressions!

• If the two dimensions for our DD are time and treatment:

$$Y_i={\beta }_0+{\beta }_{1}Trea{t}_i+{\beta }_{2}Pos{t}_i+{\beta }_{3}Trea{t}_i\times Pos{t}_i+{\epsilon }_i$$ where $Treat=1$ for the treatment group, and $Post=1$ for the after period.

β₃ is the causal effect!

Let's see it with data

minwage <- read.csv("https://raw.githubusercontent.com/maibennett/sta235/main/exampleSite/content/Classes/Week7/1_DiffInDiff/data/minwage.csv")
minwage <- minwage %>% mutate(treat = ifelse(location=="PA", 0, 1), # treat group: the treated state
                              post = ifelse(date=="nov1992", 1, 0)) # post: time after treatment was set in place
head(minwage)

##        chain location wage full part    date treat post
## 1     wendys       PA 5.00   20   20 feb1992     0    0
## 2     wendys       PA 5.50    6   26 feb1992     0    0
## 3 burgerking       PA 5.00   50   35 feb1992     0    0
## 4 burgerking       PA 5.00   10   17 feb1992     0    0
## 5        kfc       PA 5.25    2    8 feb1992     0    0
## 6        kfc       PA 5.00    2   10 feb1992     0    0

Let's see it with data

summary(lm(full ~ treat*post, data = minwage))

## 
## Call:
## lm(formula = full ~ treat * post, data = minwage)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -10.664  -5.971  -2.405   3.653  52.029 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)   10.664      1.007  10.589   <2e-16 ***
## treat         -2.693      1.117  -2.411   0.0162 *  
## post          -2.493      1.424  -1.750   0.0805 .  
## treat:post     2.927      1.580   1.853   0.0643 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 8.243 on 712 degrees of freedom
## Multiple R-squared:  0.008207,    Adjusted R-squared:  0.004028 
## F-statistic: 1.964 on 3 and 712 DF,  p-value: 0.118

• Can you interpret the treatment effect?

Important things to note

• In Difference-in-Differences, groups do not need to be balanced

  • If differences are stable over time, they get cancelled out when doing the Diff-in-Diff.

• Difference-in-Differences provides an estimate for an average treatment effect for the treated group

  • The estimated effect is not generalizable for the entire sample, only for the treated group.

Assumptions

Parallel Trends

In the absence of the intervention, treatment and control group would have changed in the same way

If parallel trends assumption hold...

If parallel trends assumption doesn't hold...

... the DD estimate will be biased

Robustness Check

Pre-Parallel Trends

Check by pretending the treatment happened earlier; if there's an effect, there's likely an underlying trend

Use the pre-intervention period and conduct a placebo DD

Your turn

Wrapping up

References

• Angrist, J. and S. Pischke. (2015). "Mastering Metrics". Chapter 2.

• Angrist, J. and S. Pischke. (2015). "Mastering Metrics". Chapter 5.

• Heiss, A. (2020). "Program Evaluation for Public Policy". Class 8-9: Diff-in-diff I and II, Course at BYU.