The Data
Both the program county and the comparison county show improved outcomes after 2020. DiD subtracts out the common trend to isolate what is unique to the program county. (Data are simulated for illustration.)
DiD Calculation Table
| County | Pre (2019) | Post (2022) | Change |
|---|---|---|---|
| County A (Has Program) |
85.2 | 70.0 | -15.2 |
| County B (No Program) |
88.0 | 75.2 | -12.8 |
| Difference | -2.8 | -5.2 | -2.4 |
The DiD Formula
Visualizing the Counterfactual
ED Visit Rate Over Time by County
County A improved by 15.2 points, but County B also improved by 12.8 points.
What changed for everyone? And how do we separate the program effect from these common shocks?
How DiD Works
Difference-in-differences removes the effect of anything that changed for everyone at the same time. What remains is the effect unique to the treated group.
What Is Difference-in-Differences?
DiD is a quasi-experimental design that estimates causal effects by comparing changes over time between a treated group and a comparison group. It works by:
- First difference: Comparing before vs. after within each group
- Second difference: Subtracting the comparison group's change from the treatment group's change
- Result: Removing any shared trends that affected both groups equally
Core assumption: Without treatment, both groups would have followed the same trajectory (parallel trends).
The Core Logic
If Medi-Cal expansion helped everyone equally, both counties should improve by the same amount. The extra improvement in County A beyond what County B experienced is attributed to the program.
- County B's change = statewide effect only
- County A's change = statewide + program effect
- Subtract to isolate program effect
The Counterfactual
We can never observe what would have happened to County A without the program. DiD constructs this counterfactual by assuming County A would have followed County B's trajectory.
- The dashed line shows this "what if"
- Gap between actual and counterfactual = effect
- Relies on parallel trends assumption
Why This Matters for Policy Evaluation
Without DiD, we might credit County A's program for the full 15.2-point improvement. With DiD, we see that 12.8 points would have happened anyway (it happened in County B too). Only 2.4 points appear attributable to the program.
This is what economists mean by "differencing out common shocks." The comparison group reveals the counterfactual trend, so we can isolate the treatment effect.
DiD isolates a 2.4-point effect. But should we trust this estimate?
The method relies on assumptions that may or may not hold. The next panel audits each assumption for this scenario.
Assumptions Audit
DiD estimates are only valid if key assumptions hold. This audit evaluates each assumption and flags potential vulnerabilities.
Checking DiD Assumptions
-
Parallel Pre-Trends
Before 2020, both counties showed similar flat trends in ED visits. This supports (but does not prove) that they would have continued similarly without intervention.
-
No Spillover Effects
We assume County B was not affected by County A's program. If County A's success attracted patients from neighboring counties, our comparison may be biased.
-
Stable Unit Treatment Value (SUTVA)
Each county's outcome depends only on its own treatment status. No evidence of interference between counties in this scenario.
-
No Differential Shocks
We assume no other county-specific events occurred around 2020. If County A also opened a new hospital, that confounds our estimate.
-
Correct Timing
We correctly identify 2020 as the treatment year. The program started in January 2020 and we compare 2019 (pre) to 2022 (post).
Overall Assessment
This DiD analysis has reasonable support: parallel pre-trends are visible and timing is clear. We cannot rule out spillover or other county-specific shocks. The 2.4-point estimate should be interpreted as suggestive, not definitive.
Some assumptions are uncertain. How sensitive is our conclusion?
The next panel explores how the estimated effect changes under different assumption violations.
Sensitivity Analysis
These questions help identify how robust the DiD estimate is to plausible assumption violations. They reveal where the analysis is most vulnerable to bias.
Effect Estimates Under Different Scenarios
| Scenario | Assumption Violation | Adjusted Effect | Interpretation |
|---|---|---|---|
| Baseline DiD | None (parallel trends hold) | -2.4 | Program reduces ED visits by 2.4 per 10k |
| Diverging Trends | County A was improving 1 pt/yr faster pre-2020 | -0.4 | Most of "effect" was pre-existing trend |
| Positive Spillover | Program also helped County B residents | -3.5 to -4.5 | True effect larger than measured |
| New Hospital Opened | New facility reduced ED use independently | -1.0 to -1.5 | Hospital explains part of the drop |
| Measurement Shift | County A changed ED coding in 2020 | Unknown | Effect could be artifact of measurement |
| Alternative Comparison | Using a different comparison county | -1.8 to -3.2 | Effect sensitive to comparison choice |
What This Tells Us
The baseline DiD estimate of -2.4 is sensitive to assumptions. Under plausible violations:
- Lower bound: If County A was already improving faster, the true effect could be near zero
- Upper bound: If the program spilled over to help the comparison county, the true effect could be larger (-4 or more)
- Recommendation: Report the 2.4-point effect with bounds of roughly 0 to -4.5 depending on assumptions
Key Takeaway
DiD does not prove causation, but it reveals which assumptions matter most. When parallel trends hold and no differential shocks occur, DiD provides a credible estimate of treatment effects. When these assumptions fail, the method can mislead.
Policy changes, eligibility cutoffs, and geographic boundaries can provide the variation needed for credible DiD estimates. This is what economists mean by "identification."