Trend-Drift Sensitivity Bounds Quantify How Much Pre-Policy Noise a County Evaluation Can Withstand
Testing how much post-policy trend drift a modeled county panel can take before its result crosses zero.
Analysts use multi-year county records to measure how public health budgets and local policy changes affect community outcomes. Before trusting a two-group comparison, teams check whether treated and comparison counties followed similar paths before the policy began. Short pre-policy windows with wide year-to-year swings can let slow county drift pass that check.
When auditing California's 57-county public health panel (N = 943 pre-COVID county-years), the 2.38 fewer deaths per 100,000 annual finding (building to 4.32 over 13-year cumulative horizons) holds across urban and rural strata and wild cluster bootstrap checks (p = 0.007).[1] In a shorter 1,408-county panel, a -0.47 percentage point result holds up only if modeled post-policy drift stays below 35% of the pre-policy swing.[2] Passing a pre-trend check alone leaves a ±1.27 percentage point noise band (2.5 times the effect) unexamined, so reporting drift limits shows which findings can guide capital budgets.[3]
Table 1. Pre-Trend Noise Bands, Drift Bounds & LHA Panel Checks
Stated event-study inputs from the 1,408-county panel with modeled drift bounds and California 57-county panel benchmarks.
| Trend-Check Measure |
Model 1: Zero Drift
Base Event Study
|
Model 2: Noise Check
Pre-Trend Noise Band
|
Model 3: Drift Bound
Post-Policy Drift Limit
|
Model 4: Slope Check
County Linear Slopes
|
|---|---|---|---|---|
| Post-Policy Point Estimate | -0.47 ppclear drop | Leads: -0.056, -0.030, 0.00 | -0.47 ppbounded set | +0.003 pp+109% shift |
| Year-to-Year Margin | ±0.22 pp base margin | ±0.58 to ±0.62 pp pre-noise | Folds in pre-period noise | ±0.016 pp margin |
| Plausible Effect Range* | -0.91 to -0.03 pp0.88 span | -1.27 to +1.16 pp2.43 span | At 0.35x drift: -0.96 to 0.00 pp | -0.028 to +0.034 pp0.062 span |
| Trend Check & Drift Limit* | Clear drop at zero drift | Passes checkwide band | 35% drift limit0.35x | Crosses zero at 0.35x bound |
| Unseen Drift vs. Effect Size | Assumes 0.00x drift | ±1.27 pp>2.5x the effect | At 2.0x drift: -1.08 to +0.15 pp | Placebo date check flags drift |
| CA 57-County Panel Benchmark | Annual panel: 2.38 fewer deaths | 13-yr cumulative: 4.32 fewer | 2003-2016 single-regime: -2.62 | 13-yr routine-budget IV: -5.31 / 100K |
Table 1 Notes & * Modeled Estimates
Expand notes
Table 1 Notes & * Modeled Estimates. Exact statistics behind the plain cells (stated base inputs, N = 1,408-county DiD panel). Base TWFE: -0.47 pp, SE = 0.22 pp, 95% CI [-0.91, -0.03], p = 0.036 at M = 0. Pre-period leads: SE = 0.620 (e = -3), SE = 0.582 (e = -2), SE = 0.549 (e = -1); joint pre-trend test Pre-F p = 0.9997 (low power). Rambachan-Roth bounds: robust sets fold in pre-trend SE; M* = 0.35 is a Mode A calculation. County linear slopes: +0.003 pp, SE = 0.016 pp, p = 0.87; placebo date test p = 0.04. California LHA panel (57 counties, N = 943 pre-COVID county-years): Annual TWFE OLS 2.38 fewer deaths per 100,000 per $10/person (SE = 0.78, p = 0.0022), 13-year cumulative OLS -4.32 (SE = 1.54, p = 0.0052), and Long-Difference Lewbel 2SLS -5.31 (first-stage F = 111.20).
Column Scope. Base Event Study. Assumes Zero Post-Trend Gap. Pre-Trend Noise Band. 3-Year Pre-Policy Noise Check. Post-Policy Drift Limit. 35% Drift Limit vs. 2.0x Drift. County Linear Slopes. County-Specific Slope Control.
Model Crosswalk. Plain table labels map to standard study models. Raw Yearly Average / Simple Avg is an unadjusted OLS model where routine and outbreak budgets mix. County & Year FE is a Two-Way Fixed Effects (TWFE) panel model. Routine-Budget Analysis / 21-Year County Budget Model is a Two-Stage Least Squares (2SLS / Lewbel 2012) model that isolates routine base budget shifts. County Board Budget Rules uses Brown (2014) county board rules.
Two-Ruler Benchmark Summary. Mode A Drift Bounds on Stated Event-Study Inputs (N = 1,408 Counties); Pre-Test Power & Drift Limits. Ruler 1: Standard Pre-Trend Gate (Passes Standard Trend Check): ATT = -0.47 pp (clear drop at zero drift). Stated base inputs (N = 1,408 counties): pre-policy lead points at e = -3, -2, -1 are near zero (-0.056, -0.030, 0.000) and pass the joint pre-trend check. Standard practice treats parallel trends as met and reports a -0.47 pp (±0.22 pp margin) effect. Ruler 2: Mode A Post-Policy Drift Limit (M* = 0.35): M* = 0.35 | Shifts to +0.003 pp. Pre-period noise bands span -1.27 to +1.16 pp (2.5x wider than the ATT). In the Mode A calculation on these inputs, if post-policy drift tops 35% (M* = 0.35) of the max pre-period slope change, the range crosses zero. Method details sit in the Table Notes.
Method & Data Audit (TooEarlyToSay.com): Rambachan-Roth Pre-Trend Sensitivity Case → · Choosing Among Difference-in-Differences Designs →
Table 2. Trend Rules, Budget Risks & Drift Limits
How analysts separate a noisy pre-trend check from a sturdy multi-year finding.
| Choice Factor | Base Event Study | Pre-Trend Noise Band | Post-Policy Drift Limit | County Linear Slopes |
|---|---|---|---|---|
| Core Trend Assumption | Post-policy paths stay 100% parallel across treated and comparison counties (zero drift). | A passing pre-policy trend check is read as proof that post-policy paths stay parallel. | Allows post-policy trend drift up to a chosen multiple of the largest pre-policy slope shift. | Fits a straight-line time slope for each county using only 3 pre-policy years. |
| What Drift Bounds Add | Bounds fold pre-period lead noise into the post-policy range that [-0.91, -0.03] omits. | With ±0.62 pp pre-noise, a 1.0 pp/yr drift passes the check yet accounts for the -0.47 pp drop. | A 35% drift limit shows how little drift crosses zero. | Fitting 1,408 county slopes on 3 pre-years overfits noise and absorbs real phased-in gains. |
| Lesson for County Studies | The 57-county LHA panel has high power across 17 pre-COVID years, holding in urban and rural strata. | When a short 3-year pre-window gives wide ranges, teams should not read low power as zero drift. | Regime shifts like the 2020-2023 COVID-19 grant surge require separate windows. | Long multi-year panels and 13-year cumulative models (-4.32) beat fitting county slopes on 3 pre-years. |
| Best-Fit Policy Rule | Never Report Zero-Drift Alone. Pair base event-study numbers with pre-period noise widths and drift bounds. | Show the Pre-Trend Noise Band. Report the pre-period band beside any pre-policy trend check. | Check 1.0x Drift for Capital Shifts. If the drift limit sits below 1.0x (here 0.35x), pair the study with a multi-year panel model. | Use 6+ Pre-Years for County Slopes. Skip county linear slopes in 3-year pre-panels and lengthen the pre-policy window instead. |
Table 2 Notes: Roth (2022) Pre-Test Power & Rambachan-Roth (2023) Drift Bounds
Expand notes
• Pre-Period Noise vs. Signal (Stated Base Inputs, N = 1,408 Counties): At lead e = -3, the point is -0.056 pp with SE = 0.620 (95% CI: [-1.27, +1.16]). At e = -2, it is -0.030 pp with SE = 0.582 ([-1.17, +1.11]). At e = -1, it is 0.000 pp with SE = 0.549 ([-1.08, +1.08]). The pre-period CIs are 2.5x wider than the -0.47 pp ATT, so the joint test (p = 0.9997) has low power (Roth, 2022, AER: Insights).
• HonestDiD Drift Limit (M* = 0.35): Under Rambachan & Roth (2023, Review of Economic Studies), M = 1.0 allows post-policy trend drift up to 100% of the max pre-period slope shift. Here, the 95% robust range hits 0.00 at M* = 0.35 ([-0.96, 0.00] pp) and widens to [-1.08, +0.15] pp at M = 2.0.
• Unit Rules: The ATT, leads, SEs, and M* bounds come from the 1,408-county panel (percentage points). The -2.38 annual, -2.62 single-regime, -4.32 cumulative, and -5.31 Long-Difference 2SLS (F = 111.20) figures come from the separate 57-county California LHA panel (deaths per 100,000 per $10/person; N = 943 pre-COVID county-years).
• Extended Cell Context. What Drift Bounds Add (Post-Policy Drift Limit). A 35% drift limit (0.35x pre-period swing, below the 1.0x benchmark) shows how little drift crosses zero. · Lesson for County Studies (Post-Policy Drift Limit). Regime shifts like the 2020-2023 COVID-19 grant surge (+2.47 vs. -2.38) require separate windows. · Best-Fit Policy Rule (Pre-Trend Noise Band). Report the pre-period band (-1.27 to +1.16 pp) beside any pre-policy trend check.
Compare How Allowing Post-Policy Drift Shifts the Plausible Effect Range
Comparing the zero-drift base model against drift limits and county linear slopes across 1,408 counties.
Assuming post-policy county paths stay 100% parallel passes the pre-policy trend check and yields a clear drop.
Allowing post-policy drift equal to 35% of the max pre-period slope shift brings the top of the range to zero.
Fitting county linear slopes across three pre-policy years shifts the point estimate to near zero (+0.003 pp).
Scenario Math, Pre-Period Standard Errors & HonestDiD Bounds
Expand scenario notes
• Bound 1 Base Event-Study Inputs (N = 1,408 Counties): Pre-policy leads are -0.056 pp (SE = 0.620) at e = -3, -0.030 pp (SE = 0.582) at e = -2, and 0.000 pp (SE = 0.549) at e = -1 (joint pre-policy trend check p = 0.9997). The post-policy M = 0 TWFE estimate is -0.47 pp (SE = 0.22, 95% CI: [-0.91, -0.03], p = 0.036).
• Bound 2 Rambachan-Roth HonestDiD Limit (M* = 0.35): Under Rambachan & Roth (2023), the 95% robust confidence set hits [-0.96, 0.00] pp at M* = 0.35 (below the M = 1.0 benchmark) and widens to [-1.08, +0.15] pp at M = 2.0.
• Bound 3 Short-Window County Slopes: Adding 1,408 county linear trends fit on 3 pre-years shifts the estimate by +109%, from -0.47 pp to +0.003 pp (SE = 0.016, p = 0.87).
Why Passing a Pre-Trend Check Is Only the First Step
In county program reviews, analysts check that pre-policy lead points cross zero and cite a high joint p-value (such as p = 0.9997). Yet wide pre-period noise bands can let slow county drift pass that check. The smaller the sample or the wider the pre-period margin, the easier it is to pass a pre-trend check while an unseen slope accounts for the post-policy gap.
Modern health economics pairs pre-trend checks with explicit drift bounds. Reporting the Rambachan-Roth drift limit (35% of the pre-period swing) and comparing short panels against multi-year models (such as the 57-county California LHA panel, N = 943 pre-COVID county-years, where the annual county panel yields -2.38 and 13-year cumulative differences yield -4.32) shows leaders how much trend drift a finding can take.
Data Sources, Model Inputs, Notes & Endnotes
(Numbered endnotes [1]–[4], state data sources & math rules)
Expand details
Case Study Focus: Program Impact · Pre-Trend Power & Drift Limits (Mode A on Stated Baseline Inputs)
Numbered Endnotes & Main Sources
- [1]Multi-State 1,408-County Event-Study Panel: USDA SNAP Retailer Historical Database and FDIC Bank Failures & Branch Panel (N = 1,408 counties; base post-policy estimate -0.47 pp, SE = 0.22, p = 0.036; pre-policy leads -0.056, -0.030, 0.000 with 95% CIs from -1.27 to +1.16 pp).
- [2]Methodological reference: Roth, J. (2022), AER: Insights and Rambachan, A., & Roth, J. (2023), Review of Economic Studies. CAPHE case-study sensitivity calculation (1,408-county panel): post-policy trend drift limit M* = 0.35, where the 95% robust CI hits [-0.96, 0.00] pp.
- [3]California 57-County LHA Panel Comparison (2003-2023): California State Controller (N = 943 pre-COVID county-years; -2.38 deaths/100K per $10/person annually, -4.32 over 13-year cumulative horizons, and -5.31 in Long-Difference Lewbel 2SLS with F = 111.20).
- [4]Rambachan & Roth (2023), "A More Credible Approach to Parallel Trends," Review of Economic Studies
Mode Classification (Mode A Drift Bounds on Stated Base Inputs): Event-study pre-trend leads (-0.056, -0.030, 0.000), standard errors (0.620, 0.582, 0.549), and the base ATT (-0.47 pp, SE = 0.22, p = 0.036) are stated inputs from the 1,408-county DiD panel. The Rambachan-Roth limit (M* = 0.35) and its robust ranges are Mode A calculations on those inputs. California LHA benchmarks (-2.38 annual TWFE, -2.62 single-regime, -4.32 13-year cumulative, and -5.31 Long-Difference 2SLS) come from the separate 57-county LHA panel (N = 943 pre-COVID county-years).
Primary References & Data Sources: Rambachan & Roth (2023), "A More Credible Approach to Parallel Trends," Review of Economic Studies; Roth (2022), "Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends," AER: Insights; USDA SNAP Retailer Historical Database; and California State Controller County Financial Transactions (2003-2023).
Companion Method & Code Guides (TooEarlyToSay.com)
- Parallel-Trends Sensitivity: An Article-Reported Case. Article-reported bank-closure case (1,408 counties) in which a pre-trend test returned p = 0.9997 yet Rambachan-Roth bounds changed the conclusion; not publicly reproduced.
- Using difference-in-differences in practice. Decision rules for when difference-in-differences fits and which assumptions matter, with an article-reported rolling DiD simulation labeled by evidence status.
Common Questions & Method Notes
- Why does a high pre-trend p-value (such as p = 0.9997) not prove that parallel trends hold?
- A high p-value only means the data cannot rule out zero pre-trend. When pre-period standard errors are wide (0.55 to 0.62 pp), the 95% CI spans -1.27 to +1.16 pp. A drift slope 2.5 times larger than the -0.47 pp effect can still sit inside that band.
- What does a sensitivity limit of M* = 0.35 mean for budget choices?
- If post-policy trend drift exceeds 35% of the max slope change seen in the pre-period, the 95% robust range crosses zero and the finding is no longer statistically clear.
- How does the California 57-county LHA panel test stability across county size strata?
- Across the 2003 to 2019 pre-pandemic window (N = 943), the mortality reduction is statistically significant in both large urban counties (-2.59 per $10, p < 0.0001) and smaller/rural counties (-2.42 per $10, p = 0.0136), with a wild cluster bootstrap p-value of 0.007.