Post-Enrollment Clinic-Visit Controls Remove the Primary Care Pathway From Modeled Hospital Evaluations
Splitting pre-program background factors from care steps on the path in modeled hospital savings checks.
Health teams decide which household and clinic measures to hold constant before scoring a program. Pre-existing conditions like family poverty come before the program starts, whereas scheduled doctor visits happen as part of the care plan. Keeping pre-existing traits separate from active care steps shows the full program impact.
In California Community Health Worker evaluations, leaving post-start clinic visits open keeps the full 30% drop in hospital stays tied to a 14.6 point rise in primary care visits (71.7% vs. 57.1%).[1] Controlling for post-start clinic visits removes that care path, while restricting a provider test set to 55,018 active 2024 billers leaves only 9 of 95 excluded providers (after 90.5% exited before 2024).[2] Adjusting only for pre-program traits keeps the full policy impact intact.[3]
Table 1. Four Variable Roles in Health Policy Data and Models
Each column uses a separate CAPHE dataset to show how variable roles guide model controls.
| Variable Role |
Role 1: Confounder
Baseline County Poverty
|
Role 2: Care Step
Primary Care Visits
|
Role 3: Collider
Active Billing Subset
|
Role 4: Instrument
Supply-Side Budget Shift
|
|---|---|---|---|---|
| Source File | Food access tracts408 tracts | CHW trial dataTOT rates | Medicaid claims & OIG list | County spending panel57 counties |
| Causal Path | Poverty -> Distance & Food Need | CHW -> Clinic Visits -> Stays | Bans -> Tenure ← Bill Features | Shift -> Spending -> Deaths |
| Result When Left Out | Distance looks helpful-0.578 slope | Total effect kept-30% stays | Full list kept: 95 of 95 exclusions | Pandemic slope mixes surge aid+2.47 |
| Result When Added as Control | Distance: -0.017 flat slope | Direct part onlypath dropped | Active-only subset: 9 of 95 left | Stage 1 use: 5.31 fewer deaths / 100K |
| Base Input | Poverty explains 62.2% of gap | 71.7% vs. 57.1% visits1.90x | Active months: 38.0th vs. 50.0th | 13-yr IV: 5.31 fewer / 100K |
| Model Rule | ADD as baseline control | LEAVE OPEN in total model | DO NOT filter or control | USE in Stage 1 only |
Table 1 Notes & * Modeled Estimates
Expand notes
Table 1 Notes (Mode A Synthesis on Exact Source Statistics): Each column draws on a separate CAPHE dataset; the four roles are not estimated in one joint regression. Role 1: raw distance β = -0.578 (p < 0.0001); with poverty controlled, distance β = -0.017 (p = 0.827); poverty model R² = 62.2%. Role 2: visit uptake 71.7% vs. 57.1%, OR = 1.90 (TOT). Role 3: full-universe screening AUC = 0.724 (95% CI 0.667-0.777) vs. observed active-only subgroup AUC = 0.442 (95% CI 0.212-0.664; exit-year reweighted AUC = 0.653). Role 4: -5.31 Long-Difference Lewbel 2SLS slope per $10/capita (first-stage F = 111.20; Hansen J p = 0.2132; alongside -2.38 annual TWFE and -4.32 13-year cumulative OLS).
Column Scope. Baseline County Poverty. Shared Cause (X ← C -> Y) | N = 408 California urban tracts (ACS); unit: tract. Primary Care Visits. Program Path (X -> M -> Y) | IMPaCT CHW RCTs / CA CHW model (TOT); unit: member. Active Billing Subset. Shared Effect (X -> K ← Y) | N = 488,032 Medicaid billing NPIs (55,018 active in 2024); unit: billing NPI. Supply-Side Budget Shift. Stage 1 Shift (Z -> X -> Y) | 57 CA counties, 2003-2016 long-diff; unit: county.
Model Crosswalk. Plain table labels map to standard study models. Raw Yearly Average / Simple Avg is an unadjusted OLS model where routine and outbreak budgets mix. County & Year FE is a Two-Way Fixed Effects (TWFE) panel model. Routine-Budget Analysis / 21-Year County Budget Model is a Two-Stage Least Squares (2SLS / Lewbel 2012) model that isolates routine base budget shifts. County Board Budget Rules uses Brown (2014) county board rules.
Two-Ruler Benchmark Summary. Mode A Synthesis With Observed CAPHE Estimates as Base Inputs. Ruler 1: Model Controlling for the Care Step: Reports β_direct Only (Care Path α·γ Held Fixed). Adding post-start primary care visits as a control holds the care step fixed. The model then reports only the direct part. It leaves out the gain that runs through the +14.6 point rise in clinic visits (71.7% vs. 57.1%). Ruler 2: Path-Matched Model (Total Effect): Reports τ_total = β_direct + α·γ. Controlling only for pre-program traits like baseline poverty, and leaving clinic visits open, keeps the total effect. In the CHW trials, that total effect is a 30% drop in hospital stays.
Table 2. Decision Guide: When to Control, Leave Open, or Use Stage 1
Choosing model controls that match each variable's role in county health evaluations.
| Choice Factor | Confounder: Baseline Poverty | Care Step: Primary Care Visits | Collider: Active Billing Subset | Instrument: Supply-Side Shift |
|---|---|---|---|---|
| Timing & Path Check | Set before the program starts, such as baseline tract poverty (62.2% of food need gap). | Changed by the program on the way to health gains, such as CHW clinic visits (+14.6 points). | A post-billing status where 90.5% of excluded billers exited before 2024. | Shifts county health spending in Stage 1 without a direct link to death rates. |
| Model Choice by Role | Add baseline poverty as a control so income gaps do not slant store distance slopes. | Measure the CHW effect on hospital stays without controlling for post-start clinic visits. | Keep the full provider list instead of limiting the test set to 2024 active billers. | Put the budget shift in Stage 1 and report first-stage strength in the table notes. |
| Budget Consequence | Skipping baseline controls can steer food grants toward high-income areas. | Controlling for clinic visits removes the care path and understates CHW hospital savings. | The 2024 active subset drops 86 of 95 excluded billers and falls below a coin-flip rank. | Adding the Stage 1 shift as a control absorbs the clean spending changes. |
| Best-Fit Audit Rule | Lock Pre-Program Controls. Measure and control for background factors at baseline before enrollment starts. | Report Total Effect First. Estimate the total program effect first, then show care-path shares in a second step. | Keep Full List Totals. Run full-list checks before looking at active-biller or admitted-patient subsets. | Keep Instruments in Stage 1. Use instruments only in Stage 1 and never add them as outcome controls. |
Table 2 Notes: Over-Control Math, Collider & Instrument Notes, Data Provenance
Expand notes
• Program Step Over-Control Math: Suppose treatment X shifts care step M (M = α·X + u) and Y = β_direct·X + γ·M + ε. The total effect of X on Y is τ_total = β_direct + α·γ. Adding M as a control estimates only β_direct. The share removed is α·γ / τ_total. It reaches 100% only when β_direct = 0. The CAPHE CHW inputs (71.7% vs. 57.1%, OR = 1.90; -30% hospital stays) do not split those two pieces.
• Separate Datasets (Mode A Synthesis): The four columns show roles using four separate CAPHE files. Values are observed base inputs from each file. They are not from one joint regression and should not be compared across columns.
• Subgroup Selection Note: In the linked Medicaid file, the 55,018 providers still billing in 2024 (9 exclusions vs. 55,009 controls) show an observed screening AUC of 0.442 (95% CI 0.212-0.664), compared with 0.724 (95% CI 0.667-0.777) across all 487,479 providers and 0.653 when exit-year reweighted. Months Active has a SHAP correlation of r = -0.9009, and excluded providers sit at the 38.0th vs. 50.0th median peer percentile of active months, though with 9 active cases the subgroup difference does not isolate early exit as the sole cause.
• Instrument & Unit Note: Instruments play a different role from controls. In the LHA panel (57 California counties, 2003-2023), long-difference Lewbel (2012) instruments isolate multi-year budget changes from residual outbreak spending mix. The 5.31 fewer deaths/100K per $10/capita Long-Difference 2SLS estimate (2003-2016, first-stage F = 111.20, Hansen J p = 0.2132; and -11.49 in 2003-2019 excl. 5 LGRS outliers, F = 13.09, J p = 0.4726) uses these instruments in Stage 1 of 2SLS instead of as outcome controls.
• Data Provenance: Inputs come from FoodSecurity/analysis/outputs/regression_results_poverty_distance.csv and regression_model_statistics.csv (R² = 62.2%), METHODOLOGICAL_REVIEW_ARTICLES_5-12.md (disability OR = 1.76), CHWs/08_PhD_Final/DERIVATIONS.md and Kangovi et al. (2020) (71.7% vs. 57.1% TOT, OR = 1.90, -30% hospital stays), Mcaid/05_fraud_detection/paper/output/results.json (prospective_auc = 0.724, active_subset_auc = 0.442, shap_months_active_r; peer ranks in output/tables/tab_a3_base_rates.tex), and public/tools/lha-calculator/method.html (-2.38 annual TWFE, -4.32 13-yr cumulative OLS, -5.31 Long-Difference 2SLS, F = 111.20, Hansen J p = 0.2132, 57 counties).
Why Mapping the System Path Comes Before Running the Model
Adding every health and income variable to a tract regression mixes pre-program income gaps with health steps that sit on the path between food and chronic disease. If a variable is a pre-program confounder (such as poverty explaining 62.2% of tract food-need variance), leaving it out slants the slope. If a variable is a post-start care step, such as a Community Health Worker linking a patient to primary care (raising visits from 57.1% to 71.7%, OR = 1.90) to prevent a hospital stay, controlling for clinic visits subtracts part of the program's gain.
Before running a model, health economists map each variable in a causal diagram. Confounders are shared causes measured before treatment, so analysts control for them. Care steps sit on the causal path, so analysts leave them open when measuring total impact. Colliders and post-outcome filters like active 2024 billing alter the evaluation sample (where the 2024 active subset shows an observed 0.442 AUC vs. 0.724 on the full list), so analysts avoid filtering on them. Instruments shift treatment with no direct link to the outcome, so analysts use them in Stage 1 (as the LHA 13-year routine-budget IV model does with F = 111.20).
Data Sources, Model Inputs, Notes & Endnotes
(Numbered endnotes [1]–[6], state data sources & math rules)
Expand details
Case Study Focus: System Dynamics & Feedback · Confounders, Program Steps, Colliders & Instruments (Mode A Synthesis)
Numbered Endnotes & Main Sources
- [1]Pre-Program Baseline Confounders vs. Program Mechanisms: U.S. Census ACS California urban tract panel covers 408 tracts (baseline poverty explains 62.2% of food-need variance). Kangovi, S., et al. (2020), Health Affairs shows 71.7% vs. 57.1% primary care visits (OR = 1.90) and -30% hospital stays.
- [2]Filtered Subsample & Supply-Side Variables: HHS Medicaid Provider Spending linked to OIG LEIE shows a 0.724 full-list AUC vs. 0.442 AUC in the 2024 active-biller subset (0.653 when exit-year reweighted). The 57-county LHA spending panel finds a -2.38 annual TWFE slope, -4.32 13-year cumulative OLS slope, and -5.31 Long-Difference Lewbel 2SLS slope (first-stage F = 111.20, Hansen J p = 0.2132).
- [3]Methodological Reference on Control Selection: Cinelli, C., Forney, A., & Pearl, J. (2022), Sociological Methods & Research.
- [4]Kangovi, S., et al. (2020), Health Affairs (IMPaCT CHW RCTs)
- [5]Lewbel, A. (2012), JBES
- [6]OIG LEIE
Mode Classification (Mode A Synthesis on Observed Inputs): The role rules and model guide are a Mode A synthesis. The numbers in Table 1 (tract poverty R² = 62.2%; CHW primary care visits 71.7% vs. 57.1% TOT, OR = 1.90, and -30% hospital stays; Medicaid screening AUC = 0.724 full list vs. observed AUC = 0.442 in the 2024 active subset; LHA Long-Difference 2SLS -5.31, first-stage F = 111.20, Hansen J p = 0.2132) are observed outputs from four separate CAPHE files.
Primary References & Data Sources: Cinelli, C., Forney, A., & Pearl, J. (2022), "A Crash Course in Good and Bad Controls", Sociological Methods & Research; Kangovi, S., et al. (2020), Health Affairs (IMPaCT CHW RCTs); Lewbel, A. (2012), JBES; U.S. Census ACS; HHS Medicaid Provider Spending; and OIG LEIE.
Common Questions & Method Notes
- Why shouldn't an analyst control for post-start clinic visits when measuring whether CHWs cut hospital stays?
- Raising primary care visits (from 57.1% to 71.7% TOT, OR = 1.90) is a main care step through which CHWs prevent hospital stays (-30%). Controlling for post-start clinic visits holds that step fixed. That blocks part of the program path and understates total CHW value.
- How can an analyst tell whether a variable is a confounder or a care step?
- Check the clock and the direction of cause. A confounder is set BEFORE the program starts (like baseline age, pre-program poverty, or county land area) and affects both participation and health. A care step changes AFTER the program starts because of the program.
- What is an instrument, and why is it kept out of the outcome controls?
- An instrument shifts the policy input (here, county public health spending) without a direct path to the health outcome. It enters Stage 1 of two-stage least squares to isolate spending shifts that are not driven by mortality shocks. In the LHA long-difference model of 57 California counties (2003-2016), the Lewbel instruments have a first-stage F of 111.20 and Hansen J p-value of 0.2132. Adding an instrument as an outcome control would absorb the clean spending variation.