Active-Only Provider Subsets Omit Earlier-Exiting Sanctioned Billers in Modeled Risk Audits
Comparing modeled full-list provider risk screens against active-biller subsets that drop providers who left early.
Program integrity teams use billing alerts to decide which Medicaid providers to audit first. Many sanctioned providers stop billing years before a formal ban posts, and confirmed fraud cases make up a tiny fraction of all active billers. Checking alerts across every historical biller helps audit managers plan investigator workloads.
Across 487,479 Medicaid billing providers, the screening model flags sanctioned providers at 9.5 times the random rate in the top 1% of alerts (9 caught cases across 4,844 flags).[1] In the subset of 55,018 providers still billing in 2024 (9 exclusions), the screen falls below a coin-flip rank alongside shorter prior tenure (38.0th vs. 50.0th percentile), an observed subgroup difference instead of an isolated causal effect.[2] Calibrated alert rates across the full provider list set realistic caseload goals for audit staff.[3]
Table 1. Subset Results vs. Full-List Calibrated Results for One Fraud Screen
Observed screening scores with Bayes base-rate rescaling of holdout precision across four test settings.
| Screen Measure |
Setting 1: Unfiltered
All Billers, Top 1% Cutoff
|
Setting 2: Filtered
Active-Biller Subset Only
|
Setting 3: Caseload
Top 5% Cutoff, Caseload Check
|
Setting 4: Peer Check
Peer-Group & Matched Checks
|
|---|---|---|---|---|
| Test Group Size | 95 | 487,3840.0195% base rate | 9 | 55,009active in 2024 | 95 | 487,3840.0195% base rate | 204 | 487,384230 vs. 1,150 |
| Risk Ranking Pattern | 0.724 rank scorefull list | 0.442 rank scoreactive billers | 0.724 rank scoreTop 5% | 0.818 rank scorepeer match |
| Precision: Holdout vs. Full List | 0.922% holdout0.186% full list | Not reported at a cutoff9 cases, wide bounds | 0.512% holdout0.103% full list | Not reported at a cutoffin-panel |
| Flags & Caught Cases | 4,844 flags | 9 caught9.47% | No cutoff scored | 24,295 flags | 25 caught26.32% | No cutoff scored |
| Lift & Audit Caseload | 9.53x lift29.0% of caseload | Not scored | 5.28x lift145.6% of caseload | Not scored |
| Active Months Peer Percentile | Months active keptno filter | 38.0th vs. 50.0th0.653 reweighted | Months active keptno filter | 38.0th vs. 50.0thpeer cells |
Table 1 Notes & * Modeled Estimates
Expand notes
Table 1 Notes (Observed Screening Results; Mode A Base-Rate Calibration): Each column states its own group size. The target is a formal LEIE financial fraud exclusion. Ranges are 95% CI bounds on AUC: full universe AUC = 0.724 (95% CI 0.667-0.777, 487,479 providers); active-only subset AUC = 0.442 (95% CI 0.212-0.664; 9 exclusions vs. 55,009 controls; exit-year reweighted AUC = 0.653). Holdout precision is rescaled to the 0.0195% full-population base rate.
Column Scope. All Billers, Top 1% Cutoff. N = 487,479 providers (95 exclusions + 487,384 controls); unit: billing NPI. Active-Biller Subset Only. N = 55,018 providers billing in 2024 (9 exclusions + 55,009 controls); unit: billing NPI. Top 5% Cutoff, Caseload Check. N = 487,479 providers; 16,688 national MFCU active cases; unit: billing NPI. Peer-Group & Matched Checks. N = 204 exclusions vs. 487,384 controls (peer ranks); 230 vs. 1,150 (matched); unit: billing NPI.
Model Crosswalk. Plain table labels map to standard study models. Raw Yearly Average / Simple Avg is an unadjusted OLS model where routine and outbreak budgets mix. County & Year FE is a Two-Way Fixed Effects (TWFE) panel model. Routine-Budget Analysis / 21-Year County Budget Model is a Two-Stage Least Squares (2SLS / Lewbel 2012) model that isolates routine base budget shifts. County Board Budget Rules uses Brown (2014) county board rules.
Two-Ruler Benchmark Summary. Mode A Calibration Applied to Observed Screening Results (Random Forest, 95 Post-Panel Exclusions vs. 487,384 Controls). Ruler 1: Active-Only & Holdout Check: 0.442 AUC Active-Only; 0.922% Top 1% Holdout Precision. In the subset of providers still billing in 2024 (9 excluded vs. 55,009 controls), observed AUC is 0.442 (95% CI 0.212-0.664; exit-year reweighted AUC is 0.653). Excluded providers sit at the 38.0th vs. 50.0th median peer percentile of active months. A holdout with 97,476 controls (20%) shows 0.922% Top 1% precision. Ruler 2: Full-List Base-Rate Calibrated Check: 0.724 AUC; 0.186% Top 1% Precision at 0.0195% Base Rate. Across all 487,479 providers (95 post-panel exclusions, a 0.0195% base rate), the same model reaches 0.724 AUC (95% CI 0.667-0.777). Rescaled to that base rate, the Top 1% cutoff gives 4,844 flags, 9 caught cases, 0.186% precision, and 9.53x lift over random choice.
Method & Data Audit (TooEarlyToSay.com): Medicaid Fraud Classifier Validation Guide → · Rare-Outcome Classifier Evaluation Workflow →
Table 2. How Filtering on a Shared Effect Shifts Screen Scores
Why subset scores should be rescaled to the full billing list before setting audit cutoffs.
| Choice Factor | Full Unfiltered List, Top 1% | Active-Biller Subset Only | Top 5% Cutoff, Caseload Check | Peer-Group & Matched Checks |
|---|---|---|---|---|
| Filter Path in Each Setting | Every billing provider stays in the list, so active months remain a model input. | Keeping only 2024 billers leaves 9 of 95 excluded providers after 90.5% exited before 2024. | The full list stays in place, while a wider cutoff sends more flags to staff. | Ranks are scored inside state and specialty peer cells, and matching reweights controls. |
| How the Setting Shifts Scores | Holdout precision (0.922%) is 4.96 times full-list precision (0.186%) due to 20% control sampling. | Ranks fall below a coin flip in the 9-case 2024 subset and recover when exit years are reweighted. | Full-list precision drops to 0.103% while flags rise to 24,295, or 145.6% of caseload. | Matched and balanced checks keep strong risk ranking on in-panel cases. |
| Staffing Impact for Audit Teams | At $23,719 per case, 4,844 Top 1% flags cost $114.9M to review and catch 9 cases. | With only 9 excluded billers left in 2024, active-only claims scores have wide bounds. | A Top 5% cutoff exceeds the 16,688-case national caseload, so cutoffs need capacity checks. | Peer-group ranks keep normal state and specialty billing differences from triggering flags. |
| Best-Fit Calibration Rule | Rescale Precision to the Full List. Adjust downsampled holdout precision to the 0.0195% full-list base rate before setting rules. | Keep All Providers in the Test Set. Score the full provider list first and show active-only results as a separate subgroup check. | Match Cutoffs to Staff Caseload. Compare total flags against the 16,688 active MFCU cases before picking a cutoff. | Rank Within Peer Cells. Score billing features inside state and specialty cells and report matched checks. |
Table 2 Notes: Bayes Base-Rate Rescaling, Subgroup Selection Structure & Data Provenance
Expand notes
• Target Definition: The positive class is a formal OIG LEIE exclusion under Specification A financial fraud statutes (1128(a)(1), 1128(a)(3), 1128(b)(1)). The prospective check scores the 95 providers excluded in 2025 or later against 487,384 non-excluded controls (487,479 total).
• Bayes Base-Rate Rescaling (Mode A Calibration): The holdout keeps all 95 cases and 97,476 of 487,384 controls (20%). False positives scale by 1 / 0.20 = 5. In odds form, Odds_pop = Odds_holdout / 5. For Top 1%, 0.922% / 99.078% = 0.009306, and dividing by 5 gives 0.001861, or 0.186% precision. For Top 5%, 0.512% rescales to 0.103%. Lift equals precision divided by base rate: 0.1858% / 0.0195% = 9.53x, and 0.1029% / 0.0195% = 5.28x.
• Observed Subgroup Difference & Exit-Timing Sensitivity: In the empirical data, 90.5% of post-panel fraud providers stopped billing prior to 2024 (vs. 43.6% of controls), and excluded providers sit at the 38.0th vs. 50.0th median peer percentile of active months (Months Active SHAP correlation r = -0.9009). Reweighting controls to match the fraud exit-year distribution shifts prospective AUC from 0.724 to 0.653, and restricting to the 55,018 providers still billing in 2024 (9 exclusions vs. 55,009 controls) yields an observed AUC of 0.442 (95% CI 0.212-0.664). These descriptive subgroup and reweighting comparisons document how conditioning on active 2024 billing alters the evaluation sample, but do not separately isolate early exit as the sole cause of the 0.442 subgroup score.
• Caseload & Cost: MFCU unit cost is $23,719 ($395.8M / 16,688 active cases, HHS-OIG FY 2024). Top 1% gives 4,844 × $23,719 = $114.9M and 4,844 / 16,688 = 29.0% of caseload. Top 5% gives 24,295 / 16,688 = 145.6%.
• Data Provenance: Values come from Mcaid/05_fraud_detection/paper/output/results.json (prospective_auc, active_subset_auc, top1_prec_holdout, top1_prec_pop, top1_flags, top1_captured, top1_lift, top5_*, mfcu_*, shap_months_active_r, matched_sample_rf_auc, entropy_balanced_rf_auc), Mcaid/05_fraud_detection/paper/main.tex (L398-402: 90.5% vs. 43.6% pre-2024 exit, 0.653 exit-year reweighted AUC), output/tables/tab6_economics.tex (N = 487,479, 0.0195%), and output/tables/tab_a3_base_rates.tex (peer percentiles).
Why Limiting a File to "Providers Still Billing" Shifts Every Ratio
Collider bias was first noted by Joseph Berkson in 1946 in hospital data. A related sample-selection shift appears in health claims when a team keeps only providers still billing in the latest year. In the linked Medicaid file, 90.5% of post-panel excluded billers stopped billing before 2024 (vs. 43.6% of controls), and reweighting controls by exit year shifts AUC from 0.724 to 0.653. In the subset of 55,018 providers still billing in 2024 (9 exclusions), the observed AUC is 0.442 (95% CI 0.212 to 0.664), reflecting both sample selection and the small number of remaining cases.
Downsampling shifts precision in a similar way. A holdout that keeps all 95 cases and 20% of controls reports 0.922% Top 1% precision. At the full-list base rate of 0.0195%, that same cutoff gives 0.186% precision, or 9 caught cases among 4,844 flags. Rescaling every subset score to the full billing list and checking the 16,688-case national MFCU caseload keeps agencies from sizing audit budgets on a filtered sample.
Data Sources, Model Inputs, Notes & Endnotes
(Numbered endnotes [1]–[6], state data sources & math rules)
Expand details
Case Study Focus: System Dynamics & Feedback · Selection Skew & Base-Rate Calibration (Mode A Calibration on Observed Screening Results)
Numbered Endnotes & Main Sources
- [1]Full Medicaid Billing Universe Evaluation (2018-2024): HHS Medicaid Provider Spending claims cover 487,479 billing providers (95 post-panel fraud exclusions vs. 487,384 non-excluded controls). Linked to OIG LEIE, the full-list ranking score is 0.724 AUC (95% CI 0.667-0.777, 9.5x lift in Top 1%).
- [2]Observed Active-Only Subgroup Comparison & Tenure Differences: In the subset of 55,018 providers still billing in 2024 (9 exclusions vs. 55,009 controls), the observed ranking score is 0.442 AUC (95% CI 0.212-0.664). Sanctioned providers also show shorter observed billing tenure (38.0th vs. 50.0th median peer percentile of active months; 90.5% exited before 2024 vs. 43.6% of controls, with exit-year reweighted AUC = 0.653). Because the 2024 active subset contains only 9 excluded providers and a wide confidence interval, the subgroup difference does not isolate early exit as the sole cause of the lower score.
- [3]Population Base-Rate Calibration: Exact Bayes base-rate math rescales Top 1% precision from the 20%-control holdout (0.922%) to the full 487,479-provider billing list (0.186%, vs. 0.0195% random base rate).
- [4]Berkson, J. (1946), "Limitations of the Application of Fourfold Table Analysis to Hospital Data", Biometrics Bulletin, 2(3), 47-53
- [5]Griffith, G. J., et al. (2020), "Collider bias undermines our understanding of COVID-19 disease risk and severity", Nature Communications, 11, 5749
- [6]OIG LEIE
Mode Classification (Mode A Calibration on Observed Screening Results): AUC scores (0.724 full list, 0.653 exit-year reweighted, and 0.442 active-only subgroup), holdout precision, flags, caught cases, lift, and peer percentiles are observed outputs in output/results.json, main.tex (L398-402), output/tables/tab6_economics.tex, and output/tables/tab_a3_base_rates.tex. Rescaling from holdout to full-list precision is a Mode A calibration using Bayes rule. It matches the full-population precision in output/results.json (0.1858%).
Primary References & Data Sources: Berkson, J. (1946), "Limitations of the Application of Fourfold Table Analysis to Hospital Data", Biometrics Bulletin, 2(3), 47-53; Griffith, G. J., et al. (2020), "Collider bias undermines our understanding of COVID-19 disease risk and severity", Nature Communications, 11, 5749; HHS Medicaid Provider Spending; OIG LEIE; and HHS-OIG Medicaid Fraud Control Unit FY 2024 annual statistics.
Companion Method & Code Guides (TooEarlyToSay.com)
- Can a Classifier Find What Investigators Miss?. Temporal validation and interpretable baselines for fraud screening; the article-reported classifier metrics are under reconciliation because sample sizes and base rates conflict.
- Imbalanced Classification: An Article-Reported Example. Instructional scikit-learn workflow for rare outcomes, including why a model that labels every case negative can score about 91% accuracy; dataset counts and metrics are article-reported.
Common Questions & Method Notes
- What is a "collider" in a causal diagram, and how does it differ from a confounder?
- A confounder is a shared CAUSE of both the treatment and the outcome (A <- C -> Y). Analysts control for it to remove bias. A collider is a shared EFFECT of two variables (A -> C <- Y). Filtering or controlling on a collider opens a false link between its causes.
- How does the active-only subgroup score compare with the full-list score?
- Across all 487,479 providers, the screen scores 0.724 AUC (95% CI 0.667 to 0.777). Because 90.5% of post-panel excluded providers stopped billing before 2024 (38.0th vs. 50.0th active-months percentile), reweighting controls by exit year shifts AUC to 0.653, and the 2024 active-biller subset (9 exclusions vs. 55,009 controls) shows an observed AUC of 0.442 (95% CI 0.212 to 0.664). That subgroup difference reflects sample restriction and the small 9-case sample instead of isolating a single cause.
- How should analysts report precision from a downsampled holdout?
- Rescale it to the full-list base rate. The holdout here keeps every case and 20% of controls. Top 1% precision of 0.922% rescales to 0.186% across the full billing list (0.0195% base rate), or 9 caught cases among 4,844 flags.