Healthcare revenue cycle is moving toward autonomous operations. AI systems are increasingly expected to classify denials, identify appropriate next steps, and recognize when human intervention is necessary. But before they can be trusted with those decisions, someone has to define what “right” actually means.
Key takeaways
- High scores alone are insufficient. In a 20-case repeatability experiment, three frontier models recorded closely overlapping results, limiting any claim of a durable winner.
- Benchmark design can manufacture errors. Subsequent 100-case experiments surfaced ambiguous answer keys, including defensible alternate workflows that scoring rules initially penalized.
- Operational performance is multidimensional. Grounding, deadline judgment, critical failures, API reliability, and latency must be reported alongside decision scores.
- Synthetic decisions are not recovered dollars. The present research does not establish overturn rates, autonomous execution reliability, or collections improvement.
The problem with evaluating denial AI
Consider a claim denied because the submitted authorization number is wrong. A valid authorization exists; the error is in the claim data. The operational resolution could be a corrected claim or another payer-prescribed administrative step. Whether either action is appropriate depends on facts a generic benchmark may omit.
That problem is fundamental: classification, evidence interpretation, workflow selection, and material risk judgment are not the same task. A model can correctly identify the issue and still recommend the wrong next step. Equally, a benchmark can incorrectly mark a defensible next step as wrong because its answer key is too narrow.
The goal of DenialsBench is to make these distinctions measurable rather than burying them within a single score.
How DenialsBench evaluates a decision
DenialsBench uses controlled synthetic denial scenarios with a frozen evaluation framework. Each model receives the supplied facts and must return structured operational judgments: a denial category, recommended action, escalation assessment, deadline state, and grounded supporting claims.
The initial framework contains 20 denial scenario families, including authorization, eligibility, timely filing, coding, documentation, coordination of benefits, payment issues, and cases with insufficient information. Primary validation banks include five surface variations per family for 100 cases per model.
Case facts and answer keys are defined before a model receives a scenario. Scoring changes require a new benchmark version and fresh cases rather than retroactive changes to historical results.
What the first repeatability experiment found
Run C evaluated four frontier models on the same 20 synthetic cases, repeated independently three times. That produced 60 scored decisions per model under rubric v0.2.0. The objective was not to identify a winner; it was to test whether small performance differences were stable enough to interpret.
Claude Opus 5 averaged 90.87. GPT-5.6 Sol averaged 90.17. Gemini 3.1 Pro Preview averaged 90.03. Their observed ranges overlap substantially. Grok 4.6 averaged 86.43 but showed the smallest run-to-run standard deviation in this small sample.
All four valid Run C model configurations completed their 60 requests with zero API or parse failures, zero framework-defined critical failures, and zero grounding violations. These are limited, protocol-specific findings—not a certification of production safety.
Speed was not equivalent
Operational systems cannot ignore response time. The same benchmark that produced relatively close scores for three models also revealed substantial differences in mean and long-tail latency under the tested configurations.
Mean response times ranged from 3.33 seconds for GPT to 15.94 seconds for Grok. The longer tail also matters: Grok's P95 response time reached 28.17 seconds, while GPT and Claude were both below five seconds in this test. A separate low-reasoning experiment for Grok suggested a materially faster configuration at nearly the same average score, but it was exploratory and should not be compared as if fully replicated.
These results suggest that the useful unit of evaluation is a performance profile combining decision quality, variation, latency, failure severity and implementation economics—not a leaderboard position.
How the later scoring framework works
Subsequent DenialsBench revisions separated core judgment from administrative workflow choice and output completeness. The frozen v0.4 rubric allocates 80 points to core decision quality, 15 to workflow selection, and 5 to structured response completeness.
Critical failures are reserved for materially unsafe actions, unsupported factual assertions, missing explicit urgent or expired deadlines, or invalid structured responses. Merely choosing a less-preferred but defensible administrative path should not automatically trigger the same severity.
The benchmark exposed flaws in the benchmark
Later 100-case experiments made a crucial limitation visible. In some scenarios, models interpreted the facts correctly yet selected administrative actions that differed from the locked label. An appeal versus reconsideration distinction, for example, may depend on payer-specific procedural requirements. A rigid answer key risks mistaking workflow diversity for reasoning failure.
Another audit revealed cases in which monitoring an open reconsideration was classified as critical despite the absence of a clear, irreversible loss of claim rights. Reviewers concluded that the action could be suboptimal without necessarily being materially unsafe.
The publication response was conservative. We retained the validation results and documented the scoring issues rather than declaring a winner based on a rubric with known ambiguity. A later provider transport incident was also excluded from model-quality comparisons because failed API responses do not constitute evidence of poor reasoning.
From benchmark scores to financial outcomes
DenialsBench currently measures reasoning about synthetic denial facts. It does not test full payer portal execution, clinical-document retrieval, automatic appeals filing, or realized reimbursement recovery. Those tasks require separate research environments and different forms of evidence.
For revenue cycle operators, the most valuable questions are not simply which model answers correctly, but whether the system recognizes missing evidence, avoids avoidable harm, executes correctly, reduces human rework, and increases net collections at an acceptable fully loaded cost.
A strong synthetic benchmark score establishes a hypothesis about capability. Workflow testing evaluates whether the system can act reliably. Financial measurement establishes whether it creates measurable value.
What DenialsBench will test next
The next phase should strengthen practitioner adjudication, expand truly held-out examples, challenge models with ambiguous or adversarial scenarios, and test their choices against payer-specific requirements where appropriate. Only after the methodology is defensible should the program produce a stronger comparative release.
The longer-term agenda expands to coding, eligibility, prior authorization and payment variance workflows—and, ultimately, to task completion and economics. The central objective is not to generate a durable model ranking. It is to establish a reproducible method for determining when AI can be trusted with consequential revenue cycle decisions.
The next frontier for healthcare AI is not merely generating better answers. It is demonstrating better decisions, safer execution, and measurable economic outcomes.
Appendix: methods and supporting data
A. Run C performance summary
| Model | Mean /100 | Range | SD | Mean sec. | P95 sec. |
|---|---|---|---|---|---|
| Claude Opus 5 | 90.87 | 89.0–93.3 | 1.80 | 3.40 | 4.94 |
| GPT-5.6 Sol | 90.17 | 88.0–92.5 | 1.84 | 3.33 | 4.88 |
| Gemini 3.1 Pro Preview | 90.03 | 88.0–91.3 | 1.45 | 6.04 | 9.73 |
| Grok 4.6 | 86.43 | 85.3–87.0 | 0.80 | 15.94 | 28.17 |
B. Protocol and reporting boundaries
Run C repeated 20 synthetic scenarios three times per flagship model under the v0.2.0 rubric. Case facts and answer keys are locked in advance. Subsequent v0.2–v0.4 experiments each involved fresh or frozen 100-case validation banks, and their findings should not be combined into an undifferentiated leaderboard. A fourth v0.4 provider did not produce a valid complete result because of transport failures.
C. Source material
- RevCycleAI Research, Run C repeatability analysis — model results, run variation, latency and reporting limitations.
- RevCycleAI Research — published DenialsBench scorecard and overview.
- DenialsBench v0.1 methodology, v0.2 Run B audit, v0.3 primary validation and case-bank audit, and v0.4 frozen scoring rubric and transport incident record, maintained in the RevCycleAI research codebase.
- R1 R37 research on long-context medical reasoning and appeal generation — a related but distinct evaluation task.