Benchmarks
Episodes are designed to support comparisons at the same points in a case. Clinical Reasoning Bench, in development, examines decisions as information becomes available, including predictions made before test results are revealed.
Data
What a visit becomes
Each visit is designed to become an episode: one dated record of what was expected, what was found and what happened next.
An episode
Visit
Persistent tiredness
The model is asked to reason as the record becomes available.
Initial differential· ranked
- 1Anaemia
- 2Disrupted sleep
- 3Thyroid dysfunction
Findings and orders
Symptom history recorded
OrderBlood count requested
Results sealed
ExpectedHaemoglobin below the reference range
Test result withheld at this checkpoint.
- Committed reasoning
- The model ranks anaemia first and records what it expects the test to show.
Results revealed
FoundHaemoglobin within the reference range
Earlier expectation: retained alongside the new finding.
- Diagnosis and plan
- No final diagnosis established. The model revises its initial explanation and proposes further review.
Follow-up
Still reported, cause unresolved
Tiredness is still reported. The cause remains unresolved in this illustrative record.
The dates, reasoning and findings here are fictional. They show the shape of an episode.
Follow-up adds evidence obtained later. The evaluation method is designed to compare a model’s predictions with that outcome, including when the record remains unresolved.
What episodes are for
Datasets
Research records are designed to keep the question, the evidence available at the time and the later findings together. The sequence gives context to an answer.
RL environments
Episodes are intended to support reinforcement learning environments, designed to let a model work through decisions as new evidence arrives.
Safety evaluations
The method is designed to support evaluation against the outcome at follow-up. Keeping predictions separate from later evidence makes the basis of a decision available for review.
Research papers
The dated record is designed to support reports of the method, the evidence and its limits. Research publications are an intended output of this work.
Research records
De-identification
The preparation method is designed to separate identifying information from research records and retain a record of what was changed or withheld. This work remains in development.
Access to the sample
Researchers request access with a short form. Once approved, they sign in at crbench.arcophos.com with an emailed link to inspect the Clinical Reasoning Bench sample. The public sample and tools below can be explored directly.
Public data and tools
Health Optimization Bench sample
The public sample includes tasks, rubric criteria and reference responses. It is available under CC BY-NC 4.0.
Open evaluation harness
Arcophos Evals provides an Inspect task pack and a reference runner with the same grading protocol. The code is MIT licensed. A sample run is not expected to reproduce a full leaderboard score exactly.
Published results and sources
Clinical Benchmarks Index result tables and source records are available as CSV and JSON under CC BY 4.0. Each published result can be traced to its supporting document.