Longitudinal evaluation of AI agentsHow we checked the grader ↗

ClinOpsBench: Trial Oversight v1.0

19 concern families

84 graded requirements ⓘ

84 graded requirements: 8 critical, 53 major, 5 minor, 18 calibration. The 53 major requirements are grouped into 17 major concerns.

42 timed transitions15 models · 45 runs

An expert-curated simulated trial with 32 participants, 4 sites and 3,799 structured record versions. Explore the world ↗ · 31-action tool API

This publication covers one public version of the environment. Private held-out worlds are available on request.

Open evaluation workspace

The strongest frontier agents catch most critical issues. None met the complete oversight standard

9 of 45 runs met all eight critical requirements. But every run left required follow-up open, and most closed or cleared issues before the evidence supported it.

Key findings · across all 45 runs

40/45runs

Issues closed or cleared without enough evidence

On critical or major concerns, agents marked items verified, resolved or clear before the records supported it, without correcting those conclusions.

Example: follow-up closed before all requested evidence was supplied ↗

40/45runs

Findings the records didn’t support

Agents raised findings the records didn’t support, such as reporting a check as not performed when only its documentation was missing.

Example: a record conflict reported as a proven violation ↗

45/45runs

Follow-up left open

Every run left required work incomplete in at least 4 of 17 major areas. The median run completed 24 of 42 required actions by the end of the trial; 3 runs completed none.

See open follow-up in a run ↗

15/15models

The final report looks better than the record behind it

On average, every model’s final report scored higher than its running oversight record, and the same held in 44 of 45 runs.

Compare record vs. report ↗

Runs can appear in more than one count. Each example is a single case.

Even without the major-follow-up threshold, every run still misses a critical requirement or leaves an unsupported clearance or closure in place.

15 models, 3 runs each, evaluated in the published environment. A benchmark standard, not a regulatory determination. Methods ↗

Compare the models

Three runs per model. Select a model or run to inspect it.

Find a model

Sorted by Composite, highest first. Scores do not replace the readiness standard.

ModelReadinessPassing runs / 3Critical metMean / 8 · rangeIncomplete majorMean / 17 · rangeCompositeMean · 95% intervalScore / 100Mean, 95% interval and individual runs

0 of 45 runs meet the readiness standard. Mean 95% bootstrap interval Individual runs.Overlapping intervals are not reliably distinguished from GPT-6 Astra, the selected model. Three runs do not establish a definitive ranking. Hover a row for its sample standard deviation.

GPT-6 Astra · Composite. No other model’s 95% bootstrap interval overlaps in this comparison. With only three runs, this does not establish a definitive ranking.

Inspect a run

Model

Inside the result

GPT-6 Astra

Run 1 · frozen diagnostic scores

State63.47The evolving oversight record

Trajectory85.01Investigation and follow-through

Outputs86.08The final report

Composite · diagnostic78.19Average of the three axes

29 / 42 required transitions attained7 / 8 critical requirements met5 / 17 incomplete major concerns

These saved criteria make up the frozen diagnostic scores above. Scores reflect rubric ratings, not the share of work completed.

State breakdown

Correctness50.00

40% of this axis

Saved ratings: 21 of 21 checkpoints at 2/4.

Calculation

Mean rating 2 / 4 × 100 = 50.00. Displayed ratings are rounded; scores use the saved values.

All saved rating counts: 21 of 21 checkpoints at 2/4.

Continuity100.00

25% of this axis

Saved ratings: 21 of 21 checkpoints at 4/4.

Calculation

Mean rating 4 / 4 × 100 = 100.00. Displayed ratings are rounded; scores use the saved values.

All saved rating counts: 21 of 21 checkpoints at 4/4.

Timeliness52.76

35% of this axis

Most common: 5 of 20 checkpoints at 2.67/4; 15 other saved ratings in Calculation.

Calculation

Mean rating 2.11 / 4 × 100 = 52.76. Displayed ratings are rounded; scores use the saved values.

All saved rating counts: 4 of 20 checkpoints at 0/4; 1 of 20 checkpoints at 2.33/4; 2 of 20 checkpoints at 2.4/4; 2 of 20 checkpoints at 2.5/4; 2 of 20 checkpoints at 2.57/4; 5 of 20 checkpoints at 2.67/4; 2 of 20 checkpoints at 2.8/4; 2 of 20 checkpoints at 3/4.

Trajectory breakdown

Investigation82.38

45% of this axis

Most common: 11 of 21 checkpoints at 4/4; 10 other saved ratings in Calculation.

Calculation

Mean rating 3.3 / 4 × 100 = 82.38. Displayed ratings are rounded; scores use the saved values.

All saved rating counts: 5 of 21 checkpoints at 2/4; 4 of 21 checkpoints at 3/4; 1 of 21 checkpoints at 3.2/4; 11 of 21 checkpoints at 4/4.

Verification81.19

35% of this axis

Most common: 12 of 21 checkpoints at 4/4; 9 other saved ratings in Calculation.

Calculation

Mean rating 3.25 / 4 × 100 = 81.19. Displayed ratings are rounded; scores use the saved values.

All saved rating counts: 7 of 21 checkpoints at 2/4; 1 of 21 checkpoints at 3/4; 1 of 21 checkpoints at 3.2/4; 12 of 21 checkpoints at 4/4.

Follow-up97.62

20% of this axis

Saved ratings: 1 of 21 checkpoints at 2/4; 20 of 21 checkpoints at 4/4.

Calculation

Mean rating 3.9 / 4 × 100 = 97.62. Displayed ratings are rounded; scores use the saved values.

All saved rating counts: 1 of 21 checkpoints at 2/4; 20 of 21 checkpoints at 4/4.

Outputs breakdown

Correctness91.18

30% of this axis

Based on 17 finding ratings. Individual rating counts are not saved.

Calculation

Mean rating 3.65 / 4 × 100 = 91.18. Displayed ratings are rounded; scores use the saved values.

Coverage74.69

30% of this axis

Based on 19 concerns. Individual rating counts are not saved.

Calculation

Mean rating 2.99 / 4 × 100 = 74.69. Displayed ratings are rounded; scores use the saved values.

Evidence88.24

20% of this axis

Based on 17 finding ratings. Individual rating counts are not saved.

Calculation

Mean rating 3.53 / 4 × 100 = 88.24. Displayed ratings are rounded; scores use the saved values.

Actions91.18

15% of this axis

Based on 17 finding ratings. Individual rating counts are not saved.

Calculation

Mean rating 3.65 / 4 × 100 = 91.18. Displayed ratings are rounded; scores use the saved values.

Clarity100.00

5% of this axis

Based on 1 portfolio rating. Individual rating counts are not saved.

Calculation

Mean rating 4 / 4 × 100 = 100.00. Displayed ratings are rounded; scores use the saved values.

Example from this run

GPT-6 Astra · Run 1

Dosing after progression was left without follow-up

What the records showed: Participant 022-001 had investigator-assessed progressive disease on 4 March 2027.
Treatment was then administered on 11 March 2027, 18 March 2027, and 1 April 2027.
What a monitor would do: Ask the site and medical monitor to reconcile the progression assessment with subsequent dosing and document the treatment decision.
What the agent did: The agent did not raise a finding or follow-up request about participant 022-001’s progression and later dosing, and did not mention the issue in its final report.
Result: Missed critical requirement; this run fails the readiness standard.
A query is required; continued treatment may still be justified.

Source records and scoring details

tumor_assessments.csv, row 19 (participant 022-001) ↗
treatment_exposure.csv, row 75 (participant 022-001) ↗
treatment_exposure.csv, row 76 (participant 022-001) ↗
treatment_exposure.csv, row 77 (participant 022-001) ↗
EV-PROTOCOL-A9.pdf, page 3 ↗
EV-PROTOCOL-A9.pdf, page 40 ↗

tumor_assessments.csv, row 19 (participant 022-001) (available from checkpoint 9): SUBJID: 022-001; RSDTC: 2027-03-04; OVERALL_RESPONSE: Progressive disease; ASSESSOR: Investigator
treatment_exposure.csv, row 75 (participant 022-001) (available from checkpoint 11): subject_id: 022-001; administered_datetime: 2027-03-11T10:59:00; decision: administered
EV-PROTOCOL-A9.pdf, page 3 (available from checkpoint 0): Patients will continue to receive study treatment until disease progression, unacceptable toxicity, investigator decision, consent withdrawal, start of a subsequent anticancer therapy, pregnancy, or study termination by the sponsor.
EV-PROTOCOL-A9.pdf, page 40 (available from checkpoint 0): The investigator will make treatment decisions based on site assessments of scans by RECIST. Clinical response of CR, PR, SD, or PD will be determined at each assessment. Response and progression will also be assessed by an IRF.

Assessed requirement: Recognize recorded progression followed by treatment and assign accountable source reconciliation.
Recognize recorded progression followed by treatment, and assign accountable follow-up to reconcile the source records and treatment decision, directly or through clearly linked work. Use only evidence available at that checkpoint. This does not require every participant or dose to be listed, and does not establish unauthorized treatment or harm.
Saved assessment: The saved review found no accountable reconciliation of recorded progression and subsequent treatment in this run’s submitted and linked work.
Review classification: Missing critical requirement
Scoring treatment: This missing critical requirement fails the critical-coverage readiness gate. It is not a separately counted error penalty.

Inspect final report ↗

Additional analysis Not part of the diagnostic composite

Error-adjusted score

This separate analysis subtracts error penalties from requirement coverage. The review is still in progress and does not change the diagnostic scores.

Requirement coverage80.9Required work completed, weighted by importance

Error penalty24.1Points deducted for reviewed errors

Net attainment56.7Coverage less penalty, floored at zero

Rounded to one decimal. The score cannot fall below zero.

The measures below use separate definitions. They are not the criteria used to calculate the axes above.

Record precision90.2Correctness, evidence and actions in filed records

Transition timing67.9All 42 required transitions, with partial credit for late completion

Investigation summary83.8Investigation and verification on benchmark-authored checks

Errors identified · review in progress

False reassurance / early closure 2Unsupported allegations 3Wrong attribution 0Missing evidence 1

How errors are reviewed ↗

Detailed results All metrics, sensitivity and verification

Saved diagnostic means and readiness across three runs per model. Scores are out of 100.
State	Trajectory	Outputs	Composite	ReadinessPassing runs / 3	Critical metMean / 8 · range	Incomplete major concernsMean / 17 · range
65.8	85.0	87.1	79.3	0 / 3	7.0 / 8Range 7.0–7.0	5.3 / 17Range 4.0–7.0
66.5	76.6	79.1	74.0	0 / 3	7.0 / 8Range 7.0–7.0	8.7 / 17Range 8.0–10.0
63.8	70.5	78.3	70.9	0 / 3	7.0 / 8Range 7.0–7.0	6.3 / 17Range 6.0–7.0
61.8	71.3	79.3	70.8	0 / 3	7.3 / 8Range 7.0–8.0	6.7 / 17Range 4.0–8.0
66.5	65.0	78.9	70.1	0 / 3	7.7 / 8Range 7.0–8.0	9.7 / 17Range 9.0–11.0
66.0	67.7	74.9	69.6	0 / 3	7.3 / 8Range 7.0–8.0	6.7 / 17Range 6.0–7.0
64.7	58.0	75.4	66.0	0 / 3	8.0 / 8Range 8.0–8.0	8.0 / 17Range 6.0–10.0
63.2	58.3	74.0	65.2	0 / 3	7.3 / 8Range 7.0–8.0	10.0 / 17Range 10.0–10.0
57.9	59.2	72.8	63.3	0 / 3	7.3 / 8Range 7.0–8.0	11.0 / 17Range 11.0–11.0
58.7	60.5	66.6	61.9	0 / 3	7.0 / 8Range 7.0–7.0	11.7 / 17Range 11.0–12.0
53.6	53.3	76.6	61.2	0 / 3	7.0 / 8Range 7.0–7.0	12.3 / 17Range 12.0–13.0
55.1	58.7	66.5	60.1	0 / 3	7.0 / 8Range 7.0–7.0	13.3 / 17Range 13.0–14.0
49.7	57.4	61.7	56.3	0 / 3	7.0 / 8Range 7.0–7.0	13.0 / 17Range 12.0–14.0
44.1	34.9	53.9	44.3	0 / 3	2.3 / 8Range 0.0–6.0	16.7 / 17Range 16.0–17.0
6.7	29.1	18.1	17.9	0 / 3	0.3 / 8Range 0.0–1.0	17.0 / 17Range 17.0–17.0

0 of 45 runs meet the readiness standard. An authored reference record does meet it. See the check ↗

Diagnostic scores are frozen. The error review is a separate analysis and does not change them.

The error review is still in progress: some corrective-action and missing-documentation judgments remain under review. Methods and limitations ↗

What if errors carried a different penalty?

This compares the separate error-adjusted score under three penalty choices. The current audit uses 3.75 points for errors with an observed consequence. None changes the diagnostic composite.

Model	2.5 pointsNet mean · range	3.75 pointsNet mean · range	5.0 pointsNet mean · range
GPT-6 Astra	65.8Range 62.4–68.7	60.5Range 56.7–64.3	55.2Range 51.1–59.9
GPT-6 Luna	58.5Range 55.9–63.0	54.5Range 52.0–58.6	50.4Range 47.3–54.2
Claude Opus 5.5	47.6Range 41.0–57.9	35.1Range 27.6–47.9	22.6Range 11.3–37.9
Claude Sonnet 5.5	54.4Range 51.4–58.8	47.5Range 42.7–55.1	40.6Range 33.9–51.3
GLM 5.3	49.7Range 41.6–59.3	39.0Range 26.6–49.9	28.4Range 11.6–40.5
Claude Fable 5.1	41.2Range 26.7–64.5	26.0Range 4.8–56.3	16.5Range 0.0–48.2
Qwen 3.8-2.4T-A95B	30.6Range 17.9–41.2	13.0Range 0.0–20.0	1.8Range 0.0–5.3
MiMo V2.6 Pro	32.9Range 24.0–39.2	17.9Range 7.8–25.4	5.8Range 0.0–11.7
Kimi K3	33.1Range 24.1–40.8	18.5Range 7.3–28.3	7.0Range 0.0–15.8
GLM 5.3 Flash	32.2Range 11.9–45.1	20.7Range 0.0–35.1	13.2Range 0.0–25.1
GPT-5.6 Terra	46.0Range 45.0–47.1	43.9Range 41.4–47.1	41.8Range 37.0–47.1
Claude Sonnet 5	35.8Range 26.8–46.2	27.8Range 17.5–42.5	19.9Range 8.1–38.7
Gemini 3.8 Flash	19.5Range 12.9–31.2	6.0Range 0.0–18.0	1.6Range 0.0–4.9
Gemini 3.1 Pro	6.2Range 0.0–18.5	4.5Range 0.0–13.5	2.8Range 0.0–8.5
Gemini 3.5 Flash Lite	0.0Range 0.0–0.0	0.0Range 0.0–0.0	0.0Range 0.0–0.0

Error examples

Examples of failures

Three errors from the provisional review. These examples do not show a model’s overall performance or how often each error occurs.

GPT-6 Astra · Run 1

A dose hold still needs a clear reason

Task: Check the reason for withholding a participant’s dose.
What the agent wrote: “Correct withholding for 022-006 is a negative control, not a finding.”
Checkpoint 6 · summary
What the evidence showed: The blood test showed a neutrophil count (ANC) of 3.7 (grade 0), within the recorded 1–7.5 range. The dose record cited unmet local criteria but did not include them. Keep the question open and ask for the missing criteria; do not declare the hold correct or incorrect.
Lesson: A recorded decision is not proof that it was justified.

Unsupported assurance2.5 penalty points for this error

View the run ↗

Claude Opus 5.5 · Run 1

A visit is not the same as medical follow-up

Task: Track the participant’s medical assessment after a pre-dose neuropathy breach.
What the agent wrote: “Sponsor documented the trigger decision in MTG-2027-02-10 (medical monitor present ...)”
Checkpoints 11 and 20 · follow-up updates
What the evidence showed: The agent marked follow-up complete without evidence of the required medical assessment for this participant. A meeting, monitoring visit and recovery did not prove that assessment happened. Keep it open or assign the remaining work to a named owner.
Lesson: Check that all required work is complete before closing an issue.

Premature completion5.6 penalty points for this error

View the run ↗

Qwen 3.8-2.4T-A95B · Run 1

The right dates, the wrong starting point

Task: Determine whether a response assessment fell within the protocol’s eight-week ± seven-day window.
What the agent wrote: “First post-baseline response assessment for 014-001 performed on day 64 — one day outside the protocol’s every-8-weeks ±7-day window; no deviation recorded.”
Checkpoint 8 → final report
What the evidence showed: The protocol starts the clock at the first treatment day, not the baseline scan. January 4 to March 1 is 56 days, within the window. The final report still told sites and the CRO to record deviations.
Lesson: Check the required start date before reporting a timing error.

Unsupported allegation3.8 penalty points for this error

View the run ↗

Connect with us

Evaluate your model on a private held-out world, request environments for post-training, or explore access to graded trajectories and the full run set.

Open evaluation workspace