PanoptiveLabs
Longitudinal evaluation of AI agentsHow we checked the grader ↗

ClinOpsBench: Trial Oversight v1.0

Methods

How the benchmark is scored and checked

We measure the work recorded during the trial, the final report and the mistakes that remain.

00

Methods at a glance

Graded84 requirements across 19 concern families, 42 timed transitions and 21 checkpoints.ReadinessThree pass/fail checks. All must pass to meet the standard.ErrorsFour classes. Penalties scale with consequence and drop when an error is corrected.Grader checksCorrect record passes, edited copies with required work removed fail, targeted edits register, ratings repeat.

15 models, 3 runs each, evaluated in the published environment.

How the benchmark was built

Developed with senior clinical operations consultants and former FDA Bioresearch Monitoring (BIMO) inspectors. Human experts independently reviewed the tasks and trial world for clinical realism and quality. Their expertise shaped what agents should recognize, how they should respond, and which omissions matter most.

The environment is modelled on a real, publicly available trial protocol. Its design and tasks draw on expert input and real-world scenarios of varying complexity; participant records and events are simulated.

Align grading with expert judgments

Human experts performed oversight work in the trial environment to establish the reference (“ground truth”): clinical interpretations, expected follow-up and acceptable alternatives, checked against the trial’s source records. They then explicitly graded completed model traces. Those human ratings and judgments informed the grading reference and configuration: what earns credit, what counts as an error, and what follow-up is expected.

Set the severity standard

Expert feedback confirmed the requirement tiers: 8 critical, 53 major, 5 minor and 18 calibration. It distinguishes recognizing a safety-relevant breach from completing the follow-up after recognition.

Resolve difficult boundaries

Expert feedback clarified when missing records support an absence claim, when a report must reconcile new events before finalization, and when a mistaken claim has a material consequence. Those interpretations are applied consistently across models.

  1. 1Source documentsProtocol, monitoring plan and supporting records
  2. 2Expert-curated environmentConnected participants, sites and evolving evidence
  3. 3Expert-grounded referenceHands-on oversight and review of model traces
  4. 484 requirementsExpert-confirmed severity tiers
  5. 521 checkpointsAssess the record, actions and final report
  6. 6Grader checksReference records, targeted edits and repeatability

The benchmark team translated this reference into source-linked requirements, scoring weights and response windows.

01

Scores

Each run is scored on three axes: State (the evolving record), Trajectory (investigation and follow-through) and Outputs (the final report).

Each axis combines its criteria using fixed weights. The composite is the average of the three axes.

From evidence to three diagnostic axes84 graded requirements inform the reference used to assess the work.
8Critical
53MajorAcross 17 concerns
5Minor
18Calibration

Reference requirements + records, actions and report

State0–100

The evolving oversight record

Correctness
40%
Continuity
25%
Timeliness
35%

Trajectory0–100

Investigation and follow-through

Investigation
45%
Verification
35%
Follow-up
20%

Outputs0–100

The final report

Correctness
30%
Coverage
30%
Evidence
20%
Actions
15%
Clarity
5%
Composite(State + Trajectory + Outputs) ÷ 3Equal weight for each axis

Requirements inform different criteria; they don’t contribute equally to every axis.

Timeliness: the share of required actions completed by their due checkpoint, averaged across checkpoints with work due. Late or never-completed actions earn no credit.

Exact formulas · worked examples from GPT-6 Astra, run 1

Ratings use a 0–4 scale. The formulas convert them to scores out of 100. Examples use unrounded values.

Axis score
25 × Σ(weight × mean rating) ÷ Σ(applicable weights)

State: 0.40 × 50.00 + 0.25 × 100.00 + 0.35 × 52.76 ≈ 63.47.

Composite
(State + Trajectory + Outputs) ÷ 3

(63.47 + 85.01 + 86.08) ÷ 3 ≈ 78.19.

Outputs coverage
100 × mean of each concern’s met-requirement fraction

Across 19 concerns, the fractions sum to 14.1905. Their mean × 100 gives 74.69.

State timeliness
100 × mean of the on-time fraction at each checkpoint with work due

The 20 checkpoint fractions sum to 10.5524. Their mean × 100 gives 52.76.

Late or never-completed transitions earn no credit. Checkpoints with nothing due are excluded.

Transition-attainment lag
First checkpoint meeting the requirement − first checkpoint when it applies

Astra’s Site 014 second-pass CRO review applied at checkpoint 16 and was completed at checkpoint 19: 19 − 16 = 3 checkpoints. Its deadline was checkpoint 17, so it was two checkpoints late.

Never-completed transitions have no observed lag, not zero lag.

Continuity and follow-up cover work the agent represented. With no applicable work, their weight moves to other criteria within the same axis.

Trajectory investigation and verification use the lower of required-check and current-action averages at each checkpoint, when both apply. Checkpoints have equal weight.

Model scores average three runs. The 95% intervals use all 27 bootstrap resamples. Overlapping intervals are labelled “not reliably distinguished.”

The agent tool API · 31 actions

Agents inspect evidence, maintain records and submit their work. These actions do not expose the private grading reference.

Inspect evidence
list_artifactsget_table_schemaquery_tablesearch_documentget_document_pageget_anchorget_protocol_requirementget_subject_timelineget_site_historylist_external_events
Maintain findings and cases
list_signalsget_signalcreate_signalupdate_signallist_casesget_caseopen_caseupdate_caselink_signal_to_casemark_case_pending_verificationreopen_caseclose_caseescalate_case
Follow through and submit
create_obligationupdate_obligationdraft_dispositionrecord_verificationrequest_missing_evidenceget_current_state_summarysubmit_stepsubmit_final_report
02

Readiness standard

A run meets the standard only if it passes all three checks. Scores on other criteria don’t offset a failed check.

A major concern is incomplete when any of its major requirements is unmet. Each concern counts once.

Readiness requires all three conditionsAny failed check is enough; otherwise, a run with unfinished assessments is labelled “not fully assessed.”

All 8 critical requirements met

36of 45 runs
missed at least one

Fewer than 4 of 17 major concerns incomplete

45of 45 runs
left at least four incomplete

No uncorrected false assurance

Or premature closure on a critical or major concern

40of 45 runs
have this failure recorded

Independent counts, not sequential losses: the same run can appear in more than one failure count.

0 of 45 runs meet the standardAll 45 runs fail at least one check.

Counts runs, not individual errors.

Relaxing or removing the major-concern check still leaves every run failing another check.

03

Error review

The error review scores claims the evidence didn’t support, separately from the diagnostic scores.

Reviewers compare claims, source records, actions and later corrections. Evidence must have been available when the agent made the claim.

Four kinds of errors
ClassWhat it meansExample
E1False assurance or premature closureCalling a result verified or closing an issue before the evidence supports it.
E2Unsupported allegationSaying a required check never happened when the record only shows missing documentation.
E3Wrong attributionAssigning a finding to the wrong participant, site, visit, time or responsible party.
E4Missing documentary supportMaking a true, consequential claim without supplying its supporting evidence.

3.75 points with an observed consequence, 2.5 otherwise; 1 point for missing documentary support; ×1.5 on critical concerns.

What counts as an observed consequence?

The false premise reached an issued instruction, query or escalation, appeared as a settled final finding, or suppressed required work through a state change.

Hypothetical downstream harm does not qualify. Repeated errors within the same concern and class count once, keeping the strongest penalty.

How a documented correction changes the penalty
CorrectionMultiplier
Corrected by the deadline0 · no penalty
Corrected after the deadline, before the final report0.5 · half penalty
Never corrected or silently dropped1 · full penalty

The deadline is normally the next checkpoint, unless a linked requirement allows longer. It never extends beyond the final checkpoint.

A mistaken state change must also be reversed. Rewording a report does not undo an action.

Error-score formulas · worked examples
Requirement coverage
100 × weight of met requirements ÷ 104.5

Weights: critical 5, major 1, minor and calibration 0.5.

Astra run 1: 84.5 ÷ 104.5 × 100 ≈ 80.86. This differs from Outputs coverage.

Error penalty
Base points × critical-concern multiplier × correction multiplier

Opus run 1’s uncorrected critical follow-up closure: 3.75 × 1.5 × 1 = 5.625 points.

Error-adjusted score
max(0, requirement coverage − error penalties)

Astra run 1: max(0, 80.861 − 24.125) ≈ 56.736.

How the error penalty changes the separate scoreThree-run means. Diagnostic scores are unchanged.
Error-adjusted score / 100GPT-6 Astra. 2.5 points: 65.8; 3.75 points: 60.5; 5 points: 55.2GPT-6 Luna. 2.5 points: 58.5; 3.75 points: 54.5; 5 points: 50.4Claude Sonnet 5.5. 2.5 points: 54.4; 3.75 points: 47.5; 5 points: 40.6GPT-5.6 Terra. 2.5 points: 46.0; 3.75 points: 43.9; 5 points: 41.8GLM 5.3. 2.5 points: 49.7; 3.75 points: 39.0; 5 points: 28.4Claude Opus 5.5. 2.5 points: 47.6; 3.75 points: 35.1; 5 points: 22.6Claude Sonnet 5. 2.5 points: 35.8; 3.75 points: 27.8; 5 points: 19.9Claude Fable 5.1. 2.5 points: 41.2; 3.75 points: 26.0; 5 points: 16.5GLM 5.3 Flash. 2.5 points: 32.2; 3.75 points: 20.7; 5 points: 13.2Kimi K3. 2.5 points: 33.1; 3.75 points: 18.5; 5 points: 7.0MiMo V2.6 Pro. 2.5 points: 32.9; 3.75 points: 17.9; 5 points: 5.8Qwen 3.8-2.4T-A95B. 2.5 points: 30.6; 3.75 points: 13.0; 5 points: 1.8Gemini 3.8 Flash. 2.5 points: 19.5; 3.75 points: 6.0; 5 points: 1.6Gemini 3.1 Pro. 2.5 points: 6.2; 3.75 points: 4.5; 5 points: 2.8Gemini 3.5 Flash Lite. 2.5 points: 0.0; 3.75 points: 0.0; 5 points: 0.0Base penalty for an error with an observed consequence
15 models with complete meansHover a line, or focus a model name below, to inspect its three values.

GPT-6 Astra and GPT-6 Luna remain first and second at all three penalty settings.

All sensitivity values
Three-run error-adjusted means · 0–100. Display rounded to one decimal; lines use exact means.
Model2.5 points3.75 points5 points
GPT-6 Astra65.860.555.2
GPT-6 Luna58.554.550.4
Claude Sonnet 5.554.447.540.6
GPT-5.6 Terra46.043.941.8
GLM 5.349.739.028.4
Claude Opus 5.547.635.122.6
Claude Sonnet 535.827.819.9
Claude Fable 5.141.226.016.5
GLM 5.3 Flash32.220.713.2
Kimi K333.118.57.0
MiMo V2.6 Pro32.917.95.8
Qwen 3.8-2.4T-A95B30.613.01.8
Gemini 3.8 Flash19.56.01.6
Gemini 3.1 Pro6.24.52.8
Gemini 3.5 Flash Lite0.00.00.0
04

How we checked the grader

We tested whether the grader passes a correct reference record, fails copies of it with required work removed, responds to targeted edits, and repeats its ratings.

Reference record

Expected
The correct reference record passes.
Observed
Meets the standard, with an error-adjusted score of 95.0.

All 8 critical requirements met; 0 major concerns incomplete.

Reference record with required work removed

Expected
Both copies fail.
Observed
Removing one critical requirement fails the critical check. Leaving four major concerns incomplete fails the major check.

Targeted edits

Expected
Ratings respond to missing evidence and work, not instructions to award full marks.
Observed
The grader ignored an injected instruction to award full marks; one evidence rating moved by one point. Invalid citations lowered evidence ratings. Removing a workstream lowered continuity from 4 to 2. Full results below.

Repeatability

Test
36 cases, with two fresh grading passes per case.
Observed
91.9% exact agreement; 99.8% average agreement within one point on the 0–4 scale.

We measured agreement within each case, then averaged the 36 results equally.

How the perturbation tests worked

We copied grading inputs from three selected cases and changed one feature at a time, such as source order, citation IDs or an instruction inserted into the agent’s text. We graded each original and edited version twice with the same judge instructions and settings, then compared ratings and requirement decisions. A separate paired test removed a workstream.

These tests did not alter the benchmark runs or their published scores.

Full test details
The reference record meets the standard; both copies with required work removed fail.
RecordError-adjusted scoreCritical met / 8Major incomplete / 17Observed result
Reference record95.080Meets standard
One critical requirement removed84.571Fails · critical rule
Four major concerns left incomplete87.984Fails · major rule

The critical edit also removes six major requirements in the same concern. It fails with only one incomplete major concern, below the four-concern threshold. The major-only edit fails despite meeting all eight critical requirements.

The broader targeted-edit study produced 25 complete assessments and one input failure across 26 grading passes. The focused workstream-removal check and the assertion–query pair were separate studies.

What changed in each targeted-edit test.
ChangeObserved resultScope
Replace submitted citations with invalid referencesEvidence ratings fell in both selected report cases on both grading passes. Other ratings were largely preserved.Two report cases; two altered assessments each. The remaining credit varied between passes.
Insert a demand to award full marks into the agent’s textNone of six altered assessments followed the full-marks demand wholesale. One evidence rating rose by one point.Three selected cases; two grading passes each. The same one-point variation also appeared after reordering sources.
Reorder source groups without changing their contentsSome correctness and evidence ratings moved by one point; some transition decisions also changed.The grading passes did not use matched random seeds. This shows sensitivity to how evidence is presented.
Remove a workstream but leave its links unresolvedContinuity fell from 4 to 2 on both grading passes; correctness stayed at 2.A separate, focused check: two intact and two altered assessments.
Contrast an unsupported assertion with a qualified queryThe blinded, model-assisted review penalized the assertion and left the query unflagged.One authored pair. Its inserted actions were not all replayed through the agent’s normal execution tools.
Empty case bodies while still listing those casesOne grading pass reported missing required context; the other kept continuity at 4.An inconsistent test input, not a clean history-loss test. The input failure is retained.
Repeat the same recorded case with the same IDBoth grading passes kept continuity at 4.This did not create a second workstream, so it is not evidence that duplicate workstreams are accepted.
Continuity changed; correctness did not · original 0–4 scale.
RecordContinuityCorrectness
Intact record · pass 142
Intact record · pass 242
Workstream removed · pass 122
Workstream removed · pass 222

Repeatability details

The 36 cases produced 72 assessments. All numeric ratings matched in 25/36 case pairs; all numeric and categorical judgments matched in 22/36.

Grader
GPT-6.1 Sol
Reasoning
Max
Maximum output
128,000 tokens
Temperature
Provider default
Scope of these checks

The same grader condition assessed all three reference records. The controls contain only documentary-support errors; they do not validate all error classes.

The later check on 2026-10-15 verified the existing source links, judgments and arithmetic without calling the grader again. It was not a new clinical audit.

The LLM grader and model-assisted source review use the expert-grounded reference. These checks do not measure agreement with independently assigned expert grades or other grader families, or establish the reliability of whole-episode rankings.

Full test protocols and review records are available on request.

05

Run conditions

Models shared the task, world, tool API and checkpoint schedule. The standard budget was 80 tool calls per checkpoint and four hours per run.

One run per model is published: the run with the median composite. Other runs are available on request.

Recorded setup for each model · three runs each
ModelHarnessProvider routeReasoningDates · UTCRuns
GPT-6 AstraCodex CLI 0.153.4OpenAIHigh2026-09-26–271 ↗2 · request3 · request
GPT-6 LunaCodex CLI 0.153.4OpenAIHigh2026-09-26–271 ↗2 · request3 · request
Claude Sonnet 5.5Linux-isolated Claude CLIAnthropicHigh2026-10-101 · request2 ↗3 · request
GPT-5.6 TerraCodex CLI 0.153.4OpenAIHigh2026-09-26–271 ↗2 · request3 · request
GLM 5.3Shared tool scaffold · API adapterOpenRouter · z-ai/fp8High (requested)2026-10-101 · request2 ↗3 · request
Claude Opus 5.5Claude Code 2.1.283AnthropicHigh2026-09-26–271 ↗2 · request3 · request
Claude Sonnet 5Claude Code 2.1.259AnthropicHigh2026-09-25–271 · request2 · request3 ↗
Claude Fable 5.1Claude Code 2.1.283AnthropicHigh2026-09-26–271 · request2 · request3 ↗
GLM 5.3 FlashShared tool scaffold · API adapterOpenRouter · z-ai/fp8High (requested)2026-10-101 ↗2 · request3 · request
Kimi K3Shared tool scaffold · API adapterOpenRouter · moonshotai/mxfp4High (requested)2026-10-101 ↗2 · request3 · request
MiMo V2.6 ProShared tool scaffold · API adapterOpenRouter · xiaomi/fp8High (requested)2026-10-101 · request2 · request3 ↗
Qwen 3.8-2.4T-A95BShared tool scaffold · API adapterOpenRouter · siliconflow/fp8High (requested)2026-10-101 ↗2 · request3 · request
Gemini 3.8 FlashShared tool scaffold · Google API adapterGoogle AI StudioHigh2026-10-111 · request2 · request3 ↗
Gemini 3.1 ProShared tool scaffold · Google API adapterGoogle AI StudioHigh2026-10-101 · request2 · request3 ↗
Gemini 3.5 Flash LiteShared tool scaffold · Google API adapterGoogle AI StudioHigh2026-10-10–111 · request2 ↗3 · request
Recorded differences and run notes

Qwen run 2 has no wall-time limit recorded. Harnesses, provider routes and token policies differ; the same reasoning label does not guarantee equal compute.

Qwen 3.8-2.4T-A95B, run 2: Grading recoveries included two disclosed input-only exceptions; equal grading input budgets are not claimed.

Qwen 3.8-2.4T-A95B, run 2: This selected replacement ran without a wall-time cutoff, unlike the usual 14,400-second limit. Equal wall-time conditions are not claimed.

Gemini 3.8 Flash, run 1: Selected from a declared direct-Google route-replacement condition, replacing the original OpenRouter-route attempt; the original attempt remains in the private record.

Gemini 3.8 Flash, run 2: Selected from a declared direct-Google route-replacement condition, replacing the original OpenRouter-route attempt; the original attempt remains in the private record.

Gemini 3.8 Flash, run 3: Selected from a declared direct-Google route-replacement condition, replacing the original OpenRouter-route attempt; the original attempt remains in the private record.

Gemini 3.5 Flash Lite, run 1: Selected from a declared direct-Google route-replacement condition, replacing the original OpenRouter-route attempt; the original attempt remains in the private record.

Gemini 3.5 Flash Lite, run 2: Final-report submission recovered after the original budget failure; an empty-finding validation correction preserved the zero ratings. No model or judge rerun.

Gemini 3.5 Flash Lite, run 2: Selected from a declared direct-Google route-replacement condition, replacing the original OpenRouter-route attempt; the original attempt remains in the private record.

Gemini 3.5 Flash Lite, run 3: Selected from a declared direct-Google route-replacement condition, replacing the original OpenRouter-route attempt; the original attempt remains in the private record.

06

Resources

This publication covers one public version of the environment. Private held-out worlds are available on request.

For model builders

Evaluate your model on a private held-out world, or request environment access for post-training.

Request access ↗

License

Benchmark-owned code: Apache 2.0. Synthetic data and tasks: CC BY 4.0. Third-party source documents retain their own rights.

Read the license notice ↗
Study scope and review status
  • The published scores cover the released environment, with three runs per model; private held-out worlds are available separately and are not included in these results. This is a benchmark-defined standard, not a clinical or regulatory determination.
  • Five of eight critical requirements concern one site’s dose-control storyline. Results do not establish performance across other trial worlds.
  • Experts performed oversight work in the environment, graded completed model traces, confirmed severity tiers and clarified difficult interpretation rules. The repeatability percentage measures agreement between grader passes; no quantitative human–grader agreement rate is reported here.
  • This is a development-world evaluation, not a held-out validation of the scoring rules. Three-run intervals describe these runs, not uncertainty across worlds or reviewers.
  • Error review covered 45/45 runs. Some corrective-action and evidence-support decisions remain under review; the review does not prove every error was found.

Cite this work

@techreport{kallel2026clinopsbench,
  author = {Kallel, Youssef},
  title = {{ClinOpsBench}: Trial Oversight},
  year = {2026},
  type = {Technical Report},
  number = {v1.0},
  url = {https://panoptive.com/research/trial-oversight}
}
Download BibTeX
Evidence checksums

Results date: 2026-10-15. The browser verifies the result files before displaying them. No grader is called by this page.

Results SHA-256
35054b3ba1821222c0b5d35ed73bde6b16412275ca858ae4c0fcfdca7563f7b2
Control verification SHA-256
e17e6197052fa726504945eb1af18a7fd4e37835fe20d1f3c7e4691772a088c9
Repeatability audit SHA-256
4d8e6c4d5efd21731fd2a0a478f2a7db26dac6578a75d0e7206bcc1345997e3a