AI Labs Should Show Their Work Before Asking Us to Trust Their Pace

A benchmark can tell us how a model performed on a test. It cannot, by itself, tell us how much of the next model was designed with AI assistance, where a human intervened, or how the lab knows its own account is accurate.

Anthropic has proposed measurements for AI-led AI research and development, oversight or intervention, and compute allocation. The company describes an internal snapshot and plans for independent evaluators. These are proposed measures and plans, not a completed audit or a reporting standard adopted across the industry. Source

My argument is that labs should report process measures on a recurring schedule. They need not reveal sensitive code, security details or individual employee work. They should disclose enough about definitions, data collection and verification that outside readers can understand what the numbers mean and compare them over time.

Capability is an outcome; process explains the path

An evaluation score is a snapshot of performance on chosen tasks. That remains valuable, but it gives a narrow view. If an organization is increasingly using models to help build future systems, process information could help answer a different question: how is that work changing, and what checks remain in the loop?

A dashboard should separate measures that answer separate questions rather than compressing them into one headline number.

Reporting area	Useful question	Example of a careful disclosure
AI-assisted research	Where does model assistance occur in the development process?	Defined task categories and the method used to estimate assistance
Oversight and intervention	When do people review, modify or stop model contributions?	Intervention categories and sampling procedure
Compute allocation	How are resources distributed across relevant research activities?	Scope, time period and allocation method

This is a proposed structure, not a claim that any lab already measures these categories in a fully comparable way. Definitions matter. “AI-assisted” could mean a researcher asked for a code suggestion, or a model generated and evaluated a large part of an experiment. A useful report explains which activities are counted.

Verification is part of the disclosure

A number becomes more useful when readers know how it was produced. Was it estimated from sampled work records, instrumented systems or staff reporting? What activities were excluded? How were missing cases handled? Without those details, two organizations could publish the same label while measuring different things.

Independent evaluators can help test whether a reported measure follows its stated method. They do not need unrestricted access to every sensitive artifact. A process can offer protected review, redacted evidence or aggregate findings. But the lab should identify the evaluator’s role, the evidence examined, and any limitations that constrained the work.

The public should also see trend lines rather than isolated announcements. A single quarter may be unusual. Repeated measures allow readers to distinguish changes in practice from changes in definitions. If the method changes, the report should say so and explain whether earlier periods can be recalculated.

Imagine a hypothetical lab reporting that more experiments involve model assistance this quarter. A responsible report would explain whether the category changed, whether the sample covers research teams consistently, and whether “involvement” means drafting code or selecting the research direction. Those distinctions determine what the trend can support.

Confidentiality deserves a serious hearing

The strongest objection is that detailed reporting could expose valuable intellectual property, reveal security-sensitive practices or create pressure to optimize for a metric. That concern is real. A public dashboard should not expose unreleased model plans, exploit-relevant information or employee-level productivity data.

But confidentiality does not require silence about every process. A lab can publish aggregate categories, thresholds and methodology while withholding sensitive implementation details. Independent reviewers can examine more granular evidence under appropriate safeguards and report whether the public figures are supported. The goal is to reduce uncheckable claims, not to publish a blueprint for competitors.

Metrics can also distort behavior. If a lab is rewarded for minimizing human intervention, staff may be discouraged from recording it. If it reports only compute efficiency, teams may shift work into categories that are not counted. That is why measures should be plural, accompanied by qualitative context, and periodically reviewed by outside experts.

A practical standard for trust

Labs should begin with a small set of stable measures and publish definitions before presenting results. Each report should state the collection period, coverage, exclusions, uncertainty and verification method. Independent reviewers should be able to challenge both the calculations and the assumptions behind them.

Readers should not treat such a dashboard as a safety guarantee. A lab can follow a process and still make poor decisions. Process visibility instead makes governance claims easier to scrutinize and improvement easier to track. It gives regulators, researchers and the public a shared basis for asking better questions.

Independent access takes different forms. The double-blind evaluation pilot separates the lab’s assets from the evaluator’s test. The agent-authority essay develops why observing actions and preventing unauthorized actions need distinct product controls.

Trust should not rest on a company’s confident description of its own pace. Nor should every detail be forced into public view. The workable middle is disciplined disclosure: enough recurring evidence to show how development is changing, protected review where confidentiality matters, and clear limits on what the measures prove. If labs want society to take their accounts of progress seriously, they should make those accounts testable.