METR’s Anthropic Audit Finds Faster AI Progress—and a Visibility Gap

A claim that AI is speeding up AI research can sound like a dramatic turning point. METR’s new assessment is more specific: after eleven weeks of access to selected internal data and interviews, the nonprofit estimates Anthropic’s capabilities on well-specified tasks may be improving about twice as fast as a historical comparison rate. It also documents why the public cannot independently reproduce much of the analysis: Anthropic had unilateral rights to redact confidential material. ^[1]

What the “twice as fast” estimate means

METR says the benchmark component of its assessment found Anthropic’s internal model scores, measured through an index called AECI, grew at an estimated 1.9 times the pre-October 2025 rate. The report gives a 95% confidence interval from 1.5 to 2.8 and says that, under its interpretation, the company appears to have made roughly as much benchmark progress in the last year as in the previous two. This describes a change in the measured rate of capability progress, not a claim that each employee became twice as productive or that every real-world task improved by the same amount. ^[1]

METR tested alternative choices for benchmarks, model snapshots and statistical link functions. Those analyses produced a broad range of estimates; its report says the AECI result could reflect benchmark-fit assumptions, limited data or overfitting, and that more recent research progress may lag in the benchmark data. The estimated acceleration also changed when METR included non-Anthropic models. AECI is one lens on capability, not a direct measure of social, economic or safety outcomes. ^[1]

The evaluators considered compute scaling, hiring, data, algorithmic changes and other possible explanations. They say they could rule out an acceleration in the rate of training-compute growth, but could not establish a single causal explanation. METR’s subjective judgment is that a majority of the current acceleration may be attributable to AI assistance. It explicitly attaches substantial uncertainty to that view. Interviews and staff surveys provide context, but self-reported speedups are not themselves evidence that capability progress will compound at the same rate. ^[1]

Outside review with defined limits

METR describes an eleven-week engagement involving about 2.5 full-time-equivalent staff, internal data and roughly 15 interviews. It says Anthropic supplied more data than expected, and that METR did not receive payment. At the same time, Anthropic could reject requests, had unilateral redaction rights over non-public information, and the evaluators did not query unreleased models directly. METR says the public version omits key evidence needed for readers to independently assess some conclusions. ^{[1] [2]}

Those limitations do not erase the work; they define how much weight to place on it. METR retained editorial control over the report and says it did not accept compensation, but the company still controlled what confidential evidence could appear in public. The outside assessment therefore gives readers a more informed view than a company-only account, while leaving a meaningful verification gap. The report also notes that METR has other evaluation relationships with Anthropic and received free token credits for this work. ^[1]

The distinction between assistance and autonomy is equally important. METR says researchers described AI as speeding parts of research and development, but that models were not yet making highly significant, fully autonomous contributions to the most important parts of model training. It reports “signs of life” that could change, while emphasizing uncertainty. The title of a viral summary should not collapse this into “AI now builds itself.” ^[1]

The question is related to how researchers measure scientific AI use, but the evidence differs: one report classifies product sessions, while this assessment analyzes internal AI R&D progress at a single company. METR’s prior research also includes findings that complicate any simple productivity story; measurement choices and task conditions matter.

What would make the next report stronger

Readers should look for transparent methods, a clear description of what data was excluded, independent access to the underlying evaluation where possible, and repeated measurements that can distinguish a short-term jump from a sustained trend. AECI estimates should be checked against real work outcomes and other capability measures. The causal story needs evidence that separates AI assistance from staffing, algorithms, data, and compute—not just a trend line alongside reports of heavy AI use.

For policymakers and AI developers, the report is a signal to improve the quality of external evaluation and public disclosure. It is not a forecast with a fixed date, a demonstration of recursive self-improvement or a proof of imminent loss of control. METR describes a possible acceleration in one lab’s measured capabilities, reports uncertainty about what caused it, and shows how redaction makes the evidence hard for the public to test. Holding all three facts together is the most responsible way to read the result.