Claude Sonnet 5.5: How to Test an Upgrade Without Fooling Yourself

The most useful AI upgrade is often the one that makes an ordinary task less annoying. Fewer retries, cleaner changes and a shorter review can matter more than an impressive demonstration on a problem your customers never encounter.

What does Anthropic claim about Sonnet 5.5?

Anthropic’s September 28 release reports faster output generation and lower per-task costs than Sonnet 5 despite unchanged standard input and output token rates. It describes Sonnet 5.5 as suited to well-scoped work, while retaining a distinction between it and Opus 5.5 for complex, open-ended judgment. The company also notes that effort settings differ by product surface. Those are claims and configuration details from the vendor, checked October 6—not results of an independent Crashtech test.

That combination should change how an engineering team evaluates upgrades. The comparison is not simply older model versus newer model. It is a specific model, effort setting, tool environment, prompt and completion policy against another complete configuration. Leaving one of those variables unstated makes the result harder to reproduce and easier to overinterpret.

Why does faster generation sometimes fail to save time?

A user experiences the whole task. Time spent identifying relevant files, waiting for a test runner, correcting a wrong interpretation and reviewing a patch all contributes to the wait. If an agent generates text twice as quickly but launches three unnecessary builds, the visible experience can still be slower.

Measure separate intervals: time to a useful first response, time to a reviewable artifact and time to an accepted outcome. A fast acknowledgment is valuable in an interactive tool, but it is not completion. A polished summary is not a substitute for the actual file. Distinguishing these timestamps prevents a dashboard from reporting a speed improvement that users cannot feel.

For coding work, record the number of unsuccessful commands and redundant checks as well. A failed tool call is not automatically a model fault; the environment may be broken. Label the cause where possible. Otherwise infrastructure instability can make a model look worse, or a cached test result can make it look better, without any real capability difference.

What should go into a private coding evaluation?

Use tasks with clear boundaries and authentic repository context. Good examples include repairing a reproduced defect, extending an existing validation rule, implementing an accessibility correction, or updating a data contract across the files that already own it. Each task should have a reviewer-written explanation of success and important behavior that must remain unchanged.

Include at least some cases where the right answer is restraint. An agent should recognize that a missing secret cannot be invented, that a failing integration may be outside its authorized environment, and that a requested UI change does not justify an unrelated schema rewrite. A coding evaluation that rewards only added code misses a large part of engineering judgment.

Separate mechanical verification from expert review. Existing tests can establish important behavior, but reviewers should still inspect unnecessary scope, brittle assumptions and error handling. Conversely, a reviewer liking the code does not establish that the altered flow works. Both forms of evidence belong in the result.

Observation	What it might mean	What to check next
Fewer tokens	Better efficiency or missing work	Acceptance criteria and evidence
Fewer tool calls	Better batching or inadequate inspection	Relevant files and executed checks
Faster final answer	Less latency or premature completion	The actual saved artifact
Higher test pass rate	Better changes or a weak test set	Uncovered failure cases

How should reasoning effort enter the decision?

Treat effort as a tunable resource allocation. Start by comparing configurations under an equivalent quality requirement, then explore the cost and latency needed to achieve it. The highest available setting is not automatically the best product default, and the lowest setting is not automatically the most economical once retries are included.

For an interactive edit, shorter turnaround can make a modest first attempt useful because a human is already present to steer it. For an unattended migration, the same first attempt may be costly if a mistake propagates through many files. Those are different products, even when both use the same model identifier.

Publish the chosen setting in internal experiment records. Also retain any time limit, tool budget and stop condition. Without those details, an apparent regression after a future release can become impossible to diagnose. A team should be able to answer whether the model changed, the harness changed, or the definition of success changed.

How do you avoid selecting for the demonstration?

Keep a held-out set of tasks that the people tuning prompts do not continually inspect. If every failed case immediately becomes a new instruction in the prompt, the team may optimize for its evaluation examples rather than its real traffic. Add fresh tasks over time and retire ones whose answers have become effectively memorized within the workflow.

Report results by task family. An aggregate score can hide a model that excels at simple bug fixes but struggles with ambiguous interfaces or data migrations. That does not necessarily make it unusable. It may make it a strong implementation worker behind a more deliberate planning and review process.

An illustrative decision rule is to adopt the new configuration for a task family only when acceptance stays within your predefined tolerance and either total completion time or total cost improves. The tolerance must come from the product’s consequences, not from a desire to declare the migration successful.

What should happen after the rollout?

Watch reopened tasks, reviewer corrections and escaped defects. These are slower signals than a benchmark, but they reveal whether the improvement survives contact with actual work. Preserve a sample of the complete trace, with appropriate privacy controls, so a reviewer can reconstruct why a result was accepted.

Write a short migration record containing the previous configuration, new configuration, measured benefit and known weak cases. This makes the next upgrade cheaper to assess. It also prevents organizational memory from collapsing into an unsupported claim that one brand is always best.

The companion guide on cost per accepted task provides the financial calculation. The larger lesson from this release is methodological: efficiency is valuable when the work remains complete. A faster model earns trust by reducing the total burden on the people using it.