TiGanEval Tests Whether AI Agents Deliver the Outcome, Not Just Follow the Prompt

A coding agent can follow every sentence in a prompt and still deliver the wrong change. The user may describe a symptom, while the actual requirement lives in a failing test, an adjacent component, or an established project convention. An October arXiv preprint called TiGanEval asks whether benchmarks capture that gap between literal instruction following and achieving the intended outcome.

The hidden requirement problem

The authors examine 100 software-engineering chat trajectories and report that 56% of initial instructions omitted requirements that later mattered to the task. They also report that in 52% of sampled cases, useful follow-up requirements could be inferred from project context. These figures describe the paper’s sample and coding workflow; they are not prevalence estimates for every agent interaction. ^[1]

This is a familiar software problem. “Fix the broken settings page” can mean preserve existing route structure, honor a design system, avoid changing account semantics, and add a regression check. A benchmark that gives all of those constraints up front tests a different skill from one where the agent must inspect the repository and discover them.

What TiGanEval changes

TiGanEval separates task conditions that vary how much of the requirement is explicit versus recoverable from project or runtime clues. The preprint reports a 53–67 percentage-point performance gap across those conditions for eight evaluated models. The authors further report that repeated feedback about consequences did not close the gap in their setup. ^[1]

The benchmark’s framing is valuable because it makes context use measurable. It asks evaluators to look at code and test outcomes alongside the prompt, rather than treating the prompt as a complete specification by default. The underlying SWE-chat dataset has since grown: its current paper page describes almost 18,000 sessions, more than 229,000 user prompts and two million tool calls. TiGanEval’s 100-trajectory sample is much narrower than that living corpus. Another 2026 preprint, SWE-RPG, independently treats clarification and implicit-requirement recovery as measurable steps; across 163 tasks it identifies requirement recovery as a major bottleneck in its own evaluated runs. These studies use different tasks and methods, so their percentages should not be combined. ^{[2] [3]}

What the result does not establish

The benchmark is a preprint and the headline gap depends on task construction, scoring rules and model selection. One hundred trajectories may reveal a meaningful pattern, but cannot establish how often requirements are implicit across all software teams or product types. The measured gap also should not be read as a product ranking without the full protocol, model versions, prompts, tooling and reproducibility details.

“Outcome alignment” here refers to task completion in a coding context. It does not settle broader questions about alignment, safety or whether an agent should infer unstated user intent in sensitive situations. In some tasks, asking before acting is the correct behavior; in others, examining local code is ordinary engineering diligence. A useful benchmark needs to distinguish productive inference from risky assumptions.

How teams can use the idea now

Engineering teams can borrow the evaluation principle without treating the paper’s scores as universal. Build a set of real tasks where the expected outcome is documented separately from the initial request. Give the agent repository access, then compare its proposed changes against tests, project conventions and the actual acceptance criteria. Track when it asks a good clarification, when it misses evidence, and when it invents requirements.

A practical scorecard should include correctness, scope control, tests, unintended changes, and whether the agent explained uncertainty. It should also test cases where the repository evidence conflicts with an ambiguous prompt, because silently guessing can be worse than a low task score. For related work on practical coding-agent evaluation, see our articles on live serving tests and context failure in autonomous coding agents.

TiGanEval’s strongest contribution is the question it puts on the table: did the agent satisfy the user’s objective in context, or merely produce a plausible response to a string of instructions? The answer deserves careful experiments, open methods and independent replication before the benchmark’s initial results become a broader claim about what coding agents can or cannot do.