What does the paper report?
The authors sampled 100 SWE-chat trajectories and report that many initial instructions omitted requirements later needed for success. Across eight evaluated models, they report a 53 to 67 percentage-point performance gap between benchmark conditions. These are results from the authors' proposed setup, not a general industry-wide failure rate.
Answered in
TiGanEval Tests Whether AI Agents Deliver the Outcome, Not Just Follow the PromptA preprint uses SWE-chat traces to study missing requirements and agent outcomes. TiGanEval is an early diagnostic, not a universal failure rate.
Read the full analysisOther questions this article answers
More development best practices questions
- What did OpenAI’s Habitat team change?
- Does the reported 12× read improvement mean another team should copy the design?
- Does the Python Typing Survey measure all Python developers?
- Do Python type hints enforce types at runtime?
- What is interactive refactoring in gopls?
- Is interactive refactoring already part of the official LSP standard?
- What does Meta mean by human-on-the-loop software engineering?
- Does Meta's Forum project prove that AI can replace a software team?
Every answer on Crashtech is written by the editor of the article it comes from — never auto-summarised. Browse all answers or the Development Best Practices beat.