What should an evaluation team preserve after an agent behaves unexpectedly?
Keep enough evidence to reconstruct the event: timestamps, model and harness versions, tool calls, rejected actions, environment state, network and sandbox logs, monitor decisions and relevant checkpoints. Then verify containment and restore from a known-good baseline before resuming.
Answered in
NIST’s New AI Incident Playbook Starts After an Agent Crosses a BoundaryNIST, NSA and CISA offer ten voluntary practices for control incidents in frontier-model training and evaluation, from preparation through reporting.
Read the full analysisOther questions this article answers
More development best practices questions
- What did OpenAI’s Habitat team change?
- Does the reported 12× read improvement mean another team should copy the design?
- Does the Python Typing Survey measure all Python developers?
- Do Python type hints enforce types at runtime?
- What is interactive refactoring in gopls?
- Is interactive refactoring already part of the official LSP standard?
- What does Meta mean by human-on-the-loop software engineering?
- Does Meta's Forum project prove that AI can replace a software team?
Every answer on Crashtech is written by the editor of the article it comes from — never auto-summarised. Browse all answers or the Development Best Practices beat.