Inside OpenAI Codex: Agent Loops, Prefix Prompt Caching, Context Compaction, and App Server Protocol
The systems architecture behind OpenAI Codex: agent execution loops, prefix prompt caching, context compaction, and the Model Context Protocol harness.
On this page
When OpenAI released Codex, the specialized reasoning and code-generation agent, public discussion focused almost exclusively on the underlying frontier model (a fine-tuned variant of reasoning architectures like o3).
Yet, in a production coding agent, the neural model is merely the computational engine.
A high-performance coding agent capable of autonomously diagnosing bugs, navigating multi-million-line repositories, and executing terminal commands across dozens of iterations is fundamentally a distributed systems engineering challenge.
The true competitive moat of Codex lives in the orchestration layer around the model: how turns are managed, how prefix prompt caching eliminates catastrophic latency, how context compaction preserves attention efficiency, and how the App Server Protocol decouples execution from client IDEs.
+-------------------------------------------------------------------------------+
| OPENAI CODEX SYSTEM ARCHITECTURE |
+-------------------------------------------------------------------------------+
[Client Interface: Web, Desktop, IDE Extension, Terminal CLI]
|
| App Server Protocol (Bidirectional JSON-RPC over WebSockets)
v
[Codex Agent Orchestrator Gateway]
|
+---> [Ephemeral Cloud Sandbox (Firecracker MicroVM / COW RootFS)]
| |-- Cloned Repository Workspace
| |-- Sandboxed Bash Shell / Tool Execution Harness
| \-- Local LSP & File Indexer
|
+---> [Prefix-Cached Context Manager]
| |-- Static Project System Prompt (Cached in GPU Memory)
| |-- Dynamic Compacted Tool History
| \-- High-Priority Context Window Anchor
|
v
[Frontier Reasoning Model (Codex Core)] ---> Emits Structured Tool Calls
1. The Core Agent Turn Loop: The Model-Harness Split
At the heart of Codex is a strict separation of concerns between the Model (the brain) and the Harness (the body):
+-------------------------------------------------------------------------------+
| THE CODEX MULTI-TURN EXECUTION LOOP |
+-------------------------------------------------------------------------------+
[Turn 1 Begins: User Task "Fix failing auth test"]
|
v
1. Harness builds prompt: System Directives + Repo Map + User Task.
|
v
2. Model generates thought + Tool Call: `bash(cmd="pytest tests/test_auth.py")`
|
v
3. Harness executes command inside Sandboxed MicroVM.
Captures stdout/stderr: "AssertionError: expected 200, got 401"
|
v
4. Harness appends tool output to context window.
|
v
5. Model reads error -> issues Tool Call: `edit_file(path="auth.py", ...)`
|
v
Loop repeats until Model emits final text completion: "Tests verified and passing!"
The model never touches a real operating system directly. It emits structured intents; the Harness validates boundaries, executes actions within isolated sandboxes, and formats observations back into tokens.
2. Compounding Latency and Prefix Prompt Caching
In a multi-turn agent interaction spanning 25 turns, context grows monotonically. By turn 20, the prompt contains:
- 10,000 tokens of project documentation.
- 5,000 tokens of codebase structure.
- 15,000 tokens of previous terminal logs, file diffs, and reasoning steps.
If the model had to re-process all 30,000 tokens from scratch on every turn, time-to-first-token (TTFT) would exceed 10 seconds per turn, and compute costs would scale quadratically ($O(N^2)$).
Codex overcomes this using Prefix KV-Cache Caching:
+-------------------------------------------------------------------------------+
| PREFIX KV-CACHE REUSE TOPOLOGY |
+-------------------------------------------------------------------------------+
Turn 1 Context:
[System Prompt] [Project Index] [User Goal] [Turn 1 Tools]
|================ CACHED IN GPU VRAM ===================|
Turn 2 Context:
[System Prompt] [Project Index] [User Goal] [Turn 1 Tools] [Turn 2 Tools]
|======== REUSED FROM VRAM (0ms Compute!) ==============|-- Computes ONLY this!
The Invariant of Prefix Caching
Prefix caching works only if the byte sequence at the beginning of the prompt remains 100% bit-for-bit identical.
If an orchestrator dynamically injects the current timestamp (“Time: 14:02:11”) or randomizes tool schemas at the top of the prompt, the GPU cache key is invalidated, forcing an expensive full-context recomputation.
Codex anchors all static directives at the very top of the prompt, appending dynamic tool outputs strictly to the tail.
3. Context Compaction: Pruning the Attention Window
As the agent executes commands (git status, cargo test, ripgrep), terminal outputs flood the context window with thousands of characters of verbose noise.
Leaving raw compiler outputs in the prompt triggers Attention Degradation: the model loses track of the original goal amidst thousands of lines of irrelevant stack traces.
Codex periodically runs Context Compaction:
Before Compaction (Turn 12 - 45,000 Tokens):
[Terminal Output: 4,000 lines of standard library build logs...]
[Compiler Warnings: 500 lines of unused variable notices...]
[Actual Error: Line 492: Undefined method 'verify_jwt']
After Compaction (Turn 12 - 8,000 Tokens):
[Compacted Summary: "Build completed with warnings. Target error identified:
Line 492: Undefined method 'verify_jwt' in auth_controller.rb"]
Old tool outputs are collapsed into concise semantic summaries, pruning 80% of transient tokens while preserving critical state for downstream reasoning.
4. The App Server Protocol: Decoupling Agents from UIs
Early AI tools were tightly coupled to specific IDE extensions (e.g., an extension hardcoded to VS Code APIs).
Codex communicates across a standardized App Server Protocol:
[Web Browser UI] [macOS Desktop App] [VS Code Extension]
| | |
+-----------------------+-------------------------+
|
v (Universal JSON-RPC 2.0 over WebSocket / stdio)
[Codex App Server Engine]
Protocol Capabilities
- Bi-Directional Streaming: Streams reasoning thoughts, file diffs, and terminal execution chunks in real time.
- Client Capability Negotiation: The client advertises its capabilities (e.g., “I support interactive inline diffs” vs “I am a pure headless CLI”), allowing the agent to tailor its output format dynamically.
- Human Approval Handshakes: When the agent attempts a destructive action (
git reset --hard), the App Server transmits an approval request to the client and blocks until the human taps confirm.
Architectural Comparison Matrix
| System Component | Naive LLM Wrapper | OpenAI Codex Orchestrator |
|---|---|---|
| Execution Sandbox | Local developer machine terminal | Ephemeral Firecracker MicroVMs with COW overlays |
| KV-Cache Strategy | Recomputed on every turn | Strict prefix caching (Zero recomputation on static tokens) |
| Context Management | Unbounded token accumulation | Automated context compaction & tool output summarization |
| Client Decoupling | Hardcoded IDE extension logic | Universal App Server Protocol (JSON-RPC over WebSockets) |
| Tool Execution | Direct bash execution | Sandboxed Harness with permission handshakes |
Summary and Key Takeaways
- The model is only the brain: Production coding agents require a resilient Harness to handle sandboxing, tool execution, and context management.
- Anchor static tokens for prefix caching: Keep system prompts and project context bit-for-bit identical at the prompt head to enable 0ms KV-cache reuse.
- Compact context aggressively: Summarize verbose terminal outputs to prevent token bloat and protect attention precision.
- Decouple agents via universal protocols: Use structured JSON-RPC protocols to allow a single agent backend to serve web apps, desktop clients, and terminal CLIs.
- Isolate actions in microVM sandboxes: Never execute unverified agent terminal commands on bare metal; isolate execution inside ephemeral microVMs with copy-on-write snapshots.


/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.