---
topic: ai-engineering
author: Crashtech Editorial
published: Jul 29, 2026 · read: 5 min
---

Paying for the Same Work Twice: Where an Agent Loop Wastes Money and How Each Layer Stops It

Tracing an agent execution through harness, API, and inference layers: prompt caching, WebSockets, BM25 tool search, and KV routing.

– –

When benchmarking AI coding agents, engineering discussions frequently fixate on benchmark accuracy: SWE-bench scores, pass@1 rates, and reasoning depth. But in real-world deployment, the primary operational constraint is cost per successful task.

An agent that solves a complex codebase refactor by making 45 sequential LLM calls can easily burn $8.00 in API tokens and take six minutes to complete. A competitor achieving the same outcome for $0.90 in 40 seconds wins the market.

An AI agent is not a solitary language model. It is an end-to-end distributed system spanning three distinct layers:

  1. The Client Harness: Manages conversation history, constructs context, and executes tool calls in a secure sandbox.
  2. The API Gateway Layer: Handles authentication, rate limiting, tokenization, and streaming.
  3. The GPU Inference Fleet: Serves the model weights across thousands of accelerators.

In a naive implementation, an agent loop repeats identical work across all three layers on every single iteration. Eliminating this redundant compute requires coordinated optimizations at each tier.

+--------------------------------------------------------------------------+
|                  THE THREE-TIER AGENT OPTIMIZATION STACK                |
|                                                                          |
|  [Harness Layer]  : WebSockets + Stable Prefixes + BM25 Deferred Tools   |
|         |                                                                |
|  [API Gateway]    : Delta Tokenization + Async Parallel Moderation       |
|         |                                                                |
|  [Inference Fleet]: Cache-Aware GPU Routing + Speculative Decoding + MLA  |
+--------------------------------------------------------------------------+

Layer 1: Optimizing the Harness

The harness is the software running on the client or worker container that hosts the loop. Most agent loops leak tokens and latency before a single byte hits the network.

1. Persistent WebSockets and Incremental Requests

Standard HTTP REST endpoints require a fresh TLS handshake, TCP slow-start negotiation, and re-authentication on every turn. In an agent workflow with 30 tool iterations, connection overhead alone accumulates seconds of idle waiting.

  • Solution: Maintain a persistent bidirectional WebSocket connection.
  • Incremental Deltas: When a tool completes, do not re-send the full 25,000-token conversation history over the wire. Send only the new tool output payload with a pointer to the existing server-side session handle (response_id: resp_88391).

2. Prefix Stability for Prompt Caching

Modern LLM inference engines support Prompt Caching, allowing pre-computed KV states for shared prefixes to be reused. However, prompt caching is brittle: if a single character changes at token position 50, the entire cache from token 51 onward is invalidated.

  • The Bug: Many frameworks dynamically sort tool definitions alphabetically or insert a changing timestamp at the top of the system prompt. This guarantees a 0% cache hit rate.
  • The Fix: Treat the prompt prefix as an append-only log. System prompts, static instructions, and tool definitions are placed at the very beginning of the prompt and remain byte-for-byte identical across every iteration.

When an enterprise agent supports 200 possible tools (GitHub, Jira, AWS, Slack, Linear), injecting all 200 JSON schemas into every prompt consumes 15,000 tokens of overhead before the user even types a question.

  • Solution: Omit tool definitions from the initial prompt. Provide the agent with a single meta-tool: search_tools(query: string).
  • When the agent realizes it needs to manipulate a Git branch, it queries the local tool catalog using lightweight BM25 lexical search and loads only the five relevant Git tool schemas into context for subsequent turns.

Layer 2: Optimizing the API Gateway

The API layer sits between the public internet and the GPU clusters.

Delta Tokenization

Normally, the API gateway receives a JSON payload containing historical messages and re-tokenizes the entire prompt using Byte-Pair Encoding (BPE). In a 60,000-token context, CPU tokenization alone can take 80 to 120 milliseconds.

  • By retaining the pre-tokenized sequence in an in-memory cache at the gateway, the gateway tokenizes only the new delta (the latest tool result), reducing tokenization latency to under 2 milliseconds.

Parallel Moderation Classifiers

Enterprise compliance mandates that inputs and outputs pass through safety and policy classifiers.

  • In naive pipelines, the request blocks for 150 ms while the safety model evaluates the prompt before the main LLM receives the input.
  • Optimized gateways launch the safety classifier in parallel with the main LLM’s prefill phase. Because the main LLM takes 200 ms to begin streaming its first token, the safety check resolves completely inside the prefill window, adding zero observable latency to Time-to-First-Token (TTFT).

Layer 3: Optimizing the Inference Fleet

The GPU cluster is where the financial cost is settled.

                                  [API Request with Session ID]
                                                |
                                                v
                                    +-----------------------+
                                    | Cache-Aware Load Bal. |
                                    +-----------------------+
                                                |
                        +-----------------------+-----------------------+
                        | (Hit: KV Cache warm)                          | (Miss: Cold GPU)
                        v                                               v
             +--------------------+                          +--------------------+
             | GPU Worker Node 4  |                          | GPU Worker Node 12 |
             | (Skips Prefill!)   |                          | (Full Recalculation)|
             +--------------------+                          +--------------------+

Cache-Aware Routing

In traditional stateless load balancing, round-robin or least-connections routing sends consecutive agent requests to arbitrary GPU servers. If Node A computed the prompt prefill for Turn 1, sending Turn 2 to Node B forces Node B to recompute all 40,000 tokens from scratch.

  • Cache-Aware Schedulers hash the session ID and prefix fingerprint to route consecutive requests back to the specific GPU worker holding that conversation’s KV cache blocks in memory.
  • This transforms an expensive $O(N)$ prefill calculation into an instantaneous cache hit, cutting token processing costs by 70% to 90%.

Speculative Decoding

LLM token generation is memory-bandwidth bound: the GPU must transfer hundreds of gigabytes of weights from High-Bandwidth Memory (HBM) to cache to generate a single token.

  • Speculative Decoding pairs a small, fast “draft model” (e.g., 1B parameters) with the primary large model (e.g., 70B parameters).
  • The draft model rapidly generates a candidate sequence of 5 tokens. The large model then verifies all 5 tokens simultaneously in a single forward pass.
  • Because the verification step runs parallel matrix multiplications rather than autoregressive decoding, throughput increases by 2x to 3x with zero loss in mathematical or semantic precision.

The Systems Takeaway

Single-technique optimizations rarely produce a 10x breakthrough in isolation. Real production efficiency emerges from end-to-end alignment: stable prompts in the harness, delta processing in the gateway, and cache-affinity scheduling on the GPU fleet.

Advertisement

/* Comments */