How Netflix Uses AI Agents for Performance Engineering
How Netflix deploys autonomous AI agents to parse flame graphs, diagnose latency spikes, analyze canaries, and optimize cloud infrastructure.
On this page
Running thousands of microservices handling hundreds of millions of concurrent video streams means performance regressions cannot be investigated through manual flamegraph triage. Human site reliability engineers (SREs) and performance specialists spend excessive hours filtering background noise, correlating metric anomalies, and navigating distributed call traces.
Netflix transformed this discipline by introducing autonomous AI performance agents. These agents ingest real-time eBPF profiles, continuous telemetry, and canary telemetry, synthesizing diagnostic summaries and automated configuration remediation within minutes.
The Architecture of Automated Performance Profiling
Netflix deployed continuous profiling across its fleet using eBPF and Java Virtual Machine (JVM) toolkits. Rather than relying on periodic CPU snapshots, streaming daemons continuously capture stack traces across hundreds of thousands of AWS EC2 instances and container runtimes.
+-----------------------------------------------------------------------+
| NETFLIX PERFORMANCE AGENT PIPELINE |
+-----------------------------------------------------------------------+
| [ eBPF / JVM Agent ] ---> [ Vectorized Trace Ingestion / Atlas ] |
| | |
| v |
| [ Anomaly & Canary Detector ] |
| | |
| v |
| [ LLM Diagnostic Agent ] |
| / | \ |
| v v v |
| [ Flamegraph AST ] [ Git Diff ] [ JVM Heap Logs ] |
| \ | / |
| v v v |
| [ Root Cause Hypotheses ] |
| | |
| v |
| [ Spinnaker Canary Action ] |
+-----------------------------------------------------------------------+
When an upstream microservice observes an increase in p99 latency during a Spinnaker canary rollout, the telemetry platform emits an event triggering the performance engineering agent.
Transforming Flamegraphs into Machine-Readable Structures
Raw SVG flamegraphs or massive folded-stack text files overwhelm standard LLM context windows. An uncompressed folded-stack file for a complex video transcoding coordinator can easily exceed 40 megabytes.
Netflix condenses stack traces into structured hierarchical abstract syntax trees (ASTs), filtering out framework overhead:
{
"service": "playback-session-coordinator",
"anomaly_type": "p99_latency_regression",
"baseline_cpu_pct": 34.2,
"canary_cpu_pct": 58.7,
"top_delta_subtrees": [
{
"root_frame": "com.netflix.playback.session.CacheResolver::getOrFetch",
"cpu_time_delta_pct": "+23.4%",
"leaf_methods": [
"java.util.concurrent.ConcurrentHashMap::computeIfAbsent",
"org.apache.http.impl.client.CloseableHttpClient::execute"
],
"hypothesis": "Lock contention on thread-local cache combined with synchronous socket fallback"
}
]
}
By presenting the reasoning model with tree-level differentials rather than millions of raw stack strings, the agent isolates hotspot regressions instantly.
Correlating Code Commits with Runtime Execution Profiles
A performance anomaly is useless in isolation; it must map directly to the offending pull request or commit. The performance agent queries Git metadata for the exact build SHA deployed to the canary cluster.
It constructs a unified context containing:
- The differential flamegraph AST: Identifying the exact methods consuming excessive CPU or lock holding time.
- The code diff: Inspecting modified Java, Kotlin, or Go source files within those methods.
- Runtime environment configuration: Changes to environment variables, JVM thread pool parameters, or memory limits.
+---------------------------------------------------------------------+
| CANARY EVALUATION PIPELINE |
+---------------------------------------------------------------------+
| Metric | Baseline (v4.2.1) | Canary (v4.2.2) |
+-------------------------+---------------------+---------------------+
| P50 Latency | 12.4 ms | 13.1 ms |
| P99 Latency | 48.2 ms | 184.6 ms (REGRESSED)|
| Garbage Collection Time | 1.8% | 7.4% (REGRESSED) |
| CPU Utilization | 42% | 61% |
| Agent Assessment | PASS | REJECT & ROLLBACK |
+-------------------------+---------------------+---------------------+
Prompt Engineering and Verification Constraints
To ensure high-precision diagnoses without hallucinations, the agent operates under strict verification protocols. It cannot recommend rolling back a deployment unless two independent criteria are verified:
- The identified call path must directly correspond to modified lines within the canary SHA.
- The telemetry delta must exceed statistically verified thresholds (e.g., p99 latency > 3.0 standard deviations over rolling 7-day baseline).
def verify_regression_attribution(diff_ast, git_diff):
hotspot_methods = extract_divergent_methods(diff_ast)
modified_code_paths = extract_modified_functions(git_diff)
# Identify intersection of execution hotspots and code changes
intersected = set(hotspot_methods).intersection(set(modified_code_paths))
if not intersected:
return `{"status": "INCONCLUSIVE", "confidence": 0.3}`
return {
"status": "CONFIRMED_REGRESSION",
"culprit_methods": list(intersected),
"confidence": 0.96
}
Automated Canary Remediation and SRE Handoff
When the agent confirms a severe regression during early canary stages, it communicates directly with Spinnaker through an automated webhook, aborting the canary deployment before user-facing streaming degradation occurs.
Simultaneously, the agent drafts a pull request comment on GitHub:
- Highlighting the exact line where unbounded memory allocation or thread contention was introduced.
- Providing an annotated flamegraph visualization focused specifically on the divergent subtree.
- Recommending algorithmic or configuration alternatives (e.g., swapping a synchronous client for a non-blocking reactive stream).
By pairing deterministic eBPF profiling with high-context reasoning agents, Netflix transformed performance engineering from an exhausting post-mortem fire drill into an automated, proactive safeguard.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.