---
topic: systems-architecture
author: Crashtech Editorial
published: Nov 15, 2026 · read: 6 min
---

Inside Datadog’s Monocle: A Rust Engine for Real-Time Metrics

Datadog's Monocle uses worker-per-shard LSM trees, private memtables, a unified series cache, and tiered compaction for recent metrics.

– –

Datadog’s August 2025 engineering post describes its sixth-generation real-time timeseries database and reports a 60× increase in ingestion performance and 5× faster queries at peak scale compared with the previous design. At its core is Monocle, a custom Rust storage engine. Its documented architecture is a case study in separating tag lookup from point storage and tuning LSM-based storage for a real-time metrics workload.

+-----------------------------------------------------------------------------------+
|                        DATADOG REAL-TIME METRICS PIPELINE                         |
+-----------------------------------------------------------------------------------+
                                          |
                                   [ Metrics Edge ]
                            (TLS Termination, Auth, Rate Limits)
                                          |
                                  [ Storage Router ]
                                          |
                    +---------------------+---------------------+
                    |                                           |
                    v                                           v
       [ Long-Term Metrics Storage ]               [ Real-Time Metrics Storage ]
         (Long-term storage)                          (Recent metrics and queries)
                                                                |
                                                      [ Sharded RTDB workers ]
                                                                |
                                                   +------------+------------+
                                                   |                         |
                                                   v                         v
                                            [ RTDB Node 1 ]           [ RTDB Node 2 ]
                                            +-------------+           +-------------+
                                            | Monocle     |           | Monocle     |
                                            | Rust Engine |           | Rust Engine |
                                            +-------------+           +-------------+

1. Bifurcated Architecture: Real-Time vs Long-Term Storage

Time-series queries exhibit extreme temporal skew:

  1. Real-time storage: RTDB stores raw metric points and serves recent data.
  2. Tag lookup: A separate index database stores series identifiers and tags, which the query service uses to locate relevant series.

Datadog’s post describes RTDB and the index database as separately deployed services, with a storage router distributing incoming metrics and the query service coordinating results.

Decoupling Metadata from Metric Values

Every metric point contains two fundamentally distinct elements:

  • Numerical Time-Series Values: A sequence of (timestamp: u32, value: f64) samples.
  • Dimensional Metadata (Tags): Key-value labels such as metric: "http.requests", host: "prod-api-04", service: "checkout", region: "us-east-1".
Query: "avg:http.requests{service:checkout, region:us-east-1} by `{host}`"

Step 1: [ Index Database ]
        Resolves tag filter -> Returns matching Series IDs: [0x8A1F, 0x9B2C, 0x4D3E]

Step 2: [ Monocle RTDB ]
        Direct lookup by Series IDs -> Fetches [t, v] arrays from memory & SSD

The platform separates two concerns:

  • The Index Database: Manages metric identifiers and tags.
  • The Real-Time Database (RTDB): Stores raw point tuples and serves recent metric data; Monocle is its Rust storage engine.

2. Sharding the real-time storage path

Datadog says each timeseries key is assigned to a shard, and sharding applies across ingestion, caches, storage files, and query processing. The article does not identify Kafka as Monocle’s replication or write-ahead-log component.

+-----------------------------------------------------------------------------------+
|                     KAFKA-CENTRIC REPLICATION AND ROUTING                         |
+-----------------------------------------------------------------------------------+

Incoming Telemetry
        |
        v
 [ Incoming timeseries data ]
   |-- Partition 01 (AZ-1 Leader, Replicas in AZ-2, AZ-3)
   |-- Partition 02 (AZ-2 Leader, Replicas in AZ-1, AZ-3)
   \-- Partition 03 (AZ-3 Leader, Replicas in AZ-1, AZ-2)
        |
        +-----------------------------------+-----------------------------------+
        |                                   |                                   |
        v                                   v                                   v
 [ RTDB Shard Node A ]               [ RTDB Shard Node B ]               [ RTDB Shard Node C ]
 (Consumes Partition 01)             (Consumes Partition 02)             (Consumes Partition 03)
   - Zero peer chatter                 - Zero peer chatter                 - Zero peer chatter
   - Independent local LSM             - Independent local LSM             - Independent local LSM
   - Worker owns one shard             - Worker owns one shard             - Worker owns one shard

Sharding spreads work across workers and lets each shard manage its own storage and cache. Datadog says this also simplifies concurrency within a shard. The post presents these as design choices behind its reported performance improvements, not as a general elimination of distributed coordination.


3. Monocle engine internals

Datadog’s new RTDB uses Monocle, a custom storage engine written in Rust. Datadog says it was designed as a standalone library for reuse across multiple systems.

Tag Hashing Data Model

Monocle discards all tag strings at ingestion. The metric stream’s identity is compressed into a 64-bit fingerprint:

Series ID = SipHash64(Org ID || Metric Name || Sorted Canonical Tags)

Monocle indexes data using a streamlined primary key:

// Core memory representation of a metric series key
#[derive(Copy, Clone, Hash, Eq, PartialEq, Ord, PartialOrd)]
pub struct SeriesKey {
    pub org_id: u32,
    pub metric_hash: u64,
    pub tag_hash: u64,
}

pub struct DataPoint {
    pub timestamp: u32,
    pub value: f64,
}

Finding any time-series record requires an $O(1)$ direct hash map lookup. When human users formulate dashboard queries, the Index Database resolves metadata labels into matching SeriesKey hashes, and passes the hash array directly to Monocle.


4. Concurrency: one worker per shard

Datadog describes a worker-per-shard design. Each worker manages its own LSM-tree and runs on a single-threaded Tokio scheduler; each worker’s private memtable is not concurrently written by other threads.

+-----------------------------------------------------------------------------------+
|                 MONOCLE THREAD-PER-CORE SHARED-NOTHING SHARDING                   |
+-----------------------------------------------------------------------------------+

Incoming metric stream
        |
        v
 [ Murmur/SipHash % Num_Cores ]
        |
        +-----------------------+-----------------------+
        |                       |                       |
        v                       v                       v
 [ Worker Thread: Core 0 ] [ Worker Thread: Core 1 ] [ Worker Thread: Core 2 ]
   |-- Dedicated MPSC Ring   |-- Dedicated MPSC Ring   |-- Dedicated MPSC Ring
   |-- Local MemTable Arena  |-- Local MemTable Arena  |-- Local MemTable Arena
   |-- Local SSTable Cache   |-- Local SSTable Cache   |-- Local SSTable Cache
   \-- CPU Affinity (Pinned) \-- CPU Affinity (Pinned) \-- CPU Affinity (Pinned)

Architectural Guarantees:

  • Worker-per-shard: Each worker handles a shard’s ingestion, queries, and background work.
  • Private memtable: A worker owns its memtable, avoiding cross-thread contention for writes to it.
  • Concurrency scope: The article describes the storage worker scheduler and memtable ownership; it does not say workers are pinned to physical cores or every path is lock-free.
// Architectural sketch: Shard Worker Channel Dispatch
pub struct IngressDispatcher {
    shard_senders: Vec<tokio::sync::mpsc::UnboundedSender<MetricBatch>>,
    shard_count: usize,
}

impl IngressDispatcher {
    pub fn dispatch(&self, key: SeriesKey, points: Vec<DataPoint>) {
        // Deterministic routing to one shard
        let shard_id = (key.tag_hash as usize) % self.shard_count;
        let _ = self.shard_senders[shard_id].send(MetricBatch { key, points });
    }
}

5. LSM storage and time-window pruning

Time-series ingestion is purely append-only; existing metrics are never updated in-place. This makes the Log-Structured Merge-Tree (LSM-Tree) the optimal storage primitive.

Incoming samples accumulate in an active in-memory buffer called the MemTable. When the MemTable reaches capacity, it freezes into an immutable buffer, flushes sequentially to disk as a sorted SSTable (Sorted String Table), and is replaced by a fresh active MemTable.

Incoming Writes
      |
      v
+-------------------------------------------------------------------+
| MEMTABLE (DRAM)                                                   |
| In-memory write buffer                                            |
+-------------------------------------------------------------------+
      |
      | Sequential Flush (Zero Fragmented Heap Allocations)
      v
+-------------------------------------------------------------------+
| L0 SSTABLES (Local NVMe Storage)                                  |
| [File A: 10:00 - 10:15]  [File B: 10:15 - 10:30]  [File C: ...]   |
+-------------------------------------------------------------------+
      |
      | Background Compaction (Merge Sorted Runs)
      v
+-------------------------------------------------------------------+
| L1 SSTABLES (2-Hour Coalesced Windows)                            |
| [Block 1: 08:00 - 10:00]          [Block 2: 10:00 - 12:00]        |
+-------------------------------------------------------------------+

An LSM tree buffers recent writes in memory, flushes immutable files, and compacts files in the background. Datadog describes a tiered compaction strategy and query-side filters to balance write and read costs.

Time-aware file selection and tiered compaction

Datadog’s article describes tiered compaction, in-memory timestamp bounds, and probabilistic filters to avoid reading irrelevant files. It notes that tiered compaction reduces write amplification and CPU work versus leveled compaction, with a potential increase in read amplification.

For a time-range query, Monocle can use each file’s minimum and maximum timestamps to skip files outside the requested window:

Query Range = [t_`{now}` - 900, t_`{now}`]

Monocle can skip files whose timestamp ranges cannot contain the requested data. The post does not quantify a fixed percentage of files pruned.


6. Query scheduling and bounded work

In a thread-per-core shared-nothing architecture, a multi-core query is fanned out across all worker cores. The query response latency is bounded by the slowest worker:

Latency_`{query}` = max(Latency_`{core_0}`, Latency_`{core_1}`, |u|dots, Latency_`{core_n}`)

If Core 7 is stalled performing an aggressive background SSTable compaction or reading a fragmented time series from NVMe, the entire query stalls. This is the classic distributed head-of-line blocking problem.

Datadog deployed a two-tier resilience architecture to prevent system collapse under unexpected query bursts.

Incoming Query Request
         |
         v
+---------------------------------------------------------+
| Layer 1: Monocle Admission Control                      |
| Evaluates queued query cost and wait                      |
+---------------------------------------------------------+
         |
         | [Accepted]
         v
+---------------------------------------------------------+
| Layer 2: Cost-Based CoDel Scheduler                     |
| Prioritizes Low-Cost Alert Monitors Over Heavy Scans     |
+---------------------------------------------------------+
         |
         +-----------------------+-----------------------+
         |                       |                       |
         v                       v                       v
    [ Core 0 ]              [ Core 1 ]              [ Core 2 ]

Layer 1: Admission Control Gate

Monocle continuously tracks two vital health metrics:

  1. Kafka Consumer Offset Lag: The time difference between the newest message produced to Kafka and the offset Monocle is processing.
  2. Resident Memory Headroom: Active memory utilization across the arena allocators.

If consumer lag crosses a safety threshold, Monocle engages admission control, rejecting expensive ad-hoc user queries at the API edge with HTTP 429 / backpressure signals. This prioritizes ingestion freshness and automated alerting monitors over interactive exploratory dashboards.

Layer 2: Cost-Based CoDel (Controlled Delay) Scheduling

To prevent complex queries from monopolizing worker threads, Monocle implements an adapted CoDel algorithm.

CoDel tracks the minimum sojourn time (queue waiting time) of requests over a sliding time window:

  • Queries are assigned estimated resource costs based on time range and target series count.
  • If queuing delay exceeds latency budgets (such as 5 milliseconds), the scheduler automatically sheds low-priority queries.
  • Automated alerts (which detect outages) are prioritized in dedicated execution queues over interactive user dashboards.

7. The Road Ahead: Columnar Unification

Datadog continues to push Monocle’s architecture forward to meet exponential data growth:

  1. Dynamic Real-Time Re-Sharding: While static partition hashing works cleanly under uniform load, high-severity incidents cause localized traffic spikes (hot spots) on specific metrics. Datadog is engineering dynamic partition re-routing to rebalance hot metric keys across idle cores without downtime.
  2. Unifying Index and Time-Series Storage: Currently, resolving queries requires two hops: the Index Database (tags to hashes) and Monocle (hashes to points). Datadog is working toward merging both tiers into an in-memory, vectorized columnar storage format. By storing string dictionary codes and metric timestamps side-by-side in SIMD-aligned columnar blocks, Monocle will evaluate complex tag predicates and numerical aggregations within a single pass.

Architectural Comparison: General-Purpose vs Monocle

DimensionGeneral-Purpose Store (e.g., RocksDB)Datadog Monocle
Implementation LanguageC++Rust (Tokio)
Concurrency ModelGeneral purposeWorker-per-shard with private memtables
Data Key ModelArbitrary String KeysHashed 64-bit Tags (Org, Metric, TagHash)
Memory AllocationGeneral purposePrivate per-worker memtables
Replication / ConsensusNot specified in this articleNot specified in this article
Query protectionQuery work can queueDatadog describes cost-based scheduling using CoDel

Summary Takeaways

Datadog’s Monocle illustrates how a purpose-built real-time storage engine can tune its data layout and background work to a specific workload:

  • Separate tag lookup from point storage: The metrics platform uses distinct services for metric identifiers/tags and raw points.
  • Own data by shard: Worker-per-shard processing scopes ingestion and storage work.
  • Choose compaction for the workload: Tiered compaction favors lower write amplification at the cost of possible extra reads; time bounds and filters help prune irrelevant files.
  • Control query contention: Datadog describes cost-based scheduling to manage competing queries.
Advertisement

Frequently asked questions

Why did Datadog split time-series metadata from metric values into two separate databases?

Datadog describes Monocle as a custom Rust storage engine used within its real-time timeseries database. The broader metrics platform separates metric identifiers and tags from raw metric points.

How does Monocle use sharding?

Datadog assigns timeseries keys to shards across ingestion, caching, storage files, and query processing. Its article says this spreads load and simplifies concurrency within a shard.

How does each Monocle storage worker process data?

Each storage worker manages a shard and its own LSM-tree instance on a single-threaded Tokio scheduler. Datadog says the shard-per-thread design avoids cross-thread contention on a worker's private memtable.

What happens when a Monocle memtable fills?

The worker seals its private memtable and flushes it asynchronously to disk, then directs new writes to a fresh memtable.

Sources & further reading

/* Comments */