MongoDB WiredTiger Internals — Storage Engine Architecture, Checkpoints, and Tuning

//

//

MongoDB WiredTiger Internals — Storage Engine Architecture, Checkpoints, and Tuning

MongoDB WiredTiger Internals — Storage Engine Architecture, Checkpoints, and Tuning

WiredTiger Overview

WiredTiger (WT) is MongoDB’s default storage engine since 3.2 (2015). It’s a **log-structured merge (LSM) inspired B-tree** with MVCC, checkpointing, and tiered storage. Not a pure LSM (like RocksDB) — it’s a **B-tree with append-only updates** and background compaction.

Key Design Principles

  • MVCC (Multi-Version Concurrency Control) — Readers never block writers; writers never block readers. Snapshot isolation via timestamps.
  • Copy-on-Write B-trees — Updates create new pages; old pages retained until checkpoint. No in-place modification.
  • Checkpoint = Consistent Snapshot — Periodic fsync of root page + metadata. Recovery replays from last checkpoint + WAL.
  • Tiered Storage — Hot data in cache, cold on disk. Configurable page size, compression, encryption.

B-Tree Structure — Pages, Blocks, and the Root

Page Organization

WT stores data in **pages** (default 4KB, configurable 512B–128MB). Three page types:

Type Description Key Structure
Leaf (Row Store) Key-value data Key = record ID (recno) or user key; Value = document bytes
Leaf (Column Store) Fixed-width columns Key = recno; Value = column chunk
Internal Index navigation Key = max key in child; Value = child page reference (block address + checksum)

On-Disk Format — Blocks, Not Pages

Pages are **compressed and checksummed** into **blocks** before writing to disk. A 4KB page → 1-2KB block (with snappy/zstd/lz4). Block header: checksum (CRC32/HW), compression type, original size, flags.

// Simplified block header (WT_BLOCK_HDR)
struct {
  uint32_t checksum;      // CRC32C or xxHash
  uint16_t flags;         // COMPRESS, ENCRYPT, etc.
  uint16_t comp_type;     // 0=none, 1=snappy, 2=zlib, 3=zstd, 4=lz4
  uint32_t orig_size;     // Uncompressed page size
  uint32_t comp_size;     // Compressed size (block size - header)
  uint64_t transaction_id;// For visibility
}

Root Page & Checkpoint

The **root page** anchors the entire B-tree. A checkpoint writes a new root page (with updated internal page refs) to a new block, then **atomically updates the metadata file** (WiredTiger.wt) with the new root block address. Old blocks become reclaimable after next checkpoint.

MVCC — Transaction IDs, Timestamps, and Visibility

Transaction ID (TXN ID)

Monotonically increasing 64-bit integer assigned at transaction start. Every page modification carries the TXN ID. Readers see pages with TXN ID ≤ their snapshot timestamp.

Read Timestamp / Commit Timestamp

  • Read timestamp — Oldest transaction a reader needs to see. Set by MongoDB at query start.
  • Commit timestamp — Assigned at commit (by WT or MongoDB). Durable timestamp for replication.

Visibility Rule

A page version is visible to a reader if:

page_txn_id <= reader_read_timestamp
AND (page_commit_timestamp == 0 OR page_commit_timestamp <= reader_read_timestamp)

Pages with future timestamps are **invisible** — reader traverses to older version (via page’s **on-page history chain** or **update restoration**).

History Store

When a page is updated, the **old version** moves to the **History Store** (separate B-tree: WiredTigerHS.wt). Key = original key + timestamp. Enables point-in-time reads and rollback. Cleaned by **HS cleanup thread** when no reader needs it.

Write Path — From Application to Disk

  1. Application Write → MongoDB translates to WT API (WT_SESSION::update, insert, remove).
  2. In-Memory Update — WT modifies page in **cache** (creates new version). Marks page dirty. Adds to **modify log** (in-memory, per-session).
  3. Transaction Commit — Assign commit timestamp. Write **WAL record** (see below). Release locks.
  4. Checkpoint (Background) — Periodically: reconcile dirty pages → write new blocks → write new root → fsync metadata.
  5. Eviction (Background) — When cache pressure: write dirty pages to disk (reconcile), evict clean pages.

Write-Ahead Log (WAL) — WiredTigerLog.wt

WT WAL is **logical** (operation + key + value), not physical (redo). Each transaction → one log record. Log files rotate (default 100MB). **Sync on commit** controlled by journal_commit config:

Setting Behavior Durability
journal_commit=sync fsync on every commit Strong (default for standalone)
journal_commit=write write() only, no fsync Risk of loss on power fail
journal_commit=none No journal (replica sets only) Relies on replication

MongoDB default: journal_commit=sync for standalone, write for replica set members (replication provides durability).

Checkpoints — Consistent Snapshots

What a Checkpoint Does

  1. Quiesce — Stop new transactions briefly (microseconds).
  2. Reconcile — Walk dirty pages in cache: serialize to blocks (compress, checksum), write to disk.
  3. Write Root — Create new root page pointing to reconciled internal pages.
  4. Update Metadata — Write new root block address to WiredTiger.wt (metadata file).
  5. Fsync — Force metadata + data blocks to stable storage.
  6. Advance — Update checkpoint timestamp. Old blocks eligible for reuse.

Checkpoint Triggers

  • Time: checkpoint=(wait=60) — every 60 seconds (default).
  • Log size: checkpoint=(log_size=2GB) — when WAL reaches size.
  • Manual: db.adminCommand({checkpoint: 1}).
  • Shutdown: Clean checkpoint on close.

Recovery

On startup: read latest checkpoint root from WiredTiger.wt. Replay WAL from checkpoint LSN forward. **No redo needed for committed data** — checkpoint already durable. WAL only replays transactions committed after checkpoint.

Cache & Eviction — The Heart of Performance

Cache Architecture

Single unified cache (default 50% RAM, min 256MB). Shared by: B-tree pages, internal pages, history store, index pages, lookaside table. **No separate OS page cache reliance** — WT uses direct I/O (O_DIRECT).

Eviction Policy

Target: Keep cache utilization at eviction_target (default 80%). Trigger: When > eviction_trigger (default 95%), aggressive eviction starts.

Eviction walks **LRU queue** (per-btree, global). For each page:

  1. If **clean** → drop from cache.
  2. If **dirty** → **reconcile** (write to disk), then drop.
  3. If **hazard pointer** (pinned by reader) → skip, try next.

Eviction Threads

  • eviction=(threads_min=4,threads_max=20) — scales with cache pressure.
  • Application threads can evict too (eviction=(threads_min=1) by default) — “cooperative eviction”.
  • Server threads (checkpoint, history store cleanup) also participate.

Cache Pressure Symptoms

Metric Healthy Pressure Action
cache bytes inuse < 80% > 95% Increase cache, reduce working set
eviction walks Low High More eviction threads
eviction pages walked < 1000/s > 10000/s Cache too small
application threads evicting 0 > 0 Cache critically full
reconciliation time < 1ms > 10ms Compression too slow, page too large

Compression — Page-Level, Not Block-Level

WT compresses **individual pages** before writing to disk. Configurable per-collection/index:

// Collection option
db.createCollection("events", {
  storageEngine: { wiredTiger: { configString: "block_compressor=zstd" } }
})

// Index option
db.events.createIndex({ ts: 1 }, { storageEngine: { wiredTiger: { configString: "block_compressor=zstd" } } })
Compressor Ratio Speed (comp/decomp) Best For
snappy ~2:1 Very fast / Very fast General purpose, CPU-constrained
zstd (default 6.0+) ~3:1 Fast / Very fast Best balance (default)
zlib ~3.5:1 Slow / Medium Archival, cold data
lz4 ~1.8:1 Extremely fast / Extremely fast High-throughput, low latency
none 1:1 N/A Pre-compressed data (images, video)

Prefix compression (enabled by default): Common key prefixes stored once per page. Critical for ObjectId, UUID, timestamp prefixes.

Production Tuning — MongoDB 7.0/8.0

Cache Sizing

# /etc/mongod.conf
storage:
  wiredTiger:
    engineConfig:
      cacheSizeGB: 24  # 50% of 64GB RAM; leave room for OS, connections, aggregation
      journalCompressor: zstd
      directoryForIndexes: true  # Separate data/index files — easier tiering

Checkpoint Tuning

storage:
  wiredTiger:
    engineConfig:
      checkpointDelaySecs: 60  # Default 60. Increase to 120-300 for write-heavy (less fsync)
      # Warning: longer checkpoint = more WAL to replay on crash

Eviction Tuning

storage:
  wiredTiger:
    engineConfig:
      evictionTarget: 80      # Default. Lower (70) = more headroom, more CPU
      evictionTrigger: 95     # Default. Lower = earlier aggressive eviction
      evictionDirtyTarget: 5  # Default 5%. Dirty page target %
      evictionDirtyTrigger: 20 # Default 20%. Aggressive dirty eviction trigger

Compression Per Workload

// Time-series / logs — zstd level 3 (fast, good ratio)
db.createCollection("metrics", {
  storageEngine: { wiredTiger: { configString: "block_compressor=zstd:3" } }
})

// High-frequency trading — lz4 (lowest latency)
db.createCollection("trades", {
  storageEngine: { wiredTiger: { configString: "block_compressor=lz4" } }
})

// Archival — zstd level 19 (max ratio)
db.createCollection("archive", {
  storageEngine: { wiredTiger: { configString: "block_compressor=zstd:19" } }
})

Monitoring Queries

// Cache stats
db.serverStatus().wiredTiger.cache

// Eviction stats
db.serverStatus().wiredTiger.eviction

// Checkpoint stats
db.serverStatus().wiredTiger.checkpoint

// Transaction stats
db.serverStatus().wiredTiger.transaction

// Block manager (disk I/O)
db.serverStatus().wiredTiger.blockManager

Common Issues & Fixes

Symptom Root Cause Fix
High CPU in eviction Cache too small for working set Increase cacheSizeGB; add RAM; shard
Checkpoint spikes (IOPS) Too much dirty data at once Increase checkpointDelaySecs; tune evictionDirtyTarget
History store growth Long-running transactions / slow secondaries Kill long ops; ensure secondary replication lag < 10s
WAL disk full Checkpoint not advancing (crash recovery stuck) Check WiredTiger.wt corruption; restore from backup
Slow queries, high cache bytes inuse Unindexed scans loading full collection Add indexes; use projection; limit results

TL;DR — Key Takeaways

  • WiredTiger = **Copy-on-write B-tree + MVCC + WAL + Checkpoints**. Not an LSM tree.
  • **Cache is everything** — 50% RAM default, monitor eviction metrics religiously.
  • **Checkpoint = durability boundary** — 60s default. Tune for write-heavy workloads.
  • **Compression** — zstd default (best balance). Per-collection override for specialized workloads.
  • **History store** grows with long transactions / replication lag. Monitor history store size.
  • **Direct I/O** — WT bypasses OS page cache. Give it RAM via cacheSizeGB, not OS.

Next in series: etcd Raft Consensus — leader election, log replication, and membership changes.

// THE TRANSMISSION LOG : FIELD NOTES

Weekly zero-fluff breakdowns of distributed systems, browser runtime performance, and production post-mortems delivered directly from Stella Sage.