MongoDB WiredTiger Internals — Storage Engine Architecture, Checkpoints, and Tuning
WiredTiger Overview
WiredTiger (WT) is MongoDB’s default storage engine since 3.2 (2015). It’s a **log-structured merge (LSM) inspired B-tree** with MVCC, checkpointing, and tiered storage. Not a pure LSM (like RocksDB) — it’s a **B-tree with append-only updates** and background compaction.
Key Design Principles
- MVCC (Multi-Version Concurrency Control) — Readers never block writers; writers never block readers. Snapshot isolation via timestamps.
- Copy-on-Write B-trees — Updates create new pages; old pages retained until checkpoint. No in-place modification.
- Checkpoint = Consistent Snapshot — Periodic fsync of root page + metadata. Recovery replays from last checkpoint + WAL.
- Tiered Storage — Hot data in cache, cold on disk. Configurable page size, compression, encryption.
B-Tree Structure — Pages, Blocks, and the Root
Page Organization
WT stores data in **pages** (default 4KB, configurable 512B–128MB). Three page types:
| Type | Description | Key Structure |
|---|---|---|
| Leaf (Row Store) | Key-value data | Key = record ID (recno) or user key; Value = document bytes |
| Leaf (Column Store) | Fixed-width columns | Key = recno; Value = column chunk |
| Internal | Index navigation | Key = max key in child; Value = child page reference (block address + checksum) |
On-Disk Format — Blocks, Not Pages
Pages are **compressed and checksummed** into **blocks** before writing to disk. A 4KB page → 1-2KB block (with snappy/zstd/lz4). Block header: checksum (CRC32/HW), compression type, original size, flags.
// Simplified block header (WT_BLOCK_HDR)
struct {
uint32_t checksum; // CRC32C or xxHash
uint16_t flags; // COMPRESS, ENCRYPT, etc.
uint16_t comp_type; // 0=none, 1=snappy, 2=zlib, 3=zstd, 4=lz4
uint32_t orig_size; // Uncompressed page size
uint32_t comp_size; // Compressed size (block size - header)
uint64_t transaction_id;// For visibility
}
Root Page & Checkpoint
The **root page** anchors the entire B-tree. A checkpoint writes a new root page (with updated internal page refs) to a new block, then **atomically updates the metadata file** (WiredTiger.wt) with the new root block address. Old blocks become reclaimable after next checkpoint.
MVCC — Transaction IDs, Timestamps, and Visibility
Transaction ID (TXN ID)
Monotonically increasing 64-bit integer assigned at transaction start. Every page modification carries the TXN ID. Readers see pages with TXN ID ≤ their snapshot timestamp.
Read Timestamp / Commit Timestamp
- Read timestamp — Oldest transaction a reader needs to see. Set by MongoDB at query start.
- Commit timestamp — Assigned at commit (by WT or MongoDB). Durable timestamp for replication.
Visibility Rule
A page version is visible to a reader if:
page_txn_id <= reader_read_timestamp
AND (page_commit_timestamp == 0 OR page_commit_timestamp <= reader_read_timestamp)
Pages with future timestamps are **invisible** — reader traverses to older version (via page’s **on-page history chain** or **update restoration**).
History Store
When a page is updated, the **old version** moves to the **History Store** (separate B-tree: WiredTigerHS.wt). Key = original key + timestamp. Enables point-in-time reads and rollback. Cleaned by **HS cleanup thread** when no reader needs it.
Write Path — From Application to Disk
- Application Write → MongoDB translates to WT API (
WT_SESSION::update,insert,remove). - In-Memory Update — WT modifies page in **cache** (creates new version). Marks page dirty. Adds to **modify log** (in-memory, per-session).
- Transaction Commit — Assign commit timestamp. Write **WAL record** (see below). Release locks.
- Checkpoint (Background) — Periodically: reconcile dirty pages → write new blocks → write new root → fsync metadata.
- Eviction (Background) — When cache pressure: write dirty pages to disk (reconcile), evict clean pages.
Write-Ahead Log (WAL) — WiredTigerLog.wt
WT WAL is **logical** (operation + key + value), not physical (redo). Each transaction → one log record. Log files rotate (default 100MB). **Sync on commit** controlled by journal_commit config:
| Setting | Behavior | Durability |
|---|---|---|
journal_commit=sync |
fsync on every commit | Strong (default for standalone) |
journal_commit=write |
write() only, no fsync | Risk of loss on power fail |
journal_commit=none |
No journal (replica sets only) | Relies on replication |
MongoDB default: journal_commit=sync for standalone, write for replica set members (replication provides durability).
Checkpoints — Consistent Snapshots
What a Checkpoint Does
- Quiesce — Stop new transactions briefly (microseconds).
- Reconcile — Walk dirty pages in cache: serialize to blocks (compress, checksum), write to disk.
- Write Root — Create new root page pointing to reconciled internal pages.
- Update Metadata — Write new root block address to
WiredTiger.wt(metadata file). - Fsync — Force metadata + data blocks to stable storage.
- Advance — Update checkpoint timestamp. Old blocks eligible for reuse.
Checkpoint Triggers
- Time:
checkpoint=(wait=60)— every 60 seconds (default). - Log size:
checkpoint=(log_size=2GB)— when WAL reaches size. - Manual:
db.adminCommand({checkpoint: 1}). - Shutdown: Clean checkpoint on close.
Recovery
On startup: read latest checkpoint root from WiredTiger.wt. Replay WAL from checkpoint LSN forward. **No redo needed for committed data** — checkpoint already durable. WAL only replays transactions committed after checkpoint.
Cache & Eviction — The Heart of Performance
Cache Architecture
Single unified cache (default 50% RAM, min 256MB). Shared by: B-tree pages, internal pages, history store, index pages, lookaside table. **No separate OS page cache reliance** — WT uses direct I/O (O_DIRECT).
Eviction Policy
Target: Keep cache utilization at eviction_target (default 80%). Trigger: When > eviction_trigger (default 95%), aggressive eviction starts.
Eviction walks **LRU queue** (per-btree, global). For each page:
- If **clean** → drop from cache.
- If **dirty** → **reconcile** (write to disk), then drop.
- If **hazard pointer** (pinned by reader) → skip, try next.
Eviction Threads
eviction=(threads_min=4,threads_max=20)— scales with cache pressure.- Application threads can evict too (
eviction=(threads_min=1)by default) — “cooperative eviction”. - Server threads (checkpoint, history store cleanup) also participate.
Cache Pressure Symptoms
| Metric | Healthy | Pressure | Action |
|---|---|---|---|
cache bytes inuse |
< 80% | > 95% | Increase cache, reduce working set |
eviction walks |
Low | High | More eviction threads |
eviction pages walked |
< 1000/s | > 10000/s | Cache too small |
application threads evicting |
0 | > 0 | Cache critically full |
reconciliation time |
< 1ms | > 10ms | Compression too slow, page too large |
Compression — Page-Level, Not Block-Level
WT compresses **individual pages** before writing to disk. Configurable per-collection/index:
// Collection option
db.createCollection("events", {
storageEngine: { wiredTiger: { configString: "block_compressor=zstd" } }
})
// Index option
db.events.createIndex({ ts: 1 }, { storageEngine: { wiredTiger: { configString: "block_compressor=zstd" } } })
| Compressor | Ratio | Speed (comp/decomp) | Best For |
|---|---|---|---|
| snappy | ~2:1 | Very fast / Very fast | General purpose, CPU-constrained |
| zstd (default 6.0+) | ~3:1 | Fast / Very fast | Best balance (default) |
| zlib | ~3.5:1 | Slow / Medium | Archival, cold data |
| lz4 | ~1.8:1 | Extremely fast / Extremely fast | High-throughput, low latency |
| none | 1:1 | N/A | Pre-compressed data (images, video) |
Prefix compression (enabled by default): Common key prefixes stored once per page. Critical for ObjectId, UUID, timestamp prefixes.
Production Tuning — MongoDB 7.0/8.0
Cache Sizing
# /etc/mongod.conf
storage:
wiredTiger:
engineConfig:
cacheSizeGB: 24 # 50% of 64GB RAM; leave room for OS, connections, aggregation
journalCompressor: zstd
directoryForIndexes: true # Separate data/index files — easier tiering
Checkpoint Tuning
storage:
wiredTiger:
engineConfig:
checkpointDelaySecs: 60 # Default 60. Increase to 120-300 for write-heavy (less fsync)
# Warning: longer checkpoint = more WAL to replay on crash
Eviction Tuning
storage:
wiredTiger:
engineConfig:
evictionTarget: 80 # Default. Lower (70) = more headroom, more CPU
evictionTrigger: 95 # Default. Lower = earlier aggressive eviction
evictionDirtyTarget: 5 # Default 5%. Dirty page target %
evictionDirtyTrigger: 20 # Default 20%. Aggressive dirty eviction trigger
Compression Per Workload
// Time-series / logs — zstd level 3 (fast, good ratio)
db.createCollection("metrics", {
storageEngine: { wiredTiger: { configString: "block_compressor=zstd:3" } }
})
// High-frequency trading — lz4 (lowest latency)
db.createCollection("trades", {
storageEngine: { wiredTiger: { configString: "block_compressor=lz4" } }
})
// Archival — zstd level 19 (max ratio)
db.createCollection("archive", {
storageEngine: { wiredTiger: { configString: "block_compressor=zstd:19" } }
})
Monitoring Queries
// Cache stats
db.serverStatus().wiredTiger.cache
// Eviction stats
db.serverStatus().wiredTiger.eviction
// Checkpoint stats
db.serverStatus().wiredTiger.checkpoint
// Transaction stats
db.serverStatus().wiredTiger.transaction
// Block manager (disk I/O)
db.serverStatus().wiredTiger.blockManager
Common Issues & Fixes
| Symptom | Root Cause | Fix |
|---|---|---|
| High CPU in eviction | Cache too small for working set | Increase cacheSizeGB; add RAM; shard |
| Checkpoint spikes (IOPS) | Too much dirty data at once | Increase checkpointDelaySecs; tune evictionDirtyTarget |
| History store growth | Long-running transactions / slow secondaries | Kill long ops; ensure secondary replication lag < 10s |
| WAL disk full | Checkpoint not advancing (crash recovery stuck) | Check WiredTiger.wt corruption; restore from backup |
Slow queries, high cache bytes inuse |
Unindexed scans loading full collection | Add indexes; use projection; limit results |
TL;DR — Key Takeaways
- WiredTiger = **Copy-on-write B-tree + MVCC + WAL + Checkpoints**. Not an LSM tree.
- **Cache is everything** — 50% RAM default, monitor
evictionmetrics religiously. - **Checkpoint = durability boundary** — 60s default. Tune for write-heavy workloads.
- **Compression** — zstd default (best balance). Per-collection override for specialized workloads.
- **History store** grows with long transactions / replication lag. Monitor
history store size. - **Direct I/O** — WT bypasses OS page cache. Give it RAM via
cacheSizeGB, not OS.
Next in series: etcd Raft Consensus — leader election, log replication, and membership changes.