etcd Raft Consensus — Leader Election, Log Replication, and Membership Changes

By Marcus Vance · Lead Systems & Backend Architect · November 2026

Raft in etcd — The Basics

etcd uses the **Raft consensus algorithm** (Ongaro & Ousterhout, 2014) for replicated state machine. Raft decomposes consensus into: **Leader Election**, **Log Replication**, **Safety**, **Membership Changes**.

Cluster Roles

Term & Log Structure

• Term — Logical time (monotonic integer). New term = new election. Embedded in every RPC.
• Log Entry — {term, index, type, data}. Types: Normal (client command), ConfChange (membership), ConfChangeV2 (joint consensus).
• Commit Index — Highest index known to be replicated on majority. Applied to state machine.
• Applied Index — Highest index applied to KV store.

Leader Election — The Election Protocol

Election Timeout

Followers reset election timer on: valid AppendEntries (heartbeat), RequestVote (granting vote). Timer range: election-timeout-ms to 2 * election-timeout-ms (default 100-200ms → 100-200ms random). **Randomization prevents split vote**.

RequestVote RPC

type RequestVoteRequest struct {
  Term         uint64
  CandidateID  string
  LastLogIndex uint64
  LastLogTerm  uint64
}

type RequestVoteResponse struct {
  Term        uint64
  VoteGranted bool
}

Follower grants vote if:

1. candidate.Term >= follower.Term
2. Follower hasn’t voted in this term (or voted for same candidate)
3. Candidate’s log is at least as up-to-date: (LastLogTerm > follower.LastLogTerm) OR (LastLogTerm == follower.LastLogTerm AND LastLogIndex >= follower.LastLogIndex)

Pre-Vote (etcd 3.4+) — Prevents Disruption

Before starting election, candidate sends PreVote (term = current + 1, special flag). Followers respond as if term+1. If majority grants PreVote, candidate proceeds to real election. **Prevents partitioned node from disrupting cluster with higher term**.

Leadership Transfer (Graceful Handoff)

etcdctl move-leader — Leader sends MsgTimeoutNow to target. Target immediately starts election (shortened timeout). Current leader steps down. Used for maintenance, rolling upgrades.

Log Replication — The Replication Pipeline

Client Write Path

1. Client sends proposal to leader (any node forwards to leader via raftpb.Message).
2. Leader appends to local log (uncommitted). Sends AppendEntries to all followers.
3. Followers append, respond Success.
4. Leader waits for **majority** (including self). Updates commitIndex.
5. Leader applies to state machine, responds to client.
6. Next heartbeat piggybacks new commitIndex → followers apply.

AppendEntries RPC

type AppendEntriesRequest struct {
  Term         uint64
  LeaderID     string
  PrevLogIndex uint64
  PrevLogTerm  uint64
  Entries      []Entry
  LeaderCommit uint64
}

type AppendEntriesResponse struct {
  Term    uint64
  Success bool
  MatchIndex uint64  // Follower's highest matched index (for optimization)
  ConflictIndex uint64 // For fast conflict resolution
  ConflictTerm uint64
}

Follower validates: PrevLogIndex/Term must match local log. If mismatch → Success=false + conflict info. Leader **backs off** (decrements nextIndex) and retries.

Flow Control — Inflights & Window

Leader tracks per-follower inflight entries (sent, not acknowledged). Window size: max-inflight-msgs (default 256). Prevents overwhelming slow followers. **Pipeline** — leader sends next batch before previous acked.

Read Index / Lease Read — Linearizable Reads

Raft log replication ensures linearizable writes. For reads:

• Read Index (default): Leader gets latest commitIndex from majority (heartbeat round), then serves read locally at that index. No log write.
• Lease Read: Leader assumes lease validity (no election) for lease-interval. Faster but risks stale read if leader partitioned.
• Follower Read (stale): Read from any follower. No consistency guarantee.

Snapshotting & Log Compaction

Why Snapshots?

Log grows unbounded. Snapshots compact applied prefix into state machine dump. etcd snapshots: **periodic** (time/size) + **manual**.

Snapshot Process

1. Leader (or any node) triggers snapshot at snapshot-count entries (default 100,000).
2. State machine serializes KV store to temp file (protocol buffers).
3. Save metadata: term, index, CRC.
4. Rename atomically → snap/db--.snap.
5. Compact raft log: delete entries ≤ snapshot index (via Storage.Compact).

Snapshot Transfer (Slow Follower / New Member)

If follower’s log too far behind (or new learner), leader sends **InstallSnapshot** RPC (chunked). Follower writes chunks, loads state machine, truncates log. **Streaming** — doesn’t block leader.

Membership Changes — Joint Consensus

Single-Node Changes (Unsafe)

Adding/removing one node at a time with simple majority overlap **can lose quorum** during transition. Example: 3-node → remove A → 2-node (B,C). If B fails during transition → no majority.

Joint Consensus (Raft PhD Thesis / etcd ConfChangeV2)

Transition uses **two configurations overlapping**: Cold ∪ Cnew. Both majorities must agree.

1. Enter Joint: Leader proposes ConfChangeV2{Joint: true, AutoLeave: false, Changes: [{Add: B}, {Add: C}]}. Cluster operates under Cold ∪ Cnew (majority = majority of both).
2. Commit Joint: Once committed, all nodes switch to joint config.
3. Leave Joint: Leader proposes ConfChangeV2{Joint: false, Changes: [{Remove: A}]}. Cluster moves to Cnew.

etcd automates this via ConfChangeV2 with AutoLeave: true — single proposal handles both phases.

Learner Promotion

Add node as Learner (non-voting) → replicates log, catches up → promote to voter via ConfChangeV2{Changes: [{Type: PromoteLearner}]}. Safe scaling.

Operational Commands

# Add member (auto joint consensus)
etcdctl member add node4 --peer-urls=https://10.0.0.4:2380

# Remove member
etcdctl member remove 

# Promote learner
etcdctl member promote 

# List members
etcdctl member list -w table

Production Operations — Kubernetes Backing Store

Cluster Sizing

**Odd numbers only** — even adds failure tolerance without quorum benefit.

Critical Configuration

# /etc/etcd/etcd.conf.yml
etcd:
  # Election timing (tune for network RTT)
  election-timeout-ms: 500      # Default 100. Increase for cross-AZ (500-1000)
  heartbeat-interval-ms: 100    # Default 100. Leader → follower heartbeat

  # Snapshotting
  snapshot-count: 100000        # Default. Lower = more frequent snapshots, more disk I/O
  quota-backend-bytes: 8589934592  # 8GB default. Max DB size. Set to 80% disk.

  # Performance
  max-request-bytes: 33554432   # 32MB max request (K8s objects can be large)
  max-txn-ops: 128              # Max ops per transaction

  # Security
  client-transport-security:
    cert-file: /etc/etcd/peer.crt
    key-file: /etc/etcd/peer.key
    client-cert-auth: true
  peer-transport-security:
    cert-file: /etc/etcd/peer.crt
    key-file: /etc/etcd/peer.key
    client-cert-auth: true

Monitoring — Key Metrics

Common Operational Tasks

Defragmentation (Reclaim Space)

# Online, per-member (run sequentially!)
etcdctl defrag --endpoints=https://:2379

# Check space before/after
etcdctl endpoint status --endpoints=https://:2379 -w table

Backup / Restore

# Backup (snapshot)
etcdctl snapshot save /backup/etcd-$(date +%F).db --endpoints=https://:2379

# Restore (new cluster)
etcdctl snapshot restore /backup/etcd-2026-11-01.db 
  --name=new-etcd-1 
  --initial-cluster=new-etcd-1=https://10.0.0.1:2380 
  --initial-cluster-token=etcd-cluster 
  --skip-hash-check

Disaster Recovery — Lost Quorum

1. Stop all etcd members.
2. On one node: etcdctl snapshot restore ... --initial-cluster=... (single node).
3. Start that node as single-member cluster.
4. Add remaining members one by one (member add).
5. Update K8s API server --etcd-servers.

Performance Tuning for Large Clusters

• Large K8s objects: Increase max-request-bytes (CRDs, large ConfigMaps).
• High write throughput: Batch updates (K8s informer resync), enable experimental-initial-corrupt-check.
• Cross-AZ latency: Increase election-timeout-ms to 5-10x RTT. Use learners for read scaling in remote AZ.
• Disk: Dedicated NVMe for WAL + DB. --wal-dir separate from --data-dir.
• Memory: etcd uses BoltDB (mmap). Ensure quota-backend-bytes < physical RAM.

TL;DR — Key Takeaways

• Raft = **Leader election + Log replication + Safety + Membership changes**. etcd implements all.
• Pre-Vote prevents disrupted elections. Joint Consensus enables safe membership changes.
• Read Index = linearizable reads without log write. Default in etcd 3.4+.
• Snapshots every 100K entries. **Defrag** reclaims space. **Backup** before upgrades.
• 3 nodes = standard HA. 5 for cross-AZ. Monitor leader_changes, wal_fsync, db_size.
• Lost quorum → restore from snapshot, rebuild cluster one node at a time.

Next in series: Kafka Partitioned Log — architecture, replication, and exactly-once semantics.