Walrus
lastmod 2025-12-26created 2025-12-261 edits
Distributed message streaming engine
About
- High-performance open-source alternative to Kafka written in Rust
- A distributed message streaming platform built on a high-performance log storage engine, providing reliable streaming and automatic leader rotation
- Maintains metadata consistency and supports failure recovery via the Raft consensus protocol
- Segment-based partitioning distributes load automatically, and leadership rotates in round-robin fashion
- Key features
- Topics are split by segment-level sharding, and a leader node per segment handles writes
- Lease-based write fencing allows only a single leader to write, preventing split brain
- Automatic rollover proposes a metadata change and performs leader rotation when a segment exceeds its size
- Sealed Segment Reads allow reading historical data directly from replicas
- High-performance I/O based on io_uring achieves high throughput on Linux
- Uses a TCP-based length-prefixed text protocol
- REGISTER, PUT, GET, STATE, METRICS commands support creating topics, sending/receiving messages, and querying state
- Simple command structure allows connecting to any node, with automatic forwarding
- Write throughput: stable performance thanks to a single writer per segment
- Read throughput: scales with the number of replicas
- Latency: around 1~2 RTT on average
- Minimal consensus overhead: Raft is applied only to metadata, not the data path
- Higher write throughput and bandwidth than Kafka and RocksDB
- Over 1.2 million operations per second with fsync disabled, and similarly stable performance even with it enabled