Skip to content
Search

UnisonDB

lastmod 2025-11-20created 2025-11-201 edits

A log-based real-time database

About

  • A high-performance sync engine that behaves like a message bus for AI and edge computing
  • Log-native real-time database : designed to solve data consistency and real-time responsiveness problems in edge AI and edge computing environments
  • Combines a B+Tree storage engine with WAL (Write-Ahead Logging)-based streaming replication, guaranteeing sub-second replication and strong consistency across hundreds of nodes
  • Multi-model structure supporting Key-Value, Wide-Column, and Large Object (LOB) storage
  • ZeroMQ-based real-time notifications provide millisecond-level change detection
  • Edge-First design optimized for local-first architectures
  • Multi-tenancy support through namespace isolation
  • Differences from existing systems:
    • LMDB/BoltDB are fast local stores but cannot replicate
    • etcd/Consul offer high consistency but limited scalability
    • Kafka/NATS are strong at streaming but cannot be queried
    • UnisonDB fills this gap, providing a single log-centric structure that unifies storage and streaming
  • Core architecture: 3-layer structure
    • 1. WALFS(Write-Ahead Log File System)* – mmap-based log file system, optimized for large-scale reads/writes
      • Segment-based log structure, optimized for both sequential writes and random reads
      • Supports zero-copy reads, offset-based seeking, and real-time tailing
      • Parallel reader structure allowing multiple replica nodes to read concurrently
    • 2. Engine – hybrid storage combining WAL, MemTable, and B-Tree
      • Combines a MemTable (skip list) and a B-Tree index on top of WALFS
      • Uses FlatBuffers so data can be transferred during replication without deserialization
      • Supports atomic multi-key transactions, guaranteeing consistency per commit
      • LOBs (large objects) can be chunked and streamed per transaction
      • Wide-Column model supports partial column updates and dynamic schema extension
    • 3. Replication – WAL-based streaming replication, including offset tracking
      • With WAL-based streaming replication, followers track offsets and sync in real time
      • Uses FlatBuffer log records to maintain a self-describing data structure
      • Efficient streaming implemented with batch transfers
      • About 2x faster reads than BoltDB, consistency-focused design
  • How UnisonDB solves it
    • Append-only log + B-Tree combination provides fast writes and efficient range reads
    • Supports transaction-based multi-key replication and column-aware sync
    • Built-in replication via gRPC WAL streaming + B-Tree snapshots
    • A structure without compaction secures predictable latency and high throughput
  • License: Apache License 2.0
  • Language: Go
  • Suitable use cases: edge AI, distributed IoT, local-first data processing, real-time analytics systems

See also

Favorite site

author Zer0 sitemap