Skip to content
Search

Lite3

lastmod 2025-12-14created 2025-12-141 edits

A JSON-compatible zero-copy serialization format

About

  • A Zero-Copy binary serialization format developed to remove the boundary between data parsing and serialization
  • Encodes data as a B-tree structure within a single contiguous buffer, allowing arbitrary field access and modification in O(log n) time
  • The in-memory format works as the wire format, so data can be sent and received immediately without separate parsing or re-serialization
    • Key lookup, value modification, and sending on receipt are all handled with memory copies (memcpy) only
  • Schemaless (self-describing) structure, usable without an IDL or schema definition
    • Supports conversion to and from JSON, ensuring compatibility with existing APIs and datasets
    • Can be output as JSON for debugging and message inspection
  • Performance-focused design
    • Made up of a cache-friendly B-tree structure and a minimal 9.3 kB C implementation
    • Up to 120x faster than the fastest JSON libraries using SIMD, and up to 242x faster than Google Flatbuffers
    • One of the fastest schemaless data formats in the world
  • Technical features
    • Supports zero-copy reads/writes, O(log n) complexity for all I/O operations
    • Built-in pointer validation and runtime type safety
    • Does not use malloc(); the caller provides the buffer directly
    • Predictable latency and low memory usage
    • Based on C11 (GNU C syntax), no external dependencies
    • Optionally uses yyjson as a sub-dependency to support JSON conversion
  • API structure
    • Buffer API: based on user-provided buffers, maximizes memory control
    • Context API: abstracts memory management, provides a simple interface
  • Comparison and benchmark results
    • Overwhelming processing speed compared with text formats such as JSON, BSON, and MessagePack
    • Offers more flexibility and modifiability than binary formats such as Cap’n Proto and Flatbuffers
    • Recorded top performance on real Twitter API data, Kostya JSON, C++ serialization benchmarks, and more
  • Security design
    • Includes overflow-protected bounds checks, generational pointer macros, recursion limits, and runtime type validation
    • Structure suited to processing untrusted input data
  • Use contexts
    • Maximizes CPU efficiency in high-performance network messaging, real-time data processing, embedded environments, and more
    • Can be applied immediately in systems based on C interfaces
  • License and extensibility
    • Released under the MIT license, a standalone C implementation
    • Further roadmap includes language bindings, compression benchmarks, and writing an official spec
  • Origin and background
    • A project independently reimplemented based on concepts proposed in the Lite² paper published in 2024
    • The name Lite³ means it is even lighter than Lite²

See also

Favorite site

author Zer0 sitemap