Skip to content
Search

Kafka

lastmod 2026-04-17created 2018-07-3124 edits

Apache Kafka is an open-source message broker project developed in Scala by the Apache Software Foundation. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

Categories

  • ktea - Kafka terminal client
  • Kafbat UI - Open-Source Web UI for managing Apache Kafka clusters

Purpose

Let's look at the means by which Kafka met LinkedIn's requirements.

Requirements

  • Process in real time with high throughput.
  • Read data at arbitrary timing.
  • Integrate easily with various products and systems.
  • Never lose messages.

Means of realization

  • Messaging model and scale-out architecture
  • Data persistence to disk
  • Easy-to-understand API
  • Delivery guarantees

Message Queues

Why is Kafka fast?

  • Why is Kafka fast? | GeekNews
    • [Original] (java) Why is Kafka fast? | frogred8
      • (Comment by @vwjdalsgkv) The post seems slightly confused about nio: nio is not non-blocking io but new io. It supports both blocking and non-blocking.
  • General reasons Kafka is fast
    • Uses low-latency I/O (RAM)
    • Uses a sequential I/O data structure (log)
    • Applies zero-copy
    • Horizontally scaling system
    • Data compression and batch processing
  • What is zero-copy?
    • An OS-supported interface that reduces the overhead of copying a file to a socket
    • The sendfile command on Linux
  • So how did zero-copy make Kafka faster?
    • Java added the transferTo function to the nio package
    • Using it, Kafka sends messages straight from kernel space to the network without bringing them into user space, which gives a large speed advantage
    • In a benchmark of the traditional approach versus transferTo, the latter was 65% faster

If we could rebuild Kafka from scratch

  • What if we could rebuild Kafka from scratch? | GeekNews
  • KIP-1150 (diskless Kafka) and AutoMQ's Kafka fork have sparked active discussion of cloud-optimized Kafka
  • Proposes that a rebuilt Kafka drop the existing partition structure in favor of a key-centric approach
  • Concurrency control and broker-sided schema support are needed
  • Also stresses the need to integrate modern system features such as extensibility, snapshots, and multi-tenancy
  • Imagines a truly cloud-native event log system built on Kafka that goes beyond Kafka's existing limits

What if we rebuilt Kafka?

  • KIP-1150 (diskless Kafka) and AutoMQ's Kafka fork were recently announced; they integrate Kafka with object storage such as S3 to improve elasticity and cut costs in cloud environments
  • Proposes various improvements while imagining a cloud-native event log system that goes beyond Kafka's existing limits

Proposal for a partitionless structure

  • Because cloud object storage behaves like infinite storage, the need for topic partitions decreases
  • In many cases only global message order, or the order of messages with the same key, matters
  • The partition concept can be hidden from users to offer simpler usability

Key-centric approach

  • Proposes a design that allows fast access to and replay of all messages for a given key, rather than partition scans
  • Supporting millions of entity-level streams lets the number of consumers scale flexibly with demand
  • Failures are isolated per key, increasing overall system processing efficiency
  • An ideal structure for event sourcing or actor model systems

Topic hierarchy support

  • Like systems such as Solace, part of the message payload should be promoted into a structured, path-like topic identifier to enable pattern-based subscriptions
  • The broker can then support subscription filtering efficiently without parsing the whole message

Concurrency control

  • Kafka currently has no way to prevent concurrency conflicts on write
  • Supporting optimistic locking per key would make it possible to verify that a message was written after seeing the latest state
  • Prevents lost updates

Broker-sided schema support

  • Kafka currently treats messages as plain byte arrays and relies on an external schema registry
  • Proposes native broker-level support for schema information such as AsyncAPI metadata to ensure schema consistency
  • This would also make it easy to integrate with open table formats such as Apache Iceberg

Extensibility and plugin architecture

  • Proposes an extensible, pluggable architecture like Postgres or Kubernetes
  • Broker-side filters and transformations should be easy to implement without protocol-aware proxies (such as Kroxylicious)
  • Rate limiting, topic encryption, Iceberg table-based backends, and so on should be implementable as plugins

Synchronous commit callbacks

  • Kafka currently guarantees only eventual consistency
  • Proposes a structure where a producer, after sending a message, can immediately check whether the derived data has been updated
  • Supporting read-your-own-writes would let Kafka be used as a true database log

Snapshots

  • Kafka's current compaction keeps only the last message, which works only when that message contains the full state
  • When only changes are recorded, all events for a key must be replayed, which takes longer
  • A logical compaction feature that summarizes events per key into a snapshot is needed

Built-in multi-tenancy

  • Every modern data system should treat multi-tenancy as a default consideration
  • New tenant environments should be cheap and instant to create, with resources, security, and access control strictly isolated

Other notes

  • Some features are already supported by systems such as S2 (high-cardinality streams), Waltz (optimistic locking), and Apache Pulsar (multi-tenancy)
  • However, no open-source system supports all the proposed features at once
  • The post is the personal view of an author at Confluent, not an official position
  • Mentions that, in theory, an LSM tree-based architecture would be a strong choice

See also

Favorite site

author Zer0 sitemap