Skip to content
Search

WarpStream

lastmod 2023-08-11created 2023-08-111 edits

Kafka is dead, long live Kafka!

Features

  • Kafka is dead, long live Kafka - WarpStream
  • Introducing WarpStream, a Kafka protocol-compatible data streaming platform built directly on top of S3.
  • WarpStream ships as a single stateless Go binary, eliminating the need to manage local disks, rebalance brokers, and operate ZooKeeper.
  • By streaming data directly to S3, the platform greatly reduces infrastructure costs and is 5-10x cheaper than Kafka in the cloud.
  • Criticizes Kafka's fit for modern workloads
    • Economics: by design, Kafka's replication strategy incurs huge inter-AZ bandwidth costs
    • Operational overhead: running a Kafka cluster requires a dedicated team and complex custom tooling
  • WarpStream's architecture differs from Kafka's. Instead of brokers, there are stateless "agents" that can act as the "leader" for any topic, commit offsets for any consumer group, or act as the coordinator for the cluster.
  • In WarpStream, all storage is offloaded to object storage such as S3, enabling easy scaling and fast failure recovery.
  • WarpStream separates data from metadata and stores the metadata for every "virtual cluster" in a custom metadata database.
  • The platform cuts the total cost of most Kafka workloads by 5-10x, but has higher latency: a P99 of about 400ms for produce requests and about 1 second from producer to consumer.
  • WarpStream is currently in developer preview and is not yet ready for production use.
  • WarpStream's creators think Kafka's developer UX is a problem, pointing in particular to the low-level abstraction of partitions. They plan to address this in future WarpStream updates.
  • WarpStream runs directly on object storage from multiple clouds (AWS S3, GCP GCS, Azure Blob Storage)

Hacker News comments

  • Kafka is dead, long live Kafka | Hacker News
  • An article on the dichotomous nature of Kafka, a data streaming technology
  • Debate over whether most tech companies use Kafka
  • The cost efficiency of pushing each message directly to S3, and the problem of running a Kafka cluster in each AZ
  • An introduction by Ryan Worl, co-founder and CTO of WarpStream, a Kafka protocol-compatible streaming system built directly on top of S3
  • Emphasis on WarpStream's cost efficiency, no need to operate stateful disks/nodes, no need for data rebalancing or ZooKeeper, and reduced cross-AZ bandwidth charges
  • Criticism of the cost of running Kafka on separate VMs at cloud providers
  • Discussion of using storage adapters in properly designed Hadoop/Kafka cloud managed services to take advantage of the provider's redundancy
  • Some users' complaints about the article's claim that Kafka requires a team of experts and a large budget
  • Emphasis on the fact that the number of partitions can be changed in Kafka
  • Debate over the article's claim that operating Kafka requires a large team of engineers
  • Questions about how WarpStream manages its service, whether it uses cloud providers or bare metal, and whether it uses foundationdb for its metadata store
  • Discussion of the potential of Kafka's API and the possibility of abstracting away the complexity of cluster management
  • Cost savings from moving large-scale ML traffic to S3, with one user reporting about 90% cost savings
  • A suggestion to change the article title to "Kafka is dead. Long live Warpstream." to reflect the arrival of the new technology

See also

Favorite site

author Zer0 sitemap