Tuplex

lastmod 2021-07-10created 2021-07-092 edits

Parallel big data processing framework

Tuplex is a parallel big data processing framework that runs data science pipelines written in Python at the speed of compiled code. Tuplex has a Python API similar to Apache Spark or Dask, but rather than invoking the Python interpreter, Tuplex generates optimized LLVM bytecode for the given pipeline and input data set.

Features

• Provides a Python API similar to Apache Spark / Dask, but
  • Does not invoke the Python interpreter
  • Generates LLVM bytecode optimized for the given pipeline and input data set
  • 5–91x faster than the interpreter
• Internally based on data-driven compilation and dual-mode processing, reaching speeds similar to pipelines hand-coded and optimized in C++
• Supports MacOS / Linux
• Paper "Tuplex: Data Science in Python at Native Code Speed" presented at SIGMOD '21

See also

• Big data
• python

Favorite site

• tuplex web site
• Github - tuplex