Dask
lastmod 2026-11-10created 2021-07-092 edits
Dask is a Python library for parallel processing of large data that does not fit in memory. With dask.array and dask.dataframe, which share the interfaces of NumPy arrays and pandas DataFrames, it splits data for processing with almost no changes to existing code, and it builds work into a lazily evaluated task graph that runs on multiple cores or a cluster. Unlike the JVM-based Apache Spark, it is distinguished by running within the Python ecosystem.