Apache Arrow

lastmod 2024-10-17created 2023-04-065 edits

Apache Arrow is a language-agnostic software framework for developing data analytics applications that process columnar data.

About

• (Spark) What is Apache Arrow? (On Zero-Copy serialization)

One of the problems when different data infrastructures use APIs to share data with each other is that the overhead of serialization and deserialization is too high. This creates a bottleneck in application performance.

Arrow defines data in a columnar structure in memory regardless of language or platform, so that the CPU and GPU can read and write memory quickly.

PyArrow

• Python — Apache Arrow v16.0.0

PyArrow - Apache Arrow Python bindings

See also

• Perspective
• Apache
• pandas
• Vortex - High-performance Columnar file format. For handling compressed Apache Arrow arrays.

Features

• Apache Arrow | Apache Arrow
• Apache Arrow - Wikipedia