Carquet
lastmod 2026-01-19created 2026-01-191 edits
A high-performance pure C library for reading and writing Parquet files
About
- A production-grade pure C implementation library developed to support the Apache Parquet format directly in C environments
- Written to the C11 standard, it works without a C++ runtime or exception handling and uses only zstd and zlib as external dependencies
- Suited to data processing in constrained environments such as embedded systems, IoT, microcontrollers, and legacy environments
- Binary size of about 200KB, a lightweight build more than 50MB smaller than Apache Arrow
- Supports various architectures with SIMD optimizations (SSE4.2, AVX2, AVX-512, NEON, SVE) and runtime auto-detection
- Supports all physical types (BOOLEAN, INT32, DOUBLE, etc.) and various encodings and compression codecs (ZSTD, LZ4, SNAPPY, GZIP)
- Handles byte order correctly on Big-Endian systems as well (s390x, SPARC, PowerPC, etc.)
- Includes production features such as Nullable columns, basic nested schemas, CRC32 page verification, column statistics, memory-mapped I/O, and OpenMP parallel reads
- The streaming API can process large files without loading them entirely into memory
- Fully compatible with PyArrow, with mutual read/write support with Python environments
- Compared with Apache Arrow
- On ARM, writes are 1.5-5x faster and reads are up to 1.3x faster
- On x86, reads are somewhat slower, but files are about 1.4x smaller
- Main strengths are minimal dependencies, a simpler build, and integration with C-only environments
- Complex nested types, encryption, and multithreaded ZSTD decompression are not supported