Smart open
lastmod 2026-11-10created 2025-04-252 edits
smart_open is a library that lets you read and write large files on remote storage such as Amazon S3, Google Cloud Storage, Azure Blob Storage, HDFS, HTTP, and SFTP in a streaming fashion, using the same interface as Python's built-in open(). It can process files line by line without downloading the whole file to memory or disk, and automatically decompresses or applies compression such as gzip and bz2 based on the file extension. It was developed by the team behind the natural language processing library Gensim and is used in data pipelines to handle local and remote paths with the same code.
Utils for streaming large files (S3, HDFS, gzip, bz2...)
See also
- python
- PyFilesystem2 - Python's Filesystem abstraction layer