Luigi
lastmod 2026-11-10created 2024-06-042 edits
Luigi is a Python-based batch job pipeline framework released by Spotify. Each job is defined as a Task class; once dependencies are declared with requires() and outputs with output(), it follows the dependency graph and runs only the outputs that have not yet been produced. It can resume from the point of failure and comes with integration modules for Hadoop, Spark, and databases, so it was used for data processing pipelines; because its scheduling features are weak, it is often compared with Airflow.