Skip to content
Search

Substrait

lastmod 2025-07-02created 2025-07-023 edits

A cross-platform interchange standard for data processing operations

About

  • An open-source format and interface designed to serialize and exchange structured data processing operations (relational algebra, query plans, etc.) between various databases and analytics/query engines
  • Being adopted by various DB, distributed processing, and query engines such as DuckDB, Spark, DataFusion, and Sirius to exchange queries/execution plans and ensure compatibility between different systems
  • Expresses the execution plans of different query languages and engines, such as SQL, DataFrame, and Pandas, in a neutral/standardized format (protocol buffers, YAML, JSON, etc.), simplifying plugin and third-party engine integration and integration with hardware accelerators

What is Substrait?

  • An open-source project that defines the "query plan (Logical/Physical Plan)" of data analysis/processing systems in a standard format
    • Serializes/deserializes the query structures of various languages such as SQL, DataFrame, and LINQ (e.g. SELECT-FROM-WHERE, Join, Aggregation, Projection, Filter) into a standard schema
  • Each database and engine has its own query parser and execution plan, which limits interoperability and the use of hardware accelerators → abstracted into a common standard (intermediate representation)
  • Execution plans from different query languages and frameworks, such as SQL, DataFrame, and Pandas, can be converted to the Substrait format and shared

Key features of Substrait

  • Neutral query representation: can express both logical (logical operators) and physical (execution operators) query plans
  • Serialization based on protocol buffers (mainly using .proto schemas), enabling efficient data exchange across various languages and systems
  • Engine/plugin extensibility: new functions, operators, and types can also be added as plugins
  • Hardware acceleration integration: easy integration at the query plan level with engines for GPUs/FPGAs/dedicated accelerators (Sirius, Velox, NVIDIA RAPIDS, etc.)
  • Cross-platform interoperability: various engines, such as data lakes, distributed processing, and local analytics, can accept and execute the same query plan

Why does Substrait matter?

  • If each engine supports only Substrait instead of building its own custom interface, ecosystem integration becomes immediately possible
  • Serves as a "common language for query/analytics execution plans" between data engines and platforms
    • Query in a DB → Substrait plan → passed to and executed on GPU/distributed engines, separate accelerators, etc.
    • Connects engines at the "execution plan level" without needing to convert/rewrite queries
  • Even when a new query engine appears, it can easily be swapped in/combined as a "plugin" (flexibility to switch engines)
  • Unifies various query languages/environments such as SQL, Python, and distributed queries
  • Allows running a mix of different engines in cluster/distributed environments (supports heterogeneous environments)
  • Provides a text version, making it easy to analyze query plans or integrate with visualization tools

Real-world examples

  • Exchanging query plans between SQL parsers and execution engines (Calcite → Arrow, DuckDB → Sirius, etc.)
    • DuckDB: supports exporting internal query plans to Substrait and importing Substrait queries from outside
    • Sirius: takes Substrait query plans generated by DuckDB as is and can run SQL on the GPU, with no query parsing/modification needed
  • Running DataFrame operations (Pandas, etc.) on an external database
  • Building third-party query plan visualization tools (D3, Web, etc.)
  • Velox, Spark, DataFusion, etc.: export each engine's query plans to Substrait to integrate with various analytics and acceleration engines

How is it structured technically?

  • Substrait plans are mainly defined with protocol buffers (.proto)
    • Composed of LogicalPlan, PhysicalPlan, Rel, Expression, Function, etc.
    • Clear schemas for each operator, such as Filter, Join, Aggregate, Project, and Sort
  • Extensible function/type catalog and support for UDF (User-Defined Function) extensions

Conclusion

  • Substrait is the "query execution plan standard" for the database and analytics platform ecosystem, key to interoperability between engines, hardware acceleration, and data ecosystem scalability
  • It is becoming a foundational standard that simplifies hardware accelerator integration and query engine integration, making it easier to build high-performance analytics infrastructure without complex query parsing/conversion

See also

Favorite site

author Zer0 sitemap