stratum: A System Infrastructure for Massive Agent-Centric ML Workloads
Published in PVLDB 2026, 2026
Recent advances in large language models enable agentic pipeline search, where agents autonomously generate, validate, and optimize machine learning pipelines. This creates thousands of pipeline executions that strain existing Python-based systems such as pandas and scikit-learn, which were designed for human-interactive workflows.
We propose stratum, a unified infrastructure that decouples pipeline execution from planning and reasoning during agentic pipeline search. stratum integrates with common Python libraries, compiles batches of pipelines into optimized execution graphs, and executes them across heterogeneous backends using a novel Rust-based runtime. Preliminary experiments show speedups of up to 16.6x for large-scale agentic pipeline search.
Recommended citation: Arnab Phani, Elias Strauss, Sebastian Schelter. (2026). "stratum: A System Infrastructure for Massive Agent-Centric ML Workloads." PVLDB. 19(11).
Download Paper