StreamFusion

Skip to content

Initializing search

datafusion-contrib/StreamFusion

Blogs

Connectors

Operators

Top-N

LIMIT

Event-time sort

Watermark assigner

Unsupported operators

Backends

Deployment

Releasing

Configuration

Alpha API stability

Memory management

Native metric parity

Upstream Flink suite

Benchmarks

Optimizations

StreamFusion¶

Note

StreamFusion is not part of Apache Flink or Apache DataFusion.

StreamFusion runs Apache Flink SQL faster by executing supported operators natively — Rust and<br>Apache Arrow/DataFusion, invoked over JNI — while Flink continues to own planning, coordination,<br>and everything not yet supported. A query is planned by Flink as usual; every part we can<br>reproduce exactly is swapped for a native implementation, and anything else keeps running on<br>stock Flink. Substitution is transparent (no query rewriting) and conservative (a single<br>unsupported operator falls the whole query back), so a job's correctness never depends on how much<br>of it accelerated.

The impact&para;

Flink's per-record, row-at-a-time execution model spends most of its CPU on interpretation<br>overhead — boxed objects, virtual dispatch, one hash/compare per field — rather than the actual<br>computation. Moving the same operators to vectorized Rust kernels over Arrow column batches turns<br>that per-record overhead into per-batch overhead: on the Nexmark benchmarks,<br>accelerated queries run 1.2–3× the throughput of stock Flink on the same hardware , with some<br>stateful shapes over 9×. In a deployment paying for compute by the core, that throughput<br>translates directly into fewer task managers for the same job, or the same task managers handling<br>more jobs.

How it works&para;

Connectors get data in and out — currently Kafka, at production<br>quality, across every wire format Flink itself supports.

Operators are where the acceleration happens — the per-operator pages<br>mark exactly what's native, what's partial, and what still falls back, with the precise<br>condition in each case.

Backends hold state for the stateful operators — an in-memory backend<br>by default, with a Rust-owned RocksDB backend for persistent state and incremental checkpoints.

Deployment and Configuration cover installing<br>StreamFusion into a Flink cluster and the runtime flags that control it.

Inspiration&para;

StreamFusion is built by porting established engines rather than reinventing operators:

DataFusion Comet — the model for the whole<br>project (a native columnar accelerator behind an unchanged SQL planner) and the reference for<br>the JNI / Arrow C Data Interface bridge, off-heap memory accounting, and fallback-reason<br>reporting.

Arroyo — the streaming-operator implementations<br>we port (it already runs on DataFusion); the reference for join/window/changelog logic.

Apache DataFusion — the native execution and<br>expression engine underneath (hash joins, aggregates, Arrow kernels).

RisingWave — the reference for changelog<br>semantics and memcomparable arrow-row state encoding.

Apache Flink — the parity target : every operator is a<br>faithful port of Flink's own, verified for identical output by a parity harness.

Determinism&para;

Results are byte-identical to stock Flink for everything admitted, with one necessary exception:<br>an inherently non-deterministic function (PROCTIME(), NOW(), random) has no well-defined<br>"correct" value to match, since Flink's own output for these depends on wall-clock and execution<br>timing. StreamFusion uses its own reasonable implementation for these rather than chasing an<br>undefined target, and does not gate or refuse a query for observing one. Everything whose result<br>is deterministic — including an operator that merely orders by processing time, such as a<br>proctime dedup or OVER — still produces output identical to Flink, because that depends only on<br>arrival order, not the clock value.

Related work&para;

Three native Flink accelerators exist, all closed source :

Flash (Alibaba Cloud) — a C++ native + SIMD vectorized engine with a custom state backend<br>(ForStDB). Stateful, production-deployed at scale; claims 5–10× on streaming Nexmark, 3×+ on<br>batch TPC-DS, and ~50% cost reduction across 100k+ compute units. Proprietary, on Alibaba Cloud.<br>(blog)

Vera X (Ververica, the original Flink creators) — a proprietary native vectorized engine<br>with a drop-in compatibility layer and a new state store. Stateful; claims 5–10× on Nexmark SQL<br>and ~52% lower resource usage. Implementation undisclosed.<br>(blog)

Iron Vector (Irontools) — the same stack as us (Rust + Arrow + DataFusion over zero-copy<br>JNI, Substrait plan serialization, transparent fallback), but stateless only today<br>(projections, filters, expressions); windows, joins, and exactly-once are described as planned.<br>Claims ~97% higher throughput on a stateless ETL pipeline.<br>(blog)

Where StreamFusion differs: it is open source , and every substitution is gated and verified<br>for identical results against stock Flink by a parity harness rather than asserted. It is...