Why - Frisky

Skip to content

Initializing search

Deploy

Dask

Agents

License

History

Why Frisky

You don't need Frisky.

Frisky is a fast version of Dask and most people don't need Dask. Parallel computing is for when you're too lazy to solve a problem properly, so you bludgeon the problem with hardware. You're better off using a database, or Polars, or asking your LLM to craft you something custom. Heck, you could even use Spark.

Frisky exists because I wanted to see how fast I could make a Dask-like thing today. Frisky is an art project. A very fast art project.

But Frisky is fun, and that's honestly the best reason I can find to use it, oh and that agents seem to enjoy using it. You're probably here for speed though, so let's talk about speed first.

Speed! Rust!

AI!

New Dashboard!

Speed Again! Fast Disk and Network!

Graph Construction!

Lightweight!

Moving Fast! Breaking Things!

Speed! Rust!

Is it still cool to reimplement libraries in Rust? I hope so, because gosh-darn-it that's exactly what I did.

Frisky is written in Rust (🎉) Frisky's core is roughly 100x faster than the core of Dask. That's not to say that your code will be 100x faster (almost certainly not) but you'll no longer be able to blame task scheduling.

The Frisky scheduler can run around 250,000-400,000 tasks per second , or<br>with an overhead of roughly 3 µs .

"$schema": "https://vega.github.io/schema/vega-lite/v5.json",<br>"title": "Scheduler throughput (higher is better)",<br>"data": {"values": [<br>{"scheduler": "Frisky", "tasks_per_s": 300000, "grp": "Frisky"},<br>{"scheduler": "Dask", "tasks_per_s": 3000, "grp": "other"}<br>]},<br>"height": 220,<br>"mark": "bar",<br>"encoding": {<br>"x": {"field": "scheduler", "type": "nominal", "title": null, "axis": {"labelAngle": 0}},<br>"y": {"field": "tasks_per_s", "type": "quantitative", "title": "tasks / second"},<br>"color": {"field": "grp", "type": "nominal", "legend": null,<br>"scale": {"domain": ["Frisky", "other"], "range": ["#e3a008", "#9aa0a6"]}},<br>"tooltip": [<br>{"field": "scheduler", "type": "nominal", "title": "Scheduler"},<br>{"field": "tasks_per_s", "type": "quantitative", "title": "Tasks / second", "format": ","}

Measured

Frisky: an offline Rust harness on an Apple M4 drives the<br>real scheduler state machine with simulated worker completions, achieving 2.5-4.5<br>µs/task. In our matching end-to-end benchmark, Dask ran at ~2,600-3,000<br>tasks/s.

AI!

Even if Rust has lost its lustre, AI is surely in vogue (at least according to my LinkedIn feed), and agents love Frisky.

The biggest benefit to using Rust isn't speed, it's that we measure every detail of the calculation guilt-free, generating mountains of valuable telemetry and context. Frisky measures these mountains of telemetry and feeds them into downstream analytics, which in turn enable an AI development cycle at the heart of Frisky's progress.

You point your favorite AI agent to a Frisky dashboard and the frisky CLI and it pulls in all the context it wants:

$ frisky --help # Agents read this<br>$ frisky observe overview YOUR_DASHBOARD_URL # Then they do this

Overview http://localhost:8787<br>state workers 6 (0 idle) waiting 34 processing 1347 memory 7684 erred 0 queued 0<br>→ stuck? observe blocked

perf wall-clock 14.0s workers 6 tasks 14683 spans 200000<br>memory 0.5 GB / 17.2 GB (3% peak) spilled 0.0 GB unspilled 0.0 GB network recv 1.0 GB<br>→ per-worker memory observe workers prefix costs observe prefixes http://localhost:8787 in-flight transfers observe transfers

Components<br>│0s 2 4 6 8 10 12 14s│ Total<br>├──────────────────────────────────────────────────────────────────────┤<br>compute │▇▇██████████████████████████████████▃█████████████████████████████ │ 275.6 s<br>network │▁▂▂▃▂▄▇▂▂▂▂▂▂▃▂▅▄▂▂▂▂▂▂▂▂▃▂▂▂▂▂▂▂▂▂▂▂██▇██▇▇▇▄▁▁▁▁▁▂▁▁▁▁▁▂▁▁▁▁▂▁▁▁ │ 57.9 s<br>scheduler │▁▁▁▁▂▂▂▁▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▁▁│ 3.0 s<br>other │▂▂▂▃▄▅▄▂▂▂▂▂▂▂▂▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▃▃▄▃▃▂▃▃▃▂▂▂▂▂▃▃▃▂▃▃▃▂▂▂▃▃▃▂▃▂▂▂▂│ 94.7 s<br>└──────────────────────────────────────────────────────────────────────┘<br>→ zoom / full view: observe timeline http://localhost:8787 --view component

Costliest span types — over time<br>│0s 2 4 6 8 10 12 13s│ Total<br>├──────────────────────────────────────────────────────────────────────┤<br>worker.exec.call │▇▇████████████████████████████████████▇█▇▇▇███████████████████████████│ 268.6 s<br>tcp.send.queue │▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▇▇▃▃▆▆▅▅▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁│ 17.2 s<br>tcp.send.write_queue │▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▆▃▃▅▆▄▄▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁│ 8.5 s<br>tcp.recv.queue │▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▆▆▁▃▆▅▄▄▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁│ 8.5 s<br>tcp.send.compress │▁▁▁▁▁▂▃▂▁▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▆▆▄▄▅▄▃▃▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁│ 7.7 s<br>tcp.send.serialize │▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▅▃▂▃▄▃▄▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁│ 6.2 s<br>worker.transfer.recv │▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▄▂▂▄▄▄▅▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁│ 5.5 s<br>worker.deserialize │▁▁▁▁▂▂▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▄▅▃▂▃▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▁▁│ 4.4...