Jaber on X: "i open-sourced bonsai-turbo -- a batch-1 decode engine that runs @PrismML's Bonsai 27B 1.76x faster than the official llama.cpp fork. same outputs, token for token
H100, tg128, greedy: ternary 85.5 >> 151 tok/s. 1-bit 90.1 >> 159 tok/s. logit parity with the fork on 32 of 32 https://t.co/i0WU3jfrC9" / X<br>Post
Log inSign up
Post
Jaber
@Akashi203
i open-sourced bonsai-turbo -- a batch-1 decode engine that runs @PrismML's Bonsai 27B 1.76x faster than the official llama.cpp fork. same outputs, token for token
H100, tg128, greedy: ternary 85.5 >> 151 tok/s. 1-bit 90.1 >> 159 tok/s. logit parity with the fork on 32 of 32 test prompts, gated before any speed number counts. not a lossy trick
why it's faster: at batch-1 the GPU isn't math-bound or bandwidth-bound, it's overhead-bound. the stock path executes 3703 GPU ops per token and spends ~97% of its time on that op overhead. bonsai-turbo fuses the whole per-token pass into a handful of large ops. --mega mode compiles the entire 64-layer token step -- embed >> layers >> logits >> next token -- into one cooperative kernel
the kernels were generated by our internal agent -- the same agent that powers @runinfrai
scope is deliberate: batch-1 decode only. Bonsai 27B is the ternary 27B small enough to run on a phone, and this makes the local single-user experience actually fast. it is not a batched-serving engine
roofline says ~440-490 tok/s is on the table. next: cp.async weight pipelining, then a speculative drafter -- targeting ~300
github.com/RightNow-AI/bo…
span:not(:empty)~span:not(:empty)]:before:content-['·'] [&>span:not(:empty)~span:not(:empty)]:before:px-1 [&>span:not(:empty)~span:not(:empty)]:before:shrink-0">12:34 AM · Jul 16, 202659Views
:host{display:inline-block;direction:ltr;white-space:nowrap;line-height:1}span{display:inline-block}:host([data-will-change]) span{will-change:transform}.number,.digit{padding:round(nearest, calc(var(--number-flow-mask-height, 0.25em) / 2), 1px) 0}.symbol{white-space:pre}2:where(number-flow-react){line-height:1}number-flow-react > span{font-kerning:none;display:inline-block;padding:calc(round(nearest, calc(var(--number-flow-mask-height, 0.25em) / 2), 1px) * 2) 0}2<br>:host{display:inline-block;direction:ltr;white-space:nowrap;line-height:1}span{display:inline-block}:host([data-will-change]) span{will-change:transform}.number,.digit{padding:round(nearest, calc(var(--number-flow-mask-height, 0.25em) / 2), 1px) 0}.symbol{white-space:pre}4:where(number-flow-react){line-height:1}number-flow-react > span{font-kerning:none;display:inline-block;padding:calc(round(nearest, calc(var(--number-flow-mask-height, 0.25em) / 2), 1px) * 2) 0}4<br>:host{display:inline-block;direction:ltr;white-space:nowrap;line-height:1}span{display:inline-block}:host([data-will-change]) span{will-change:transform}.number,.digit{padding:round(nearest, calc(var(--number-flow-mask-height, 0.25em) / 2), 1px) 0}.symbol{white-space:pre}4:where(number-flow-react){line-height:1}number-flow-react > span{font-kerning:none;display:inline-block;padding:calc(round(nearest, calc(var(--number-flow-mask-height, 0.25em) / 2), 1px) * 2) 0}4
New to X?<br>Sign up now to get your own personalized timeline!<br>Sign up with GoogleSign up with AppleCreate account<br>By signing up, you agree to the Terms of Service and Privacy Policy, including Cookie Use.
Relevant people<br>Jaber@Akashi203Follow
Trending now
Don't miss what's happening<br>People on X are the first to know.
Log inSign up