TurboPrefill | Devpost

We've detected that you are using an unsupported browser. Please<br>upgrade your browser<br>to Internet Explorer 10 or higher.

Standard pipeline versus TurboPrefill microbatch scheduling across multiple GPUs. Затем нажимайте Save & continue.

Inspiration

Large language models running across multiple GPUs are often limited not only by computation, but by communication between GPUs. This becomes especially visible on systems without NVLink or with slow PCIe connections.

TurboPrefill by Trykhlieb was created to reduce these bottlenecks during the prefill stage.

What it does

TurboPrefill changes the scheduling of prompt microbatches across multiple GPUs.

Instead of processing each microbatch through all GPUs sequentially, it pipelines microbatches so that several GPUs can perform useful work at the same time. Decode behavior remains unchanged.

The original implementation was developed for llama.cpp and tested on different GPU generations, large models, and systems with very limited inter-GPU bandwidth.

What I built during OpenAI Build Week

During OpenAI Build Week, I created an experimental port of TurboPrefill to ik_llama.cpp by Ivan Kawrakow.

The port is based on ik_llama.cpp commit 6198a356.

The published materials include:

modified source files

clean and diagnostic versions

benchmark scripts

test contexts

baseline and TurboPrefill benchmark results

How Codex and GPT-5.6 were used

Codex and GPT-5.6 assisted with the initial port, code adaptation, diagnostic logging, execution-flow analysis, error localization, cleanup, and preparation of reusable diagnostic files.

Codex did not complete the entire task automatically, but it significantly accelerated the development and debugging process.

Results

The experimental ik_llama.cpp port shows higher prefill performance across all tested context sizes.

For GPT-OSS-20B, at a context length of 1,024 tokens, prefill performance increased from approximately 184 to 602 tokens per second.

The main TurboPrefill implementation for llama.cpp has also demonstrated substantial improvements on multiple GPU configurations, especially where inter-GPU communication is the primary bottleneck.

Challenges

The main challenge was preserving correct execution while changing microbatch scheduling inside a complex multi-GPU inference engine.

Extended logging was used to trace execution, identify porting errors, and verify the modified implementation.

What I learned

The work confirmed that pipeline scheduling can substantially improve multi-GPU prefill performance, particularly on heterogeneous or bandwidth-limited systems.

It also demonstrated how Codex can accelerate low-level systems work when combined with direct testing, benchmarking, and manual engineering analysis.

Built With

c++<br>codex<br>cuda<br>gpt-5.6<br>ik-llama.cpp<br>linux<br>llama.cpp<br>llm<br>multi-gpu<br>python

Submitted to

OpenAI Build Week

Created by

Serhii Trykhlieb

Industrial Systems Architect specializing in the design, manufacturing, commissioning, and performance analysis of custom equipment

Like

1 person likes this:

Share this project:

Updates

Serhii Trykhlieb<br>started this project

Jul 21, 2026 04:29 PM EDT

Leave feedback in the comments!

Log in<br>or<br>sign up for Devpost<br>to join the conversation.