Show HN: I run 30B 22tok/s, 109tok/s not novel,6GB/16GB RAM overcoming llama.cpp

federicoTXTS1 pts0 comments

Democratisation of local AI is key. I ve been working on pushing the limits of commercial hardware, squeezing any extra bit possible. My Scientific Agentic AI hareness helped me to reallocate every single bit of it. I rewrote the Kernel, I went down the CUDA rabbit hole until I have been able to explain any bit and any ms of computational power involved in the process pushing the Qwen 30B-A3B from 8 tok7s to 19 tok/s with llama.cpp up to 22.2 tok/s with my project and 109 tok/s on not novel content and speeding up the prefill by 5-9X

novel llama pushing show 22tok 109tok

Related Articles