Rust Vs C++ - LLM coded model conversion from PyTorch

We do many conversions of AI models from the reference PyTorch<br>implementation to GGML and C++. The reason is GGML/C++ produces a<br>relatively tiny package we can run almost anywhere. The performance<br>usually matches or exceeds PyTorch with a fraction of the<br>dependencies.

Personally I despise C++; can’t stand it. I don’t have to write<br>it anymore though, LLMs do it. I sit above a layer of abstraction<br>and only descend into the code to avoid worst case scenarios in much<br>the same way I would randomly go read the code for a library I<br>depended on before AI.

Someone asked if we had considered doing these ports in Rust. I<br>am well aware of Rust, got excited about it in the past, but had not<br>seriously considered it because GGML is written in C++.

In recent years I experimented by comparing Rust against Go and<br>it was a slam dunk victory for Go on the project I was doing.<br>Neither myself, LLMs or the person advocating for Rust seemed to<br>understand how to solve basic problems without tripping over the<br>type system.

However I do like to explore options and it had occurred to me<br>already that an experiment between Rust and C++ could be<br>interesting, so having someone ask for it is the push I needed to<br>burn some tokens.

I in fact ended up doing a series of A/B experiments, none of<br>which are very scientific, but I want to test realistic scenarios.<br>Creating neat test environment tends to increase in difficulty the<br>more realistic you try to make it.

The first experiment was to pit C++ with GGML Vs Rust with Burn.<br>They both had to reach parity, in terms of correctness and<br>performance against the PyTorch implementation of FLOAT (it animates<br>portraits given voice audio to make it appear they are talking).

The final experiment was to create a pure CPU only implementation<br>using SIMD libraries in C++ and Rust. Removing the confonders<br>introduced by pitting GGML against Burn.

The original suggestion was to use GGML from unsafe blocks in<br>Rust, but I thought that this might put Rust at a disadvantage,<br>nullifying its safety properties. Rightly or wrongly I decided to go<br>with Burn instead and I can say right now Burn is great. Except that<br>is on CPU, it’s terrible on CPU. On GPU though it is competitive<br>with GGML and PyTorch on inference and it does training too.

I have published Fable’s and ChatGPT’s analyses in the rust-vs-cpp-analysis<br>repository, but here I’ll give my opinions and highly condensed<br>version of events.

I asked Claude to create 3 independent sub-agents; one to do an<br>analysis of the PyTorch implementation, create a parity harness to<br>do layer by layer numerical comparisons and do some Rust research to<br>level the playing field with C++.

You see, the thing is, the C++ agent would have a fair more prior<br>art to go on, because we have done a bunch of these conversions and<br>it helps to give access to those prior conversions. I’m not going to<br>withold that info because I want results. The best I can do is do<br>some extra work upfront to try and convert lessons learnt on C++<br>projects into lessons for a Rust proejct.

The other two subagents are the Rust and C++ implementors. They<br>ran until they reached correctness and performance parity with the<br>PyTorch implementation.

Initially I thought I had walked into a decisive result proving<br>Rust’s greatness. The Rust agent reported about 50% of the token<br>usage while achieving the same performance. However it turned out<br>the Rust agent benefited from some work the C++ agent did to debug<br>the parity harness.

The primary Fable agent also decided to do the GGML<br>implementation in CUDA and the Rust one in Vulkan. Then blamed me<br>for asking that GGML use CUDA and Burn use Vulkan. I don’t recall<br>asking for that, but in any case it is a confounder, but not a big<br>one as it turns out. Regardless of whether we use Vulkan or CUDA,<br>fused kernels are required in GGML and something similar in Burn to<br>get parity with compiled PyTorch.

After some iterations trying to get the Rust and C++ ports to be<br>equivalent, I decided to do something more drastic as a final<br>test.

I asked to create two new ports, both in new repositories, but<br>using lessons learned from the previous attempts. This time the<br>ports would be CPU only and use SIMD libraries. I demanded that they<br>be statically compiled and allow cross compiling to ARM. Do zero<br>allocations during computations and a bunch of other stuff.

Again it initally appeared that Rust had achieved a victory, but<br>it turned out that the rust agent simply decided not to create a C<br>API. Possibly because I didn’t explicitly ask for it, but of course<br>there should always be a C API/ABI otherwise only other Rust code<br>can consume the library.

The Rust port was also a bit slower, because the agent simply<br>decided to stop with perf optimizations sooner. Maybe Claude doesn’t<br>like optimizing Rust or more likely it’s just the random nature of<br>LLMs coming through.

In any case after forcing the agent to complete the C API/ABI and<br>get...