Don't Sleep on BitNet

EDIT: Apparently I’m not alone in feeling this way, here’s a post from the same day I posted this: All LLMs Will Be Sparse BitNet Hybrids

I took a break from working on my decentralized DB to dive into the latest LLM research. As usual, I focused on opportunities for optimizations and performance improvements.

This post is mainly a summary of my private notes I made while ‘researching’.

I’ve only looked at inference (generating output, i.e. actually using the models) as that should dominate the total compute cost, and it’s what needs to be efficient for local-first computing.

I read through a lot of techniques, but by far the technique with the most potential seems to be BitNet.

BitNet uses ternary weights with the values of [-1, 0, 1], which gives it extremely appealing performance characteristics.

I. An Overview of LLM Architecture

Link to heading

Before diving into optimization techniques, let me briefly explain where all the computation goes. At a simplified level an LLM consists of:

Weights : Parameters learned during training (the actual “knowledge”)

Activations : Intermediate values computed during text generation

Layers : Stacked transformations, each applying specific operations

KV Cache : Storage for already-processed tokens (grows with context length)

When generating text, the model processes input through these layers, with the primary computational workhorses being:

Matrix Multiplications (MatMuls) : These dominate compute time. Modern GPUs are built specifically to parallelize these operations, but they’re still expensive.

Memory Bandwidth : Moving gigabytes of weight data between memory and computation units is often slower than the computation itself.

KV Cache Growth : As context length increases, the memory needed to store intermediate states grows dramatically. This is why longer chats with LLMs consume far more RAM.

These bottlenecks become acute when trying to run models locally rather than in data centers. Cloud providers can throw specialized hardware and custom chips at the problem. Your laptop… cannot.

Model sizes

Link to heading

Taking a step back, I should point out how LLM sizing will likely end up working in the long run. These are just my assumptions based on the hardware realities of today.

For practical deployment you can think of models in a few different tiers, each with very distinct capabilities:

Always-on assistant models :

Sub-8B parameters, heavily quantized to fit in RAM or on accelerators. Useful for simple interactions and tasks, and can hand off to more capable models as needed.

Local Specialist models :

12-30B parameters with special training. You’d run the largest models you can locally for specialized tasks, capable of more autonomy. “Label all my photos with descriptions”, “Research a trip to Cleveland that I would like”.

Cloud fallback :

Full-precision 70B+ models for the hardest tasks. Almost everyone will access these models remotely.

The optimal sizes will depend on hardware constraints, but even modest laptops can run Local Specialist Models at acceptable speeds with aggressive quantization. BitNet would give that extra push of potentially doubling the parameter sizes that could be run locally, and would significantly improve the inference speed at the same time.

At small sizes (1B-30B), like the ones most people will interact with frequently , doubling the number of params is a significant increase in capabilities. An 8B model is much more capable than a 4B model.

II. Existing Efficiency Hacks

Link to heading

Several approaches aim to address these challenges with big efficiency wins, these are all in use in various forms today:

Quantization Techniques

Link to heading

Quantization reduces the precision of model weights and activations. While 8-bit and 4-bit quantization are now standard practice, research is pushing toward even more extreme compression:

Traditional quantization (8-bit, 4-bit) : 4-bit weights often have very little performance degradation and are quite common in local inference today.

Extreme quantization (2-bit and below) : The current focus of several research groups.

The ParetoQ paper (arXiv:2502.02631) found something particularly interesting: 1.58-bit (3 values that vary per layer) or 2-bit (4 values, varying per layer) quantization may represent an optimal balance between compression and capability. For a fixed compute budget, you’re often better off running a larger model at lower precision than a smaller model at higher precision.

To get the most out of it though, you need to further finetune the quantized model to regain as much performance as possible after quantization. They show that it’s possible to regain a lot of the capabilities ’lost’ when quantizing by finetuning on 10-30 Billion additional tokens after the quantization.

This is a very promising...