MSE-GLM: A Deterministic, Zero-Weight Graph Language Model, Step by Step
MSE-GLM: A Deterministic, Zero-Weight Graph Language Model, Step by Step
By Clifford Chivhanga ·<br>July 17, 2026 ·
github.com/fodokidza/mse_glm
Most language models today are a pile of weights nobody can fully read.<br>MSE-GLM (Matrix-Structured Edge — Graph Language Model )<br>is an attempt at the opposite: a language model with no neural<br>weights, no embeddings, and no probability distributions . Every<br>decision it makes during generation can be traced back to a specific row<br>in a specific matrix, built from the training text itself. Nothing is<br>learned in the gradient-descent sense — everything is counted,<br>deduplicated, and indexed.
This post walks through every stage of the pipeline in the order it<br>actually runs — tokenizing, building the graph, generating text,<br>inferring beyond what was literally seen, and finally, a newer piece:<br>figuring out what a cluster of interchangeable tokens actually<br>means.
Repo:<br>https://github.com/fodokidza/mse_glm
Core files: tokenizer.py, graph.py,<br>train.py, inference.py, experience.py,<br>analyse.py, interpret.py, model.py
Step 1 — Tokenizing the corpus (tokenizer.py)
Everything starts with a from-scratch Byte Pair Encoding (BPE) tokenizer.<br>No external tokenizer library, no pretrained vocabulary. Training text is<br>lowercased, stripped of anything that isn't a letter, digit, or space, and<br>split into sentences on . ! ? and newlines.
Four reserved ids anchor the vocabulary before any merges happen:
= 0 reserved<br>= 1 unknown character fallback<br>= 2 prepended to every encoded sequence<br>= 3 appended only during training
From there it's classic BPE: every character in the corpus becomes a<br>one-character token, then the tokenizer repeatedly finds the most frequent<br>adjacent pair of symbols across all words and merges it into a new token,<br>until vocab_size is reached or there are no pairs left to<br>merge. The merge list is stored in order and replayed identically at<br>encode time, so a word is always split the same way it would have been<br>split during training.
Two details matter for everything downstream:
encode() always prepends —<br>this matters later because BOS is treated as a real, if artificial,<br>"previous token" for the very first generation step.
encode_for_training() additionally appends<br>, so every training sentence has an explicit,<br>learnable end.
Step 2 — Building the graph (graph.py)
Once every training sentence is a sequence of token ids, MSE-GLM builds<br>three matrices from those sequences. All three are array-backed<br>(Python's array('i')) and CSR-indexed — the same<br>compressed-sparse-row layout used in sparse linear algebra — so<br>every lookup ("what comes after token X?") is O(1) rather than a scan.
2a. The Edge Matrix — what follows what
The simplest structure: every distinct bigram (token[i],<br>token[i+1]) seen anywhere in training, deduplicated. Given a token,<br>successors() returns every token that has ever directly<br>followed it. This is the model's notion of "grammatically legal<br>next token" — nothing more.
2b. The Bridge Matrix — triples, and where clusters come from
This is the heart of the system. For every 3-token window<br>(source, bridge, target) in the training sequences, the<br>Bridge Matrix stores a deduplicated triple. So for "the cat sat",<br>the triple is (source=the, bridge=cat, target=sat).
Every triple additionally gets a cluster_id, assigned by a<br>simple dual-axis rule:
Bridge axis — if two or more triples share<br>the same (source, target) pair, their bridge<br>tokens are grouped into one cluster. Example: "the cat sat" and<br>"the dog sat" share (the, sat), so cat<br>and dog get the same cluster_id — they're<br>interchangeable in that slot.
Target axis — if two or more triples (not<br>already grouped on the bridge axis) share the same<br>(source, bridge) pair, their target tokens are<br>grouped instead. Example: "cat is animal" and "cat is pet"<br>share (cat, is), so animal and pet<br>get grouped as things "cat" can be.
Anything that doesn't fit either rule gets cluster_id = 0<br>— the "unclustered" bucket. (More on why that bucket turns out to<br>matter later in this post.)
Alongside the triples, the Bridge Matrix keeps a t_index:<br>a reverse map from every token to the (non-zero) cluster_ids it<br>participates in as a bridge or target anywhere in the graph. This is<br>what lets the system ask "does token A show up in the same kind of slot<br>as token B, anywhere in the corpus?" without rescanning everything.
2c. The Relationship Matrix — lineage
The Relationship Matrix is deliberately thin: it stores only pairs of<br>(triple_id, relationship_id), where a relationship_id<br>is just the index of the training sentence a triple came from. It doesn't<br>duplicate any triple content — it's a foreign key back into the<br>Bridge Matrix's row order. A single triple can belong to several<br>relationship_ids if the exact same 3-token pattern showed up in more than<br>one training sentence.
This sounds like a small bookkeeping detail, but it's what...