GitHub - elpinyeknom/negative-squaring-: Pre-tilting weights before quantization to preserve reasoning — toy experiments · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
elpinyeknom
negative-squaring-
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>12 Commits<br>12 Commits
1_first_experiment.py
1_first_experiment.py
2_gradient_attempt.py
2_gradient_attempt.py
3_final_with_clipping.py
3_final_with_clipping.py
LICENSE
LICENSE
README.md
README.md
negative-squaring-paper_1.md
negative-squaring-paper_1.md
View all files
Repository files navigation
Negative Squaring — Toy Experiments
Code accompanying the writeup "Negative Squaring: Pre-Tilting Weights to<br>Preserve Reasoning in Quantized Models" (July 2026). See<br>negative-squaring-paper_1.md in this repository for the full plain-language paper.
The idea in one line
Before quantizing a model, tilt each weight against the error the rounding<br>will cause across the model's whole multi-step reasoning trajectory — clipped<br>to half a quantization step, so the tilt only decides which way borderline<br>weights round.
What's here
File<br>What it does<br>Key result
1_first_experiment.py<br>Random-search pre-tilt vs naive 4-bit quantization on a 12-layer, 30-step recurrent toy network<br>~18% trajectory error removed; decision flips 14/20 → 8/20
2_gradient_attempt.py<br>Straight-through gradient search, unconstrained<br>Backfires — test error gets worse (documented negative result)
3_final_with_clipping.py<br>Gradient + random + combo searches, with tilts clipped to half a quantization step<br>77% error removed; decision flips 20/50 → 4-5/50
Run it
Requires only Python 3 and numpy:
pip install numpy<br>python 3_final_with_clipping.py
Each script is self-contained, seeded, and reproduces the numbers in the<br>writeup. Runtime is seconds to a few minutes on any laptop. To reproduce the 3-bit and 2-bit cliff results, change BITS = 4 to 3 or 2 on line 4 of script 3.
Honest limitations
Toy scale: ~49k weights, tanh recurrence, not a transformer.
The toy's dynamics dampen errors; real LLMs often amplify them. Untested there.
Full-trajectory backprop is expensive at real scale; the clipping constraint<br>shrinks the search space (only near-boundary weights matter) but efficient<br>scaling is unsolved.
Open invitation
If you have compute and want to try trajectory-aware rounding on a real<br>sub-1B model, or you know prior literature that already does this<br>(AdaRound optimizes rounding decisions per-layer; we're looking for<br>whole-trajectory versions), please reach out in the thread or open an<br>issue here.
About
Pre-tilting weights before quantization to preserve reasoning — toy experiments
Resources
Readme
License
MIT license
Uh oh!
There was an error while loading. Please reload this page.
Activity
Stars
star
Watchers
watching
Forks
forks
Report repository
Releases
No releases published
Packages
Uh oh!
There was an error while loading. Please reload this page.
Contributors
Uh oh!
There was an error while loading. Please reload this page.
Languages
Python<br>100.0%
You can’t perform that action at this time.