GitHub - Louw115/nep-ethereum-compression: Structured encoding layer for Ethereum JSON-RPC data — improves zstd compression by 12–21% · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Louw115
nep-ethereum-compression
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>6 Commits<br>6 Commits
.gitignore
.gitignore
LICENSE
LICENSE
NEP_BENCHMARK.md
NEP_BENCHMARK.md
NEP_ONE_PAGER.md
NEP_ONE_PAGER.md
README.md
README.md
engine_v2.py
engine_v2.py
large_scale_results.json
large_scale_results.json
nep_blocks_demo.py
nep_blocks_demo.py
nep_reproduce.py
nep_reproduce.py
requirements.txt
requirements.txt
View all files
Repository files navigation
NEP — Neurop Encoding Protocol
A structured encoding layer for Ethereum JSON-RPC block data that improves zstd compression by 12–21%.<br>NEP sits in front of zstd. It converts verbose Ethereum JSON into a compact binary format before the compressor runs. Not a compressor — a transform.
Results
Tested on real Ethereum mainnet blocks fetched live from a public RPC endpoint. No pre-built datasets.
Method<br>Mean ratio<br>vs zstd<br>Win rate<br>Lossless
gzip-9<br>4.97x<br>−13%<br>0/200<br>N/A
brotli-11<br>5.34x<br>−6%<br>0/200<br>N/A
zstd-9<br>5.71x<br>baseline<br>0/200<br>N/A
zstd-9 + dict<br>5.37x<br>−6%<br>0/200<br>N/A
NEP + gzip-9<br>5.84x<br>+2%<br>0/200<br>200/200 ✓
NEP + zstd-9<br>6.18x<br>+8%<br>2/200<br>200/200 ✓
NEP + zstd + dict<br>6.38x<br>+11.7%<br>200/200<br>200/200 ✓
NEP beats plain zstd on every single block. Every output verified lossless.
Independent reproducibility runs (consecutive recent blocks) show +16% to +21% — see NEP_BENCHMARK.md for the full three-tier results and explanation of the range.
NeuropBlocks Demo — composable primitives outperform the full engine
nep_blocks_demo.py rebuilds the NEP pipeline using 5 composable primitives from the NeuropBlocks library. No custom encoder — just blocks composed together.<br>Run it yourself (requires only zstandard):
pip install zstandard<br>python nep_blocks_demo.py
Results on 20 real mainnet blocks (4,990 transactions), trained on 480 separate blocks:
Method vs zstd-22 Blocks won<br>Plain zstd-22 baseline —<br>NeuropBlocks + zstd-22 +3.91% 19/20<br>NeuropBlocks + dict + zstd-22 +5.14% (incl. 112KB dict) 20/20<br>Output verified on Windows 10, Python 3.11:
Plain zstd-22 (baseline): 1,510,287 bytes<br>NeuropBlocks + zstd-22: 1,451,193 bytes (+3.91%)<br>NeuropBlocks + dict + zstd-22: 1,318,018 bytes (+12.73% raw)<br>NeuropBlocks + dict (with overhead): 1,432,706 bytes (+5.14%)
The 5 NeuropBlocks doing real work:
Block Domain What it does<br>hex_decode data/string All hex fields → raw bytes (hashes, calldata, bloom)<br>deduplicate_by data/collection Address lookup table — from/to/miner + ABI calldata recipients<br>delta_encode data/collection 7 numeric sequences: nonce, gasPrice, value, gas, txIndex, maxFeePerGas, priorityFee<br>delta_decode data/collection Lossless verification<br>pack_integers data/collection Tx type bits packed into 4-bit slots<br>The dictionary amortises in ~17 blocks. In a production stream (Alchemy, QuickNode) it is effectively free.
Reproduce it yourself<br>No API key. No pre-built files. Fetches live mainnet data.
# Install<br>pip install zstandard brotli # Mac / Linux<br>py -m pip install zstandard brotli # Windows<br># Run interactive demo<br>python nep_reproduce.py # Mac / Linux<br>py nep_reproduce.py # Windows
The script runs in three stages:
Stage Blocks Time Held-out test<br>1 — Quick demo 20 ~1 min 12 blocks<br>2 — Confidence 100 ~3 min 60 blocks<br>3 — Full benchmark 200 ~8 min 120 blocks<br>Each stage prints a full per-block table (raw bytes × ratio × lossless flag) and saves a JSON results file. The cache builds progressively — Stage 2 only fetches the delta blocks.
How it works<br>Ethereum JSON → NEP encode → zstd → storage<br>Ethereum JSON ← NEP decode ← zstd ← storage
NEP is a fully deterministic 4-stage transform — not a learned model, not a black box. Every byte is reversible by spec.
Stage What it does Why it helps<br>1. Hex → binary "0x1a2b3c" → raw bytes Cuts every hex value to 50% of its JSON size<br>2....