Lossy vs Lossless: how the compression metaphor unfolds in practice (and how we can measure it)

The Farsighted Zipper's Substack

SubscribeSign in

Lossy vs Lossless: how the compression metaphor unfolds in practice (and how we can measure it)<br>Compression can be lossy or lossless. But while the former is very beneficial in terms of compression ratio if the performance is acceptable, the latter pays a price that can also be an investment.

The Farsighted Zipper<br>Jun 11, 2026

Share

TL;DR — We take the humblest model in machine learning (a linear regression, 1000 apartment prices in, two numbers out) and ask the compression some questions with increasing severity: how big is the data, how big is the model, and who pays for the errors? Same model, same data, three answers: 41:1 if we accept the errors and the naive packing, 21:1 if we encode the data more honestly, and 1.18:1 if we want lossless compression. The gap is the lesson: a model can look like a spectacular compressor until it pays for everything it threw away. Two surprises along the way: the honest ratio converges to a ceiling set by the world, not by the model, and the residuals we pay to keep turn out to be the to-do list of the next model. Every number below reproduces from the script.

This post is a follow-up of the book “The Farsighted Zipper”, which is freely available online.<br>GET THE BOOK

The claim

We have 1000 apartments, each of them described by size in m2 and price in k$. This represents our field experience, and we want to find a rule that allows us to predict the price given the size. This would be, in other words, a compression of the data, a rule that replaces the data with a lower bit cost. How do we find that compression rule? And how do we measure the advantage achieved? This example showcases the compression metaphor through a practical numeric example. If everything works, at the end of this post we will have a shared vocabulary and procedure, that we can apply on a wide variety of cases. Let’s go!<br>Step 1: the world (data)

We don’t have actual empirical data, but this is not game over: we generate a synthetic dataset according to an arbitrary rule: price = 4.2 · size + 50 + noise, where noise ∼ N(0,25). This is our starting point, our world (or, at least, our arbitrary representation of the world).<br>import numpy as np<br>from scipy import stats

rng = np.random.default_rng(0)<br>N = 1000<br>size = rng.uniform(30, 120, N)<br>price = np.round(4.2 * size + 50 + rng.normal(0, 25, N)).astype(int)

Step 2: The model (compression)

To find a rule, we first have to make hypotheses. In our case, we assume that the relationship between size and price is linear1, and we opt for a linear regression. Then, we test its predictive performance through 10-fold cross-validation2.<br>[1] Naturally, we know this because we literally instructed the data generation process, but let’s put ourselves in the most typical case in which the true underlying mechanism originating the data is unknown; before fitting our model to the data, we need to pick a model class, or more than one, and there is no universal rule to do that.<br>[2] Fitting the linear regression to the entire data and test it on the same set of examples would overestimate the predictive performance of the model. Instead, we should hold part of the data for out-of-sample testing. In our case, we cross-validate with a K-fold approach: we split the data set into K folds, and then we iteratively fit the model on K-1 folds to predict the response for the remaining hold-out fold. By doing this, we can get a more realistic perspective over the generalisation capability of the model. Honestly, in this trivial case, given the simple generation logic and the size of the data set, the difference between out-of-sample (K-fold cross-validation) and in-sample performance measure is not relevant, but this does not diminish the need to test the model on a hold-out test set.<br>def entropy_bits(values):<br>"""Shannon entropy of the observed distribution: -sum p*log2(p)."""<br>_, counts = np.unique(values, return_counts=True)<br>p = counts / counts.sum()<br>return -(p * np.log2(p)).sum()

def ten_fold_cv_predictions(size, price, k=10, seed=42):<br>"""Out-of-fold predictions: each price is predicted by a line fitted<br>on the other 9 folds. Returns integer predictions for every point."""<br>rng = np.random.default_rng(seed)<br>idx = rng.permutation(len(price)) # shuffle once<br>folds = np.array_split(idx, k) # 10 disjoint folds<br>oof_pred = np.empty_like(price)<br>for i in range(k):<br>test = folds[i]<br>train = np.concatenate([folds[j] for j in range(k) if j != i])<br>a, b = np.polyfit(size[train], price[train].astype(float), 1)<br>oof_pred[test] = np.round(a * size[test] + b).astype(int)<br>return oof_pred

oof_pred = ten_fold_cv_predictions(size, price)<br>oof_resid = price - oof_pred<br>rmse_cv = np.sqrt((oof_resid ** 2).mean())<br>mape_cv = np.mean(np.abs(oof_resid / price)) * 100<br>R2_cv = 1 - oof_resid.var() / price.var()

print("ERROR...