The linear algebra and calculus behind every model

Sign in<br>Subscribe

Last time we built our first Python pipeline and learned to load, inspect, and manipulate data. Now we take the next step: every machine learning model, from linear regression to deep neural networks, rests on three pillars: linear algebra, calculus, and convex optimization. We'll build each one from scratch using the MNIST dataset, and by the end we'll have compressed images with the SVD, computed gradients by hand, and solved a constrained optimization problem. The through-line is simple: every operation we perform on a model is a transformation of vectors and matrices, guided by derivatives and bounded by constraints.<br>The data: MNIST as a vector space<br>We work with the MNIST database of handwritten digits: 70,000 grayscale images, each 28 by 28 pixels. The first thing we do is flatten each image into a 784-dimensional vector. That turns our dataset into a matrix with 70,000 rows and 784 columns—a point cloud in a high-dimensional space.<br>Our exploratory data analysis reveals a clean dataset. The classes are nearly balanced, with the most frequent digit (1) appearing 11.25 percent of the time. No missing values, no duplicates in the first 5,000 images. The pixel intensity distribution is bimodal: most pixels are dark background, a small fraction are bright ink. Figure 1 shows five random MNIST images. Figure 2 displays one example per digit class. Figure 3 is a histogram of pixel intensities. Figure 4 shows additional random samples to illustrate variation.<br>Some digits are lightly written or have unusual shapes. That's fine—a linear model can handle variation, but it will need many components to capture the full diversity. We'll see exactly how many when we apply the SVD.<br>Vectors and norms<br>Every image is a vector. The L1 norm sums the absolute pixel values; the L2 norm takes the square root of the sum of squares. For our first image (a digit 5), the L1 norm is 107.94 and the L2 norm is 9.56. These measure "brightness" in different ways: L1 treats every pixel equally, while L2 penalizes large values more.<br>The dot product between two images tells us how similar they are. The first and second images have a dot product of 53.76 and a cosine angle of 0.55—they share background patterns. Orthogonality (dot product zero) is rare in natural images because most pixels are dark and align.<br>Matrix multiplication lets us compute many dot products at once. We form the Gram matrix G = X @ X^T, a 500-by-500 matrix where each entry is the similarity between two images. The diagonal entries are the squared L2 norms: 91.35, 103.81, 59.27, and so on.<br>Tensor shapes and broadcasting are the mechanics behind every deep learning framework. Our images arrive as (N, 1, 28, 28) tensors. We flatten them to (N, 784) and add a bias of 0.5 to every pixel in one line: X_broadcast = torch.from_numpy(X).float() + bias. Broadcasting stretches the scalar across all 784 columns and all 500 rows without a loop.<br>Projections and the SVD<br>Now that we have vectors and dot products, we can ask how to compress images, measure stability, and write efficient tensor operations. That requires several tools from linear algebra. We begin with projections , which let us decompose a vector into components along and perpendicular to a direction. We center the data by subtracting the mean image. The mean image is the average digit—a blurry composite that captures the typical stroke pattern. Projecting the first image onto this mean direction gives a vector with L2 norm 6.49, compared to the original's 9.56. About 68 percent of the image's energy lies along the mean direction.<br>The Singular Value Decomposition (SVD) factors a matrix into three matrices: U, S, and V^T, where S contains the singular values that measure the importance of each direction. The SVD decomposes our centered data matrix into U, S, and V^T. The singular values S tell us how much variance each direction captures. The top five eigenvalues of the covariance matrix are 5.22, 3.89, 3.37, 2.87, and 2.61. The numerical rank—singular values above machine precision—is 626 out of a possible 784. MNIST images are high-rank but compressible.

This post is for paying subscribers only

Subscribe now<br>Already have an account? Sign in

Read more

The first six ideas in machine learning

Every machine learning project, no matter how sophisticated, rests on a handful of foundational ideas. In this post we build them from scratch using the UCI Adult Census Income dataset, a classic binary classification task where we predict whether someone earns more than $50,000 per year. By the time

Powered by Ghost