The Reframe | Matt Moughtin | MediumSitemapOpen in appSign up<br>Sign in

Medium Logo

Get app<br>Write

Search

Sign up<br>Sign in

The Reframe

Matt Moughtin

6 min read·<br>Just now

Listen

Share

The Engineer’s Journey: Part seven of a series on building a real-world AI system from scratch.<br>While I was deep in the valley fixing collectors, I was also reading papers.<br>The data grind took about six weeks of explicit work. What it didn’t take was all of my attention. I fixed collectors in the morning and the evening. I read about model architecture whenever I had spare time. In bed, on the train to work, on my lunch break.<br>I had assumed, at the start of the data work, that the 42% model just needed more data. By early March I had stopped assuming that. I had a growing suspicion that the architecture I’d been training wasn’t going to work on 5,000 species no matter how much data I gave it.<br>The suspicion wasn’t founded on anything I could prove. It was founded on the mismatch between what the model was doing and what the app would eventually ask it to do.<br>The mismatch<br>The model was being trained to classify. Given an image, assign it to one of 5,000 species. Loss function: softmax cross-entropy, the standard choice for multi-class classification, which rewards the model for pushing the correct class’s probability up and the wrong classes’ probabilities down. This is how you train a classifier. It is what classifiers are for.<br>The app, at runtime, wasn’t going to do classification. The app was going to do similarity search. Given an image, compute an embedding. A 256-dimensional vector representing the image’s position in learned feature space, and find the nearest known plant in a precomputed reference set. This is a different question from classification. It asks which known example does this look most like, not which fixed label applies to this.<br>The two objectives sound similar. They aren’t. A model trained to classify correctly can produce embeddings that are terrible for similarity search. The classification objective rewards decision boundaries between classes. The similarity objective rewards distances between classes. You can satisfy the first without the second. You can have a model that classifies at 95% accuracy and produces embeddings where intra-class variation is larger than inter-class variation. Where two images of the same species land further apart than an image of that species and an image of something else entirely.<br>I had been training for classification. The app would use embeddings. I had been training the wrong thing.<br>The search<br>Once I saw the mismatch clearly, I started looking for what to do about it. I spent most of March reading. Papers, blog posts, model zoos, benchmark results. I used AI tools to help me map the space, asking them for similarity-learning approaches I hadn’t considered, then going to the primary sources to evaluate each one.<br>There are several families of approaches to similarity learning. Some try to teach the model to place known pairs at correct distances from each other. Some pull similar examples together and push different ones apart. Some learn distance functions directly. Some use text descriptions of images to learn what similar things look like.<br>Each had their own weaknesses. Some were notoriously unstable in training. Some produced collapsed solutions where every embedding ended up in the same region of space. Some required data I didn’t have. Some worked but plateaued at the scale I needed.<br>What kept coming up in my reading, and what kept looking more relevant, was ArcFace.<br>Press enter or click to view image in full size

Research — Illustration generated with AIThe choice<br>ArcFace was developed for face recognition. The problem it was built to solve is structurally the same as mine. Thousands of classes, each with significant intra-class variation. The same face in different lighting, different expressions, different angles. A runtime objective of similarity search: does this new photo match someone we know? A requirement that classes be not just separable but distant, because at runtime the decision isn’t which class is highest, it’s is this close enough to any known class to call it a match.<br>The mechanism ArcFace uses to do this is angular margin. Instead of optimising linear decision boundaries between classes, it places all embeddings on a hypersphere and pushes classes apart by angle. The margin is measured in radians. Two classes that are hard to separate don’t just get a wider decision boundary between them; they get rotated to different regions of the sphere entirely. At inference, cosine distance between any two embeddings corresponds directly to how the model sees their similarity. The training objective and the serving objective become the same function.<br>The SubCenter variant, which I ended up using, allows each class to have up to K=3 embedding clusters rather than being forced into a single point. This matters for species. Sage photographed in the wild,...