Two database migrations and a divorce - Fathom Analytics Skip to main content Free trialLogin
Pricing<br>Free trial<br>Log in<br>Menu
Two database migrations and a divorce<br>technical<br>Jack Ellis · Jul 10, 2026 I’ve just finished the most challenging database migration of my life. We have officially shipped Fathom Version 4, migrated over 65 billion rows to a brand-new database setup, and eliminated every single piece of technical debt we had. The project took hundreds of hours over a handful of months and was handled entirely by a single software engineer (me). I’m going to share all of the details with you, so sit back, grab a coffee and let me take you on a journey.
The origin story
Back in 2019, we were receiving an increasing number of questions from customers about emerging cookie laws. Despite being simple, respectful of privacy, and collecting a genuinely minimal amount of data, we were still using cookies. These customers were concerned and wanted us to stop.
The problem was that we needed those cookies to know whether a visitor was unique to the site and unique to the page. We stored data like this:
site_id: 'ABCDE',<br>hostname: 'https://apple.com',<br>pathname: '/',<br>is_unique: true, // First time on site?<br>page_unique: true // First time on page?<br>So we couldn’t just “drop the cookies” because they were a core part of our analytics ingest infrastructure.
So we were stuck. We knew the laws were changing, but our entire data model relied on cookies to determine whether a pageview was unique. Analytics software has used cookies for unique visitors since dinosaur times. How do you track uniques without cookies? You can’t use access logs. That’s raw data across millions of websites and includes personal data (IP addresses). You can’t just fingerprint users either, because then you’re able to uniquely identify them, potentially for years, and the fingerprint is re-creatable personal data. We had nowhere to go.
We don’t want the data
Raw access logs were a non-starter. An owner of a website can choose to use raw access logs on their own site, sure, but we’re an analytics company tracking billions of unique website visitors across millions of different websites. We don’t ever want to track individuals across multiple sites and store their browsing habits.
Imagine an advertising company. Let’s call them Google. They have an interesting history regarding protecting their users’ privacy. Here’s the difference between running your own raw access logs and what happened when you put Google Analytics on your website:
You fed into an advertising company’s ability to watch where individuals go around the web.
So that was our starting point. We couldn’t solve uniques by fingerprinting users or storing IPs, because it would give us data on how a user behaves across the internet. But then we realised the fingerprint might actually work.
The birth of privacy-first analytics
Typical fingerprints couldn’t work, because they’re re-creatable and can identify individual website visitors. If the contents of [ip]:[deviceTraits] always yielded the same outcome, we’d be storing personal data. That was forbidden per the specification.
Maybe we could hash the plain text? Then surely that would be okay, because we’re not storing personal data; we’re storing hashed personal data. No, the lawyers said. That’s not going to pass, because it can easily be re-created the minute you have the IP and the device traits. If it can be easily tied back to a user, you are processing personal data.
We reviewed the privacy laws and realised they were right. It reminded us that, back in the day, the industry realised a simple MD5 hash of a user’s raw password wasn’t secure because bad actors could use rainbow tables to match an MD5 hash to a raw password. This was before password managers became as popular as they are, so if someone cracked your password, all your internet accounts were often hacked. The industry’s fix was a salt: md5('$jlJhiuweh2sl40' + $rawUserPassword). And software engineers hedged further with a unique salt per user. And then it hit us, salts were the solution for Fathom’s cookie headache!
Not just any salt. Fresh, grass-fed, organic salts, that were refreshed daily, loaded in with the site ID, hostname, and user’s fingerprint. The salt would be the SHA-256 hash of a random string.
The final user identifier was:
hash('sha256', $siteId . $ip . $userAgent . $dailySaltSha256)<br>And the salt would be gone within 24 hours or less, making it impossible to reconstruct the hash. Actually, my first-ever blog post covered this in detail.
We then performed a risk analysis. What’s the worst thing that could happen? You cannot brute force a 256-bit hash. In the blog post at the time, I wrote:
“Brute forcing a 256 bit hash would cost 10^44 times the Gross World Product (GWP). 2019 GWP is US$88.08 trillion ($88,080,000,000,000) so we’re at least a few dollars short of brute forcing a 256 bit hash.”
And even if you assumed a...