Migrating 6.5 billion rows from MongoDB - Fathom Analytics Skip to main content Free trialLogin
Pricing<br>Free trial<br>Log in<br>Menu
Migrating 6.5 billion rows from MongoDB<br>technical<br>Jack Ellis · Jul 27, 2026 At the start of this year, we acquired Gauges, a simple analytics platform. We’ve just finished moving 6.5 billion rows of historical analytics data from its MongoDB database to ClickHouse, the same database we run Fathom on. I’m going to share how it all went down.
New customers and old promises
When we acquired Gauges, we had no idea how many customers would want to move to Fathom. We knew we were going to offer everyone a path forward with Fathom and offer to migrate their data for them. Lots of customers got back to us and were excited about the new home for their data.
We then made two important promises. First, Gauges customers would not need to replace the tracking code on their websites. We would route the existing Gauges ingest endpoints into Fathom.
Second, we had a conversation with an enterprise customer coming from Gauges. We promised they wouldn’t have to rewrite any of the API code that they used as part of this migration.
That meant we had to migrate all of the historical data whilst keeping the live product and its public interfaces working as they always had. Two jobs, really: replace the application, and move the data.
Who doesn’t love a good rewrite?
We inherited a Ruby on Rails application on AWS. It used Elastic Beanstalk, EC2, SQS, ElastiCache, an Application Load Balancer, VPC, CloudWatch, S3 and CloudFront. The infrastructure cost thousands of dollars each month.
I know all of those AWS services well, and I’m confident I could have modified the existing application, but I hadn’t worked with Ruby since I was 15. I didn’t want our long-term plan to depend on me deploying changes in a language that our company isn’t deeply familiar with.
So I rebuilt the application in Laravel, and it was split up into two subprojects:
Gauges Ingest conversion layer into Fathom ingest
Public API layer for the enterprise customer
I went through the API, the ingest endpoints and the underlying data structure. The new application would have to handle the exact same responses and return the exact same data structure.
After spending some time building it, testing it and making sure it worked, we had an application ready for production. The data, however, was going to be a whole different story.
Finding the data
Gauges was built in 2011, years before ClickHouse existed, and stored its analytics in MongoDB.
I had never connected to a MongoDB database in my life. When I was younger, MongoDB was a meme, and I spent my career avoiding it. Yes, I’m aware it’s now a very successful multi-billion-dollar database company.
I found the EC2 instance running MongoDB, added my IP address to its firewall and connected from my machine. The database contained 4,705 collections and nearly 7 billion documents.
The data was split into collections named by content type, year and month. Historical collections were immutable, which is my favourite kind of analytics data.
But MongoDB is a document database. I couldn’t write a familiar SQL query and start pulling rows into ClickHouse. I needed a way to read it safely, in chunks, from our production Fathom application.
Four hours from deadline to plan
To stop myself over-analysing the problem, I emailed Gauges customers and told them I would start migrating their accounts the following week.
No, I didn’t have a complete plan.
I’ve found that having a deadline makes me move faster. Less than four hours after sending the email, I had the answer.
The MongoDB version was outdated, so our modern PHP application couldn’t connect to it. I could, however, create a Laravel Forge server with a compatible MongoDB binary. That server could communicate with the database, and our production application could communicate with the server.
I turned the Forge server into a small, private, read-only migration API for us to use. Fathom would request data from it; it would read from MongoDB; and the response would be returned to our existing migration code.
This let me keep all the account mapping, ClickHouse configuration and migration state inside Fathom. The Forge server had one job: provide a narrow interface to the old database.
Building the migration API
I explored the Gauges codebase and mapped its collections into endpoints. Then I built a proof of concept in our local admin panel that read Gauges data through the migration API.
It worked.
The explorer showed the Gauges sites on an account, the months available for each site and the metrics in each collection. We could inspect the protected MongoDB data without connecting our main application directly to MongoDB.
Now we could build the actual migration.
Much ado about mapping
Some Gauges customers had already created sites in Fathom and installed our tracking code. Others still used the original Gauges...