Oops I ran the tests on prod - by chetan vashisht
Terminal
SubscribeSign in
Oops I ran the tests on prod<br>Dev Tales #1<br>May 22, 2026
Share
During my early career with Trumswear(2017-20), we used a Quartz scheduler for managing asynchronous tasks. To understand how the Quartz tests were written, I ran their test suite. Only to realize that I was connected to production… I dropped prod!<br>I was surprised at my blunder and felt a sense of dread. I frantically ran from room to room asking if everything was working. What a relief to see that business was working normally!<br>Thanks for reading Terminal! Subscribe for free to receive new posts and support my work.
Subscribe
I immediately owned up to my manager about the sin. Fortunately he didn't fire me. But it was embarrassing enough for me, so I worked doubly hard in the next few weeks.<br>Why was everything fine?
Firstly Quartz tests are written well and drop only the Quartz tables.<br>drop table qrtz_fired_triggers;<br>-- other quartz tables<br>drop table qrtz_calendars;
Second, Quartz goes one step further, allows you to create your own database and prefixes all tables to avoid confusion:<br>use your_db_name_here; -- We had two databases, one for the scheduler and our business
String DEFAULT_TABLE_PREFIX = "QRTZ_";
Third, the incident was much worse in my head because of a flawed mental model. I assumed that teardown meant:<br>drop database
Learnings
I learnt that if we can directly connect to prod, then eventually someone will connect and run the tests, accidentally.
My manager had done a good job of configuring the environment; we had two databases for one for the regular business logic and the other for quartz scheduler.
A good manager puts the guardrails because juniors make mistakes.
How do others prevent this from happening?
Here’s two that I’ve encountered, sometimes used together.<br>1. Better user management
Generally the test databases have a test user who simply doesn’t exist on production. Hence there’s no question of running any invalid queries on production.<br>2. Tests hardcoding the testing database urls (my personal fav)
It makes sense to hardcode the database url for tests. This works for all manner of tests even if are using test-containers. This idea goes against modern devops culture, which expects that an application to read config from a .env file. But the devops culture can’t help when someone accidentally runs tests with the prod config.<br>(The only time this approach will fail is if we need parallelized tests that connect to different instances of the same database.)<br>Leave a comment<br>Thanks for reading Terminal! Subscribe for free to receive new posts and support my work.
Subscribe
Share
Discussion about this post<br>CommentsRestacks
Ready for more?
Subscribe
© 2026 chetan vashisht · Privacy ∙ Terms ∙ Collection notice<br>Start your SubstackGet the app
Substack is the home for great culture
This site requires JavaScript to run correctly. Please turn on JavaScript or unblock scripts