Testing isn’t everything

Personal website of Martin Tournoij (“arp242 ”);<br>writing about programming (CV) and various other things.

Working on<br>GoatCounter and<br>more –<br>GitHub Sponsors.

Contact at<br>martin@arp242.net or<br>GitHub.

This is adopted from a discussion about<br>Want to write good unit tests in go? Don’t panic… or should you?<br>While this mainly talks about Go a lot of the points also apply to other languages.

Some of the most difficult code I’ve worked with is code that is “easily<br>testable”. Code that abstracts everything to the point where you have no idea<br>what’s going on, just so that it can add a “unit test” to what would otherwise<br>be a very straightforward function. DHH called this Test-induced design<br>damage.

Testing is just one tool to make sure that your program works, out of several.<br>Another very important tool is writing code in such a way that it is easy to<br>understand and reason about (“simplicity”).

Books that advocate extensive testing – such as Robert C. Martin’s Clean Code<br>– were written, in part, as a response to ever more complex programs, where you<br>read 1,000 lines of code but still had no idea what’s going on. I recently had<br>to port a simple Java “emoji replacer” (:joy: ➙ 😂) to Go. To ensure<br>compatibility I looked up the im­ple­men­ta­tion.<br>It was a whole bunch of classes, factories, and whatnot which all just resulted<br>in calling a regexp on a string. 🤷

In dynamic languages like Ruby and Python tests are important for a different<br>reason, as something like this will “work” just fine:

if condition:<br>print('w00t')<br>else:<br>nonexistent_function()

Except of course if that else branch is entered. It’s easy to typo stuff, or<br>mix stuff up.

In Go, both of these problems are less of a concern. It has a static type<br>system, and the focus is on simple straightforward code that is easy to<br>comprehend. Even for a number of dynamic languages there are optional typing<br>systems (function annotations in Python, TypeScript for JavaScript).

Sometimes you can do a straightforward implementation that doesn’t sacrifice<br>anything for testability; great! But sometimes you have to strike a balance. For<br>some code, not adding a unit test is fine.

Intensive focus on “unit tests” can be incredibly damaging to a code base. Some<br>codebases have a gazillion unit tests, which makes any change excessively<br>time-consuming as you’re fixing up a whole bunch of tests for even trivial<br>changes. Often times a lot of these tests are just duplicates; adding tests to<br>every layer of a simple CRUD HTTP endpoint is a common example. In many apps<br>it’s fine to just rely on a single integration test.

Stuff like SQL mocks is another great example. It makes code more complex,<br>harder to change, all so we can say we added a “unit test” to select * from foo<br>where x=?. The worst part is, it doesn’t even test anything other than<br>verifying you didn’t typo an SQL query. As soon as the test starts doing<br>anything useful, such as verifying that it actually returns the correct rows<br>from the database, the Unit Test purists will start complaining that it’s not a<br>True Unit Test™ and that You’re Doing It Wrong™.

For most queries, the integration tests and/or manual tests are fine, and<br>extensive SQL mocks are entirely superfluous at best, and harmful at worst.

There are exceptions, of course; if you’ve got a lot of if cond { q += "more<br>sql" } then adding SQL mocks to verify the correctness of that logic might be a<br>good idea. Even in those cases a “non-unit unit test” (e.g. one that just<br>accesses the database) is still a viable option. Integration tests are also<br>still an option. A lot of applications don’t have those kind of complex queries<br>anyway.

One important reason for the focus on unit tests is to ensure test code runs<br>fast. This was a response to massive test harnesses that take a day to<br>run. This, again, is not really a problem in Go. All integration tests I’ve<br>written run in a reasonable amount of time (several seconds at most, usually<br>faster). The test cache introduced in Go 1.10 makes it even less of a concern.

Last year a coworker refactored our ETag-based caching library. The old code was<br>very straightforward and easy to understand, and while I’m not claiming it was<br>guaranteed bug-free, it did work very well for a long time.

It should have been written with some tests in place, but it wasn’t (I<br>didn’t write the original version). Note that the code was not completely<br>untested, as we did have integration tests.

The refactored version is much more complex. Aside from the two weeks lost on<br>refactoring a working piece of code to … another working piece of code (topic<br>for another post), I’m not so convinced it’s actually that much better. I<br>consider myself a reasonably accomplished and experienced programmer, with a<br>reasonable knowledge and experience in Go. I think that in general, based on<br>feedback from peers and performance reviews, I am at least a programmer of<br>“average” skill level, if not more.

If an average programmer has trouble...