Twenty Years of my Open Source Project
Programming at the right level
SubscribeSign in
Twenty Years of my Open Source Project<br>Unicode, C++, and the Long Life of utfcpp
Nemanja Trifunovic<br>Aug 13, 2026
Share
In the fall of 2005, my employer was acquired by another company, and I was transferred to the team that developed a centralized translation memory system. The group had a strong not-invented-here culture, which I did not appreciate at the time1. One example of functionality that, in my opinion, should have been reused rather than rewritten was the handling of UTF‑8 encoded text. I looked around to find some portable C++ open-source libraries to prove my point, and to my astonishment, there was almost nothing. There was only ICU, which started as a Java library and then was ported to some very non-idiomatic C++. I wanted something light, portable, and unintrusive that plays well with the C++ Standard Library.<br>Beginnings
In the summer of 2006, while my family was away for a few weeks2, I got busy. The primary design goals I came up with were:<br>Portability. I wanted it to run on any reasonably C++98 compliant compiler and be OS agnostic.
Flexibility. It was meant to be easy to use with as many existing C++ string classes as possible and play well with STL and Boost.
Ease of use. It was going to be easy to install, easy to learn, and surprise-free.
Safety. It had to catch malformed data by default.
The first version was coded quickly. I was lucky to find Markus Kuhn's UTF-8 decoder capability and stress test file early, and testing against it gave me confidence that I covered the edge cases.<br>Initially, I had envisioned the library as a Code Project3 article, so the documentation was almost as important to me as the code itself. The article helped me get the first users and the first round of feedback.<br>Naming the library was confusing. Initially, I called it UTF-8 C++, which landed as utfcpp in most URLs. Then it became more common to refer it as utfcpp, and after I extended it to support UTF-16 related operations, it actually became a better name. In any case, I should have thought about the name harder. The title of the initial Code Project article was excellent, though: UTF-8 with C++ in a Portable Way. People would often search for something like “C++ utf-8 portable” and it would bring them directly to utfcpp.<br>Soon, I realized I had a little more than an article and decided to set up an open-source project. The choice of the hosting platform was easy - SourceForge was pretty much the only option back then, and it supported Subversion, my source control system of choice at the time4. SourceForge was pretty rough around the edges: setting up builds, tests, releases and the home page for the project was not a pleasant experience. That said, once I did set it up and linked it from my Code Project article, the experience was much better.<br>Choosing the license did not give me any headaches either. I’ve always disliked restrictive “copyleft” ones such as GPL. On the other hand, I was a fan of The Boost C++ Libraries which I used and advocated at work at the time5, so I picked Boost Software License - Version 1.0 which I found simple and straightforward, while still offering legal protection to the author.<br>I only vaguely remember the original build and test framework. There was a lot of Perl, which I used mostly as a portable shell to run builds, pack releases and run tests. Unit-tests were present from the very beginning, and I even think I started the project using Test Driven Development6 with simple asserts instead of any “proper” unit-test frameworks.<br>Growth
Encouraged by the initial reception of the library at Code Project, I decided to submit it to Boost. To my surprise, the process was informal and somewhat confusing. I was supposed to post a proposal to the Boost mailing lists and hope that someone important enough picks it. So I posted the proposal in July 2006, got a few generally favorable reactions, but then nothing but silence. Then, in December, I tried again. This time there was some technical discussion about appropriateness of exceptions as a mechanism for reporting invalid UTF-8 sequences, but when the discussion was over, I was left hanging again. No one rejected my proposal; it was simply ignored. So, I gave up and decided to run the library as a standalone project.<br>Perhaps it was for the best. I kept chugging along slowly, adding an occasional feature now and then and fixing bugs when they were reported. Sometimes, multiple months would pass without me doing any work on utfcpp, and it never felt bad as long as the library was in good shape and there were no outstanding bugs.<br>In the beginning, I expected the C++ Standard would eventually include a good way of dealing with Unicode encodings and my library would be retired. Interestingly, it did not happen. C++11 standard included a mechanism called codecvt that at least covered conversion between the encodings, but even that was...