When Compilers Disagree About UTF‑8

Programming at the right level

SubscribeSign in

When Compilers Disagree About UTF‑8 Why a Simple ASCII Fast‑Path Tripled Clang’s Performance but Left GCC Unchanged

Nemanja Trifunovic Jul 26, 2026

In the summer of 2006 I started an open-source C++ library for handling UTF-8 strings. I wanted it to be portable and to work well with STL but did not go crazy with optimizations at the time. Recently, as I started writing about UTF-8 decoding, I spent more time revisiting the internals of the library. The function I decided to optimize decodes a UTF-8 encoded code point: template utf_error validate_next(octet_iterator& it, octet_iterator end, utfchar32_t& code_point) if (it == end) return NOT_ENOUGH_ROOM;

// Save the original value of it so we can go back in case of failure // Of course, it does not make much sense with i.e. stream iterators octet_iterator original_it = it;

utfchar32_t cp = 0; // Determine the sequence length based on the lead octet const int length = utf8::internal::sequence_length(it);

// Get trail octets and calculate the code point utf_error err = UTF8_OK; switch (length) { case 0: return INVALID_LEAD; case 1: err = utf8::internal::get_sequence_1(it, end, cp); break; case 2: err = utf8::internal::get_sequence_2(it, end, cp); break; case 3: err = utf8::internal::get_sequence_3(it, end, cp); break; case 4: err = utf8::internal::get_sequence_4(it, end, cp); break;

if (err == UTF8_OK) { // Decoding succeeded. Now, security checks... if (utf8::internal::is_code_point_valid(cp)) { if (!utf8::internal::is_overlong_sequence(cp, length)){ // Passed! Return here. code_point = cp; ++it; return UTF8_OK; else err = OVERLONG_SEQUENCE; else err = INVALID_CODE_POINT;

// Failure branch - restore the original value of the iterator it = original_it; return err;

Pretty straightforward: based on the value of the lead byte, determine the length of the sequence, and then depending on the length, construct the value of the code point by extracting appropriate bit fields from the bytes. After the code point is successfully decoded, perform two more checks: one for validity of the code point, and another for an overlong UTF-8 sequence. If both checks pass, return the success status, and the iterator is moved to the next sequence. An opportunity to optimization was the fact that ASCII trivially satisfies all UTF‑8 validity requirements. If it turns out the high bit of the lead byte is zero, we have an “ASCII character” - a value in the range of [U+0000, U+007F] which is always valid. All we need to do is zero-extend it and we have our code point. Something like this: template utf_error validate_next(octet_iterator& it, octet_iterator end, utfchar32_t& code_point) if (it == end) return NOT_ENOUGH_ROOM;

// Save the original value of it so we can go back in case of failure // Of course, it does not make much sense with i.e. stream iterators octet_iterator original_it = it;

utfchar32_t cp = 0; // Determine the sequence length based on the lead octet const int length = utf8::internal::sequence_length(it);

// Get trail octets and calculate the code point utf_error err = UTF8_OK; switch (length) { case 0: return INVALID_LEAD; case 1: err = utf8::internal::get_sequence_1(it, end, cp); // No need for further validations if (err == UTF8_OK) { code_point = cp; ++it; return UTF8_OK; } else { it = original_it; return err; break; // just to make sure there are no warnings case 2: err = utf8::internal::get_sequence_2(it, end, cp); break; case 3: err = utf8::internal::get_sequence_3(it, end, cp); break; case 4: err = utf8::internal::get_sequence_4(it, end, cp); break;

// Failure branch - restore the original value of the iterator it = original_it; return err;

My expectation was to see a visible improvement in handling of sequences of ASCII-only strings, with a minor impact on mixed ASCII/non-ASCII strings. Testing with clang 18.1.3 beat my expectations: For pure ASCII text, UTF-8 decoding throughput tripled, and even for mixed text, the improvement was substantial: around 34%.

I actually submitted the change to GitHub, but then at some point decided to test with gcc. Compilers just never stop surprising me: this time there was no difference for the ASCII text at all! Zero!

For heavily mixed text the throughput was consistently worse by 3-4%, which was not that surprising. Time to get our hands dirty and look at the generated assembly code. Here is what happens for an ASCII code point with original version compiled with g++: 15e0: ldrb w0, [x22] ; load byte 15e4: tbz w0, #7, 1700 ; if bit 7 == 0 (ASCII) return immediately ... 1700: add x22, x22, #1 ; consume 1 byte 1704: cmp x2, x22...

When Compilers Disagree About UTF‑8

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

NHS apologises and admits Palantir have access to identifiable patient data

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills