Whats the Max Valid Length (in bytes) of an Emoji? | ma[ch|x]s'space

Introduction

Recently I stumbled upon a Twitter conversation

If I want to store a single emoji in a Postgres field, what is the LIMIT I should set on the TEXT field?

which talked about storing a single emoji in a Postgres TEXT field, with a LIMIT.

The answer to this question is already stated in the Twitter conversation, which is 35 bytes. While this answer in this context is correct, it is a bit short. In fact we also need to specify the encoding, which is the Unicode 15 standard in this case, and how the data is stored, which is UTF-8.

While in this blog post, I do not want to discuss the performance implications, of changing the database field, I find the question of how many bytes a single emoji can have very interesting, as there are several aspects to it. So this blog post will be about character encoding, the Unicode standard, and emojis. Such that at the end we can hopefully state and understand the answer to the question “What’s the Max Valid Length (in bytes) of an Emoji?”.

Character encoding

To first clear up some definitions and speak about some backgrounds, we will start with character encoding in general. A character is any input that is required to express text, which includes non-visible characters, e.g. “ZERO WIDTH NO-BREAK SPACE” or characters that are not interpreted like a single glyph, e.g. \n.

Let us start with the smallest unit which can be used to represent data, the bit. A bit can have two values, which are usually labeled $0$ and $1$. These two states are realized on hardware which can have two distinct states, that depending on the use case, can be easily changed, read, or both. For example, an HDD uses the electron spin to store data, flash memory uses special transistors, called [metal–oxide–semiconductor field-effect transistors (MOSFET), and so on.

As this two-state representation is good for hardware, but not so good to work with data from a software perspective, another data unit has been introduced, the byte. A byte, in the historical sense, was the number of bits needed to encode a single character. Nowadays, when we speak of a byte without the specification of the number of bits, we usually mean a byte consisting of $2^3 = 8$ bits, also sometimes referred to as an “octet”. But as we will see, one byte will not be sufficient to encode all needed characters.

The number of bits needed to encode a character depends on how exactly the character is translated into bits, which is specified by the used character encoding. Of course to translate the bit series back into character, one needs to know the encoding, otherwise, the bit series is interpreted as a different character.

Unicode encoding

For the rest of this article, we will focus on Unicode encoding, which is a specific rule set of how characters are mapped to bit series. The Unicode standard is for the Linux world and the web, the commonly used encoding of text, and which we need to further understand to answer the initial question.

The Unicode encoding model consists of two separate parts. One part defines how characters map to so-called code points and the other part defines how these code points map to bits.

Code points

A Unicode code point, denoted by U+, can be translated using the Unicode character code mapping, where is a hexadecimal number, from $0000_{\mathrm{hex}}$ to $10\mathrm{FFFF}_{\mathrm{hex}}$. In general, one would omit the leading zeros, e.g. $0000 = 0$, but in the Unicode standard, the digits have different meanings depending on the position.

In Unicode, the hex numbers have the following format: $[0-10]_{\mathrm{hex}} [000 - \mathrm{FFFF}]_{\mathrm{hex}}$. Where the leading group, with $17$ possible values $[0 - 10]_\mathrm{hex} = [0 - 16]_\mathrm{dec}$, specifies the so called Unicode plane. The second group specifies the code point inside the specified plane, which allows specifying $2^{16} = 65\,536$ unique values, $[000 - \mathrm{FFFF}]_\mathrm{hex} = [0 - 65\,535]_\mathrm{dec}$. Note that for values below three digits, one explicitly adds leading zeros, such that the number of digits is at least three, as there are otherwise collisions of the Unicode code points. So in total, there exist $17*2^{16} = 1\,114\,112$ available code points in the Unicode standard, from which the Unicode 15 standard currently defines $149\,186$. For example, U+0041 represents the character A and U+0061 stands for a.

Using U+0000-10FFFF or U+0000..U+10FFFF we denote a range of Unicode code points, in this example, it is the full Unicode range. When we want to write a sequence we write U+0041 U+0042 U+0043 (ABC), where the spaces between the Unicode code points are just for readability.

The Unicode standard furthermore defines so-called Unicode blocks, without a fixed size, which are commonly used together. For a list refer to Wikipedia or the official specification.

Storing code points as...