Byte order mark - Wikipedia

Jump to content

Search

Search

Donate

Create account

Log in

Personal tools

Donate

Create account

Log in

Byte order mark

22 languages

العربية<br>Čeština<br>Deutsch<br>Español<br>فارسی<br>Français<br>עברית<br>Italiano<br>日本語<br>한국어<br>Lietuvių<br>Malagasy<br>Norsk bokmål<br>Polski<br>Português<br>Română<br>Русский<br>Simple English<br>Slovenčina<br>Svenska<br>Українська<br>中文

Edit links

From Wikipedia, the free encyclopedia

Unicode character

"FEFF" redirects here. For other uses, see FEFF (disambiguation).

The byte order mark (BOM ) is a particular usage of the special Unicode character code, U+FEFF ZERO WIDTH NO-BREAK SPACE, whose appearance as a magic number at the start of a text stream can signal several things to a program reading the text:[1]

the byte order, or endianness, of the text stream in the cases of 16-bit and 32-bit encodings;

the fact that the text stream's encoding is Unicode, to a high level of confidence;

which Unicode character encoding is used.

BOM use is optional. Its presence interferes with the use of UTF-8 by software that does not expect non-ASCII bytes at the start of a file but that could otherwise handle the text stream.

Unicode can be encoded in units of 8-bit, 16-bit, or 32-bit integers. For 16- and 32-bit representations, a computer receiving text from arbitrary sources needs to know which byte order the integers are encoded in. The BOM becomes a noncharacter Unicode code point if its bytes are swapped. Hence, the process accessing the text can examine these first few bytes to determine the endianness, without requiring some contract or metadata outside of the text stream itself. Generally the receiving computer will swap the bytes to its own endianness, if necessary, and would no longer need the BOM for processing.

The byte sequence of the BOM differs per Unicode encoding (including UTF-8 and ones outside the Unicode standard such as UTF-7, see table below), and none of the sequences is likely to appear at the start of text streams stored in other encodings. Therefore, placing an encoded BOM at the start of a text stream can indicate that the text is Unicode and identify the encoding scheme used. This use of the BOM is called a "Unicode signature".

Usage<br>[edit]

The BOM is, simply, the Unicode codepoint U+FEFF ZERO WIDTH NO-BREAK SPACE, encoded in the current encoding. A text file beginning with the bytes FE FF suggests that the file is encoded in big-endian UTF-16.[2]

The name ZWNBSP (zero-width no-break space) should be used if the BOM appears in the middle of a data stream. Unicode says it should be interpreted as a normal codepoint (namely a word joiner), not as a BOM. Since Unicode 3.2, this usage has been deprecated in favor of U+2060 WORD JOINER.[1]

The Unicode 1.0 name for this codepoint is also BYTE ORDER MARK.[3]

UTF-8<br>[edit]

The UTF-8 representation of the BOM is the (hexadecimal) byte sequence EF BB BF.

The Unicode Standard permits the BOM in UTF-8,[4] but does not require or recommend its use.[5] UTF-8 always has the same byte order,[6] so its only use in UTF-8 is to signal at the start that the text stream is encoded in UTF-8, or that it was converted to UTF-8 from a stream that contained an optional BOM. The standard also does not recommend removing a BOM when it is there, so that round-tripping between encodings does not lose information, and so that code that relies on it continues to work.[7][8] The IETF recommends that if a protocol either (a) always uses UTF-8, or (b) has some other way to indicate what encoding is being used, then it "SHOULD forbid use of U+FEFF as a signature."[9] An example of not following this recommendation is the IETF Syslog protocol which requires text to be in UTF-8 and also requires the BOM.[10]

Not using a BOM allows text to be backwards-compatible with software designed for extended ASCII. For instance many programming languages permit non-ASCII bytes in string literals but not at the start of the file.

A BOM is not necessary for detecting UTF-8 encoding.[citation needed] UTF-8 is a sparse encoding: a large fraction of possible byte combinations do not result in valid UTF-8 text. Binary data and text in any other encoding are likely to contain byte sequences that are invalid as UTF-8, so existence of such invalid sequences indicates the file is not UTF-8, while lack of invalid sequences is a very strong indication the text is UTF-8. Practically the only exception is text containing only ASCII-range bytes, as this may be a non-ASCII 7-bit encoding, but this is unlikely in any modern data and even then the difference from ASCII is minor (such as changing '\' to '¥').

Microsoft compilers[11] and interpreters, and many pieces of software on Microsoft Windows such as Notepad (prior to Windows 10 Build 1903[12]) treat the BOM as a required magic number rather than use heuristics. These tools add a BOM when saving text as UTF-8, and cannot interpret UTF-8 unless the BOM is present or the file contains only ASCII. Windows PowerShell (up...