Unicode BOM Byte Order Mark Hack

Programer for Hire

Xah Lee, 2008-12-18, …, 2012-08-27

Some notes on Unicode's Byte Order Mark (BOM) character:

The BOM char's name is ZERO WIDTH NO-BREAK SPACE.
The BOM char codepoint is 65279 in decimal, U+FEFF in hexadecimal.
The primary purpose of BOM is to indicate byte-order (big endian vs little endian) in systems or situations that need this info in the file.

BOM is not needed for files encoded with UTF-8, since the smallest unit of UTF-8 encoding is a byte, so doesn't have the byte-order issue.
When used in UTF-8, it just give a indication that it is a file encoded with one of Unicode encodings (⁖ UTF-8, UTF-16, UTF-32).
You should not add BOM in UTF-8 encoded files.
The UTF-8 representation of the BOM is the byte sequence 0xEF,0xBB,0xBF. A text editor or web browser interpreting the text as ISO-8859-1 or CP1252, it will display it like this: ï»¿.

In unix-like OSes, BOM for UTF-8 conflicts with the unix shebang line hack. (i.e. the #!/usr/bin/python in first line.)
Some software from Microsoft (⁖ Notepad, Visual C++), will add BOM when a file is saved using UTF-8 encoding.
The BOM char's use as a zero-width no-breaking space is deprecated since Unicode 3.2 (published in 2002). That char's semantic is now for BOM only. “U+2060” (WORD JOINER) is now used for non-breaking space.

BOM is a Hack

Note that using BOM as first char to indicate a byte order is a hack. It's the same trick unix does with the Shebang scheme (i.e. first chars in file to be a #!, followed by a program path, to indicate that the text file is a executable program. So you can call commands without using the interpreter first, ⁖ processFile.pl fileName instead of perl processFile.pl fileName).

In both case of BOM and shebang, the first few bytes is used as a indicator for some particular meaning.

Similarly, in {Emacs, Python, Ruby}, if the first line has the form -*- coding: utf-8 -*-, it indicates that the file is UTF-8 encoded. Again, it creates a problem. For example, if a python script uses the shebang but is also UTF-8 encoded, what to put in the first line?

The real solution is a metadata file format. 〔☛ How to View Comments in JPEG, PNG, MP3 files?〕 But of course, hacks are created to solve practical problems at hand. Unix Shebang was there before there's Unicode. And Unicode BOM mark usage came before the exploration of matadata file format standards. Even today, metadata file formats are not widely used or standardized. Different file systems used by different OS may or may not have their own schemes for metadata, to various degrees of usage popularity, but they are usually not portable across file systems. As of today (2011-01-30), vast majority of files do not contain info about what encoding it is.

The word “Endianess” is from: Gulliver's Travels: PART 1, Chapter 4 — A VOYAGE TO LILLIPUT.