Byte order mark

lastmod 2023-04-10created 2016-04-136 edits

The Byte Order Mark (BOM) is a character used in Unicode to distinguish endianness; its character value is U+FEFF.

In encodings such as UTF-16 and UTF-32, the values of a string differ completely depending on the endianness, so a mark that can distinguish the string's endianness is needed. Accordingly, a BOM character is prepended to a Unicode string, and the endianness is determined by the value read first.

For example, in UTF-16, the first two bytes of the string are FE FF for big endian and FF FE for little endian.

BOM

The BOM value for each Unicode encoding is as follows.

How to check the BOM in VIM

For reference, checking it in vim looks like this:

It is displayed as <feff>, which looks like a UTF-16 big-endian BOM, but

if you check it with xxd (the vim command is :%!xxd), the BOM bytes appear in the first three bytes as follows:

Since it is ef bb bf, it is a UTF-8 BOM.

So beware of the BOM display error in VIM.

UTF-8 BOM removal script

A bash script that removes the UTF-8 BOM (ef bb bf) is as follows:

#!/bin/bash

# 디렉토리를 재귀적으로 탐색하면서 파일 목록을 얻는 함수
function find_files() {
    local dir=$1
    find "$dir" -type f
}

# 파일의 BOM(Byte Order Mark)을 제거하는 함수
function remove_bom() {
    local file=$1
    if [[ $(head -c3 "$file") == $'\xef\xbb\xbf' ]]; then
        tail -c +4 "$file" > "${file}.tmp"
        mv "${file}.tmp" "$file"
    fi
}

# 디렉토리에서 파일을 찾아서 BOM을 제거하는 함수
function remove_bom_recursive() {
    local dir=$1
    for file in $(find_files "$dir"); do
        remove_bom "$file"
    done
}

# 스크립트 실행 시작
if [[ -n "$1" ]]; then
    # 첫 번째 인자로 전달된 디렉토리에서 BOM 제거 시작
    remove_bom_recursive "$1"
else
    # 현재 디렉토리에서 BOM 제거 시작
    remove_bom_recursive .
fi

Favorite site

• Wikipedia (en) BOM