Are CSV character corruption issues caused by BOM? Investigating from the first 3 bytes in Python

About this article
This article was created using an automated generation workflow leveraging generative AI. Based on the character code specifications of the Python standard library and CSV handling, we have organized the procedure for safely inspecting UTF-8 BOM.

Verification status: 📘 Official specifications checked – execution of the published code on a real machine has not been verified

When opening a CSV file in Excel, Japanese characters are garbled. In such cases, jumping to the conclusion that it is due to a missing UTF-8 BOM is premature.Inspecting the first 3 bytes will reveal the presence or absence of a UTF-8 BOM, but the absence of a BOM alone does not definitively determine the cause of character corruption.The shortcut is to investigate the actual character encoding, how it is opened, and the delimiters separately.

First, observe the three bytes.

The UTF-8 BOM (Byte Order Mark) is a 3-byte marker placed at the beginning of a file. In hexadecimal, it is EF BB BF. In Python, you can compare it using the constant codecs.BOM_UTF8.

The following is an example that sample.csvsimply reads a file named . Please replace the filename with your own test copy.

from pathlib import Path
import codecs

path = Path("sample.csv")
with path.open("rb") as f:
    first_three = f.read(3)

print("先頭3バイト:", first_three.hex(" ").upper())
print("UTF-8 BOMあり:", first_three == codecs.BOM_UTF8)

If it is UTF-8 with BOM, EF BB BF and True are expected to be displayed, whereas for a typical UTF-8 without BOM CSV, different 3 bytes and False will be displayed. However, an error will not occur even if the file is smaller than 3 bytes; it will compare only the bytes that could be read. This is anexpected value derived from the specification, not a log executed during writing.

What can be deduced from the observation results?

A BOM is an identifier at the beginning of a file and does not guarantee the validity of the entire string. UTF-8 can be used normally without a BOM. Conversely, even with a BOM, invalid byte sequences may be mixed in the middle.

Next, verify whether it can be read as UTF-8.

Regardless of the presence of a BOM, check whether it can be decoded. First, read it strictly as UTF-8, and if an issue occurs, raise an exception.

from pathlib import Path

path = Path("sample.csv")
raw = path.read_bytes()

try:
    text = raw.decode("utf-8-sig", errors="strict")
    print("UTF-8としてデコード可能")
    print("先頭の内容:", repr(text[:80]))
except UnicodeDecodeError as error:
    print("UTF-8としてデコード不可:", error)

utf-8-sig skips the UTF-8 BOM if present at the beginning, or reads it as standard UTF-8 if absent. This is not a process to remove the BOM and overwrite the original file.

Even if this succeeds, it does not guarantee that the original Japanese text is as intended. For example, if a string that was already incorrectly converted to another character encoding is saved as UTF-8, decoding may succeed while the contents remain corrupted. Always visually inspect the Japanese column names and a few rows of values.

Pay attention to newline settings when reading as CSV.

After verifying the character encoding, parse the CSV using split(",") instead of csv.reader in order to properly handle commas and newlines inside quoted strings.

import csv

with open("sample.csv", "r", encoding="utf-8-sig", newline="") as f:
    reader = csv.reader(f)
    for index, row in enumerate(reader):
        print(row)
        if index >= 4:
            break

This is a reading example to check the first 5 rows. If the CSV delimiter is a tab or semicolon, modify it such as csv.reader(f, delimiter="\t") to match the actual file specifications. If \ufeff is visible at the beginning of a line, verify whether the BOM is being read as a character.

Make one change: Compare utf-8 and utf-8-sig

Open the same file with encoding="utf-8" and encoding="utf-8-sig", and compare the first header string. For a file with a BOM, the former preserves the leading U+FEFF as a character, whereas the latter reads the file with the BOM removed. For UTF-8 without a BOM, both generally result in the same content.

However, how applications such as Excel determine CSV formatting depends on the product version and import method. It cannot be guaranteed that simply adding a BOM will resolve character corruption in every environment. In Excel, verify separately by double-clicking the file and by explicitly specifying the character encoding during data import.

Troubleshooting in Case of Failure

Always verify operational files using a copyPlease do so. It is also important not to upload CSV files containing personal information to online conversion sites. Because character encoding conversion may result in data loss, save the original, converted file, and conversion conditions separately, and use the data only after comparing the number of columns, record count, and representative Japanese values.

Common Misconceptions

Q. If it lacks a BOM, is it not UTF-8?No. A BOM is not mandatory for UTF-8.

Q. Does a BOM guarantee that Excel will open it correctly?No. It depends on the application, version, import method, and the contents of the CSV.

Q. Can Shift_JIS be identified?Not by looking at the first 3 bytes alone. Check the specifications of the source system, and verify representative Japanese text and the decoding results. Furthermore, CP932 used in Japanese Windows and standard Shift_JIS are not identical.

Understanding with a flowchart: Investigation continues even after identifying the BOM

If the first 3 bytes of the CSV are EF BB BF, it can be determined that a UTF-8 BOM is present. However, this does not prove that all subsequent data is valid UTF-8. Conversely, the absence of a BOM does not rule out UTF-8. Separating what can be known from the initial observation and what remains unknown helps prevent misdiagnosing the cause.

flowchart TD
    A["元CSVのコピーを用意"] --> B["先頭3バイトを読む"]
    B --> C{"EF BB BFか"}
    C -->|"はい"| D["UTF-8 BOM付きとして厳密に復号"]
    C -->|"いいえ"| E["UTF-8として厳密に復号"]
    D --> F{"文字として読めたか"}
    E --> F
    F -->|"はい"| G["列名・日本語の値・CSVの構造を比較"]
    F -->|"いいえ"| H["作成元の文字コード仕様を確認"]
    G --> I["Excel等の取り込み方法を調べる"]
    H --> I
    I --> J["根拠に基づき設定・変換を判断"]

In the flowchart, the branch "Decryption successful" is not the same as "Restored to correct Japanese meaning." If a string that was already corrupted in a previous process is saved as UTF-8, decoding itself will succeed, but the original meaning will not be recovered. In practical operations, it is necessary to compare column names, record counts, and representative values with the output of the source system.

Observe the differences using two types of CSV created by yourself

There is no need to use customer data received in the course of business as study material. By using only fictitious items and quantities to create two files with identical content—one with a BOM and one without—you can safely compare the differences in the leading bytes and data content. The following code is an example of creating a new file, ensuring it does not overwrite an existing file with the same name.The provided code has not been freshly executed this time, and the results are expected.

from pathlib import Path

text = "商品,数量\n青りんご,2\nみかん,3\n"
plain = Path("sample-utf8.csv")
bom = Path("sample-utf8-bom.csv")

for path in (plain, bom):
    if path.exists():
        raise FileExistsError(f"同名のファイルがあります: {path}")

plain.write_bytes(text.encode("utf-8"))
bom.write_bytes(text.encode("utf-8-sig"))
for path in (plain, bom):
    data = path.read_bytes()
    print(path.name, data[:3].hex(" ").upper(), len(data))

In this example, while both CSVs can be handled as UTF-8, the file with the BOM has a 3-byte marker at the beginning. Therefore, if the line breaks and body text are completely identical, the file size should be 3 bytes larger. Even if a Japanese character appears as a single character on the screen, it may be represented by multiple bytes in UTF-8. Consequently, the total byte count of the file and the character count on the screen are different values. When comparing results, distinguish between "content," "character count," and "byte count."

Does having a BOM mean that Excel will always open the file correctly? Not necessarily. The reading conditions can differ between opening the file by double-clicking and specifying the character encoding in the data import screen. Furthermore, because behavior varies depending on the Excel version and settings, you need to record which operation produced the expected display.

Character Corruption and Column Misalignment Are Different Failures

The areas to investigate differ depending on whether only the Japanese text display is corrupted or if the Japanese is readable but the columns are shifted to the right. Because CSV allows commas and line breaks within quoted fields, processing that simply splits strings by commas may not read them correctly. Examining the first 3 bytes does not reveal the rules for delimiters and quotes.

For example, if a fictitious cell contains "Tokyo, Shinagawa" and is enclosed in quotes, the comma inside is not necessarily a column boundary. Even if the character encoding is specified correctly, misinterpreting the quotes will result in a mismatched number of columns. While Python's csv.reader can interpret the data as CSV, the conditions such as the delimiter used by the imported file must be checked against the source system specifications.

Things to Agree Upon When Exchanging Data in Business Operations

In CSV operations, the content format may change even if the file name remains the same. Establishing agreements between the creator and the receiver regarding character encoding, the presence of a BOM, line breaks, delimiters, quotes, and the presence of a header will speed up incident investigation. If character corruption occurs, keep the original file and record which software and operation caused the phenomenon.

Programs that allow the receiving side to sequentially try multiple character encodings may seem convenient, but they can sometimes accidentally decode data using an incorrect encoding. Do not judge correctness based on appearance alone; for important data, verify the source system specifications and representative values. If conversion is necessary, use the data only after cross-checking the converted column names, record counts, and Japanese text samples.

Avoid sending CSV files containing confidential information or customer data to online character corruption diagnostic sites. It is safer to first perform a read-only investigation on your own device and organize the conditions that can be reproduced using dummy data. Rather than ending the discussion with "it's because there's no BOM," separating the character encoding, the CSV structure, and the reading application's settings is useful in practice.

Summary

When investigating CSV character corruption, first examinethe presence or absence of a BOM in the first 3 bytes, next examinestrict decoding and content verification, and finally separatethe CSV column structure and the application-side reading method. Not overwriting the original file before identifying the cause is the greatest safeguard in practical operations.

Official Information and Primary Sources

• Python codecs: BOM constants and utf-8-sig

• Python csv: newline specification and reader

• Python Unicode HOWTO: Encoding and Decoding

* The execution examples in this text are based on the official specifications and do not guarantee success in any specific Windows or Excel environment.