bytefork

~/blog

A UTF-8 BOM breaks JSON.parse but not fetch

published

#json#javascript#encoding

TL;DR

A UTF-8 BOM is three bytes — EF BB BF — decoded as U+FEFF. JSON.parse rejects it, because JSON’s whitespace set is only space, tab, LF and CR:

JSON.parse('\uFEFF{"a":1}');
// SyntaxError: Unexpected token '\uFEFF', "\uFEFF{"a":1}" is not valid JSON

The escapes are mine. A real terminal prints that message with nothing between the quotes, which is the entire difficulty of this bug.

fetch never shows you the problem, because UTF-8 decoding strips the BOM before your code sees the string. Reading the same bytes off disk with fs.readFileSync(path, 'utf8') keeps it. Fix: JSON.parse(text.replace(/^\uFEFF/, '')).

The problem

The file is fine. You can open it, it is valid JSON, your editor shows nothing unusual. The service that fetches it over HTTP has never complained. Then a script reads the same file from disk and dies on character zero.

Everything below is reproduced on Node v26.2.0.

import { readFileSync, writeFileSync } from 'node:fs';

const bytes = new Uint8Array([0xef, 0xbb, 0xbf, ...new TextEncoder().encode('{"a":1}')]);

// Over the network — works.
await new Response(bytes).json();          // { a: 1 }

// Off disk — throws.
writeFileSync('cache.json', Buffer.from(bytes));
JSON.parse(readFileSync('cache.json', 'utf8'));
// SyntaxError: Unexpected token '', "{"a":1}" is not valid JSON

The string you are holding is not the string you think you are holding:

const text = readFileSync('cache.json', 'utf8');
text.length;                      // 8, not 7
text.startsWith('{');             // false
text.codePointAt(0).toString(16); // 'feff'

That last line is the only reliable way to see it. The BOM is a zero-width character: it does not render, it rarely survives a copy-paste into a bug report, and console.log prints what looks exactly like the file you expected.

Why it happens

Four specs, each individually reasonable.

JSON does not allow a BOM. RFC 8259 §8.1 is explicit: implementations “MUST NOT add a byte order mark (U+FEFF) to the beginning of a networked-transmitted JSON text”, and parsers “MAY ignore the presence of a byte order mark rather than treating it as an error”. May. V8 chose to error, and it is within spec doing so. The whitespace a JSON parser must skip is defined in §2 as exactly four code points — space, tab, LF, CR. U+FEFF is not one of them.

UTF-8 decoding removes it. In the Encoding Standard, TextDecoder has an ignoreBOM option that defaults to false — and the name reads like the opposite of what it does. ignoreBOM: false means “do not ignore the BOM’s meaning”: consume it as an encoding marker and drop it from the output.

new TextDecoder().decode(bytes).codePointAt(0);
// 0x7b — '{'

new TextDecoder('utf-8', { ignoreBOM: true }).decode(bytes).codePointAt(0);
// 0xfeff

Every fetch body method runs that decode, which is why res.json() and res.text() both hand you clean text.

Node’s file APIs do not. readFileSync(path, 'utf8') and Buffer.prototype.toString('utf8') transcode bytes to a string with no BOM handling at all. That is the fork in the road: same three bytes, two different results, depending on whether they arrived over a socket or off a disk.

Windows still writes them. Windows PowerShell 5.1 emits EF BB BF for Set-Content -Encoding utf8, Out-File -Encoding utf8 and plain > redirection alike — checked on 5.1.26100. PowerShell 6+ changed the default to BOM-less UTF-8 (about_Character_Encoding), so a repo where some files were written by 5.1 and some by 7 has both shapes on disk. Excel’s “CSV UTF-8” export writes one too.

What to do

Strip it at the boundary

One line, at the point where bytes become a string:

const parseJson = (text) => JSON.parse(text.replace(/^\uFEFF/, ''));

Anchor the regex. A U+FEFF in the middle of a document is a legitimate zero-width no-break space, and deleting those changes the content.

Or decode the bytes yourself

If you are reading a file anyway, read it as a buffer and let the decoder do the work — same rule as fetch, no special-casing:

import { readFileSync } from 'node:fs';

const text = new TextDecoder().decode(readFileSync('cache.json'));
JSON.parse(text); // { a: 1 }

trim() also works, for a surprising reason

U+FEFF is <ZWNBSP>, which ECMAScript lists in its WhiteSpace production — so JavaScript’s string trimming removes it even though JSON’s whitespace rules do not:

'\uFEFF{"a":1}'.trimStart().codePointAt(0); // 0x7b

That is why a stray JSON.parse(text.trim()) elsewhere in the codebase has been quietly papering over this for months. Prefer the explicit strip: trim() also eats leading newlines you may have wanted to keep, and it hides which problem you were solving.

Detecting it when nothing throws

The failure is not always an exception. Feed BOM-prefixed text to a CSV parser and the BOM lands inside the first header name, so the column is id on screen and \uFEFFid in the object:

Object.keys(row);             // [ 'id', 'name' ] — looks fine
row.id;                       // undefined
Object.keys(row)[0] === 'id'; // false

Same three bytes, no error, a silently empty column. When a lookup by a literal string fails against a key that prints identically, check code point zero before checking anything else.

Other runtimes at least tell you. Python 3.14 names the problem and the fix in the message:

JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

utf-8-sig is Python’s BOM-aware codec — decode with it and the marker is consumed, exactly like TextDecoder does.

Caveats

References

Tools mentioned