Unicode normalisation: why é can be one character or two

Two ways to write the same character

Unicode allows a character with an accent to be encoded either as a single code point — composed — or as a base letter followed by a combining accent — decomposed. Both are valid, both render identically in a font that supports them, and they are different byte sequences:

composed     é   U+00E9
decomposed   e + U+0301   (e, then COMBINING ACUTE ACCENT)

In UTF-8, the composed form is two bytes and the decomposed form is three. So the same word can be a different length depending on which keyboard or operating system produced it. This is not a bug; it is Unicode being explicit about the difference between a precomposed character and a base-plus-mark sequence.

The four normalisation forms

The distinction that matters: NFC/NFD are lossless — you can convert between them in either direction and get the same text back. NFKC/NFKD are lossy — they change characters that are merely visually or semantically compatible, so fi becomes fi, full-width Ａ becomes ASCII A, and superscript ² becomes 2. Using NFKC on content that should preserve typography will destroy it.

Why comparison breaks without it

A username stored as NFC and a login attempt typed on macOS as NFD compare unequal in every language, byte for byte, while looking identical. The fix is to normalise both sides to the same form before comparing. Which form you pick matters less than picking one consistently.

NFC("café") === NFD("café")   // false — different byte sequences
NFC(NFC(x)) === NFC(NFD(x))   // true  — both normalised to the same form

Note the asymmetry in danger: if you normalise only the stored value, the input still misses. Both sides must go through the same normalisation.

The macOS detail

HFS+ stored filenames in a variant of NFD. APFS is more permissive but still normalises for comparison. The practical consequence is a file whose name looks like café.txt but whose bytes are decomposed — and a build script that compares the name to a hard-coded string fails on one developer's machine. Normalising at the boundary (when reading a filename or a form value) is the durable fix.

Accent stripping is a normalisation, not a regex

The reliable way to remove accents is NFD followed by dropping the combining-mark range (U+0300–U+036F), then composing to NFC. A regex over a hand-written list of accented letters always misses something. The one important exception is characters that do not decompose — Vietnamese đ and Đ are separate letters, not d-plus-mark, so they must be mapped explicitly.

"Việt Nam".normalize("NFD").replace(/[\u0300-\u036F]/g, "")  // "Viet Nam"
"Đà Nẵng".normalize("NFD")                                     // đ survives intact

What changes when you normalise

• Byte length changes — NFD can be longer, NFC shorter. If a database column has a byte limit, normalise before you validate the size, not after.
• Character count changes for the same reason — a word can be 4 or 6 code points.
• Grapheme count does not change for the accent case, because a base plus combining mark and the precomposed character are both one visible character.
• Identifiers can collide — that is the point: two encodings of the same name become one key.

Practical rules

1. Normalise to NFC at every system boundary — form input, file read, API payload.
2. Normalise both sides of every comparison, never just one.
3. Use NFD only for processing (stripping, analysis), then compose back to NFC for storage and display.
4. Avoid NFKC on user content; use it deliberately for search keys and identifiers, where collapsing lookalikes is the goal.
5. Normalise before measuring if the limit is enforced in bytes or code points.

Case study: a login that fails for one user

This is the most common way normalisation reaches a bug tracker. A user registers from a Mac, where the browser or the OS may leave a name in decomposed form. The server stores it as received. The same user later logs in from Windows, types the identical-looking name, and the comparison fails. Nothing in the logs shows a difference — both strings print identically.

The fix has to happen at both ends. If you normalise only on write, existing rows stay in whatever form they were stored in, so a migration is needed: normalise the column to NFC, then add the normalisation at the write and compare paths so it cannot drift again. The migration is the part people skip, and it is why the bug reappears months later on the accounts that were created before the fix.

-- the shape of the migration, not a dialect-specific statement
UPDATE users SET username = NFC-normalise(username);
-- then enforce it at the boundary so it cannot recur

Case study: a filename that a build script cannot match

A CI step reads filenames and compares them to a hard-coded list. It passes on Linux and fails on macOS for one file. The file's name looks correct in the shell, but its bytes are decomposed while the hard-coded string is composed. The check is an exact byte comparison, so it misses.

The durable fix is to normalise the filename as soon as it is read, name.normalize('NFC') before any comparison, and to keep the hard-coded comparison values in NFC too. Comparing paths is exactly the kind of code where “it looked the same” is not evidence.

Sorting is affected too

Byte order is not alphabetical order, and normalisation changes byte order. Two strings that differ only in composition sort differently under a plain byte comparison even though they represent the same word. For user-visible sorting, use a collator (Intl.Collator) rather than <; it applies locale-aware comparison and treats equivalent forms as equal. For stable machine sorting, use NFD before comparing — a common convention precisely because it makes ordering independent of how the text was produced.

Regex is affected as well

A pattern that matches an accented character literally matches only the composed form. Run it against decomposed input and it misses. This is why accent-stripping is done as NFD-then-remove-marks rather than with a character class: the marks are a contiguous range (U+0300–U+036F) after decomposition, and no range exists before it. If you must match either form without normalising, write the pattern over the base character and make the marks optional — but normalising first is simpler and less error-prone.

What normalisation does not fix

• Homoglyphs. Cyrillic а and Latin a are different characters with different meanings; normalisation will not merge them. Only NFKC collapses compatibility lookalikes, and only some of them.
• Invisible characters. A zero-width space is a distinct character and stays distinct through every normalisation form.
• Whitespace differences. U+00A0 is not U+0020 and is not normalised away.
• Case. Normalisation is not case folding; ß and ss are a case-mapping question, not a composition one.

The practical takeaway: normalisation makes equivalent encodings comparable. It does not make different characters equal, and it should not — that would be a security hole in the other direction.

Check it yourself

Paste both forms into the code point inspector to see the difference in code points, or the character counter to see it in bytes. The Vietnamese accent tool demonstrates the NFD-plus-mapping approach on the hardest Latin-script case.