Why invisible characters exist at all
Unicode is not only a list of printable symbols. It also carries characters whose job is to affect layout, joining, direction or line breaking — which means they have no glyph of their own. That is correct and necessary for rendering many scripts. It is also why a copied string can carry bytes no one intended to type.
The usual suspects
| Character | Code point | Looks like | Common source |
|---|---|---|---|
| Zero-width space | U+200B | Nothing | Word processors, some CMS editors, copy from a web page |
| Zero-width no-break space (BOM) | U+FEFF | Nothing | Start of a UTF-8 file saved by certain tools |
| Non-breaking space | U+00A0 | A space | HTML , Word, French typography |
| Narrow no-break space | U+202F | A thin space | Copy from a formatted document |
| Soft hyphen | U+00AD | Nothing (or a hyphen at a line break) | Word documents hyphenated for layout |
| Zero-width joiner | U+200D | Nothing | Emoji sequences, Indic scripts |
| Left-to-right mark | U+200E | Nothing | Bidirectional text copied from a UI |
| Word joiner | U+2060 | Nothing | Copy from some editors |
The bug patterns
Exact-match lookups fail for one user
A user types their email; the app trims whitespace and compares. It fails. The string looks the same in
every log and on every error page. The cause is usually a U+00A0 or U+200B pasted along with the value
from a document or a password manager that copied formatting. trim() in most languages
removes ASCII whitespace and, depending on the runtime, may or may not remove U+00A0 —
String.prototype.trim in JavaScript does handle it, but many bespoke trim helpers use
/^\s+|\s+$/ which historically did not.
A BOM breaks the first CSV header
A UTF-8 file saved with a BOM starts with EF BB BF. If the reader does not strip it, the
first header is id rather than id — and every lookup of id
misses on that one file.
Two strings that look identical, compare unequal
This is the homoglyph problem: Cyrillic а and Latin a, or a Greek
ο and a Latin o. They render nearly identically and are different code
points. It matters for identifiers, domain names and any place a value is compared literally.
Line breaks that are not \n
Besides \r\n, text copied from some sources contains U+2028 (line separator)
and U+2029 (paragraph separator). They look like ordinary line breaks and behave differently
in parsers, template engines and JSON (where they must be escaped).
Finding them
An ordinary editor will not show you these, and len() will not tell you which character is
which. The reliable check is to look at the code points: a zero-width character has a code point but no
glyph, so an inspector that lists every character will surface it immediately. The
code point inspector does exactly this and flags the invisible ranges.
"admin" vs "admin" # the second has a trailing zero-width space
hex: 61 64 6d 69 6e vs 61 64 6d 69 6e e2 80 8bCleaning them safely
Blanket removal is dangerous. Some of these characters are load-bearing: a zero-width joiner is what makes an emoji sequence render as one glyph, and the bidi marks are required for correct display of Arabic and Hebrew. Removing them unconditionally corrupts legitimate text.
- For a single field (email, username, id), strip the known problem characters — U+200B, U+FEFF, U+00A0 — and normalise with NFC. These should never appear in an identifier.
- For prose, do not strip joiners or bidi marks. Normalise to NFC instead, which unifies composed and decomposed accents without changing visible content.
- For a BOM, strip it once at read time, at the start of the file only — a U+FEFF anywhere else is content.
- For comparison, normalise both sides (NFC) and, if the input may contain lookalikes, decide deliberately whether to map them.
How to find them without an inspector
An inspector is the reliable tool, but there are quick checks you can run in a shell when you cannot paste the string into a browser.
# show non-ASCII bytes in a file
grep -nP '[^ -]' file.txt
# show the hex of a suspicious line
sed -n '42p' file.txt | xxd | head
# count zero-width spaces specifically (UTF-8: e2 80 8b)
grep -c $'â' file.txt
# find a no-break space (c2 a0) — often mistaken for an ordinary space
grep -n $'Â ' file.txt
The output tells you the byte sequence, not the character name. That is enough to decide: if a line
that should be pure ASCII contains a byte above 0x7F, something invisible or accented got
in.
Zero-width characters in code and config
The worst place for an invisible character is a source file or a config, because it can make code that looks correct behave differently. A zero-width space inside an identifier produces a name that is not the one you meant, so a lookup fails while the source reads normally. A non-breaking space in an indented config (YAML, Python) is not whitespace to the parser, so the indentation level is wrong in a way the editor does not show.
This is why many editors ship a “show invisibles” mode and why linters flag non-ASCII in source. If a YAML file fails with an indentation error that you cannot see, check for U+00A0 before rewriting it.
What each character actually does
| Character | Effect when present | Why it exists |
|---|---|---|
| U+200B zero-width space | Allows a line break; counts as a character in comparisons | Let browsers break long words without a hyphen |
| U+200C zero-width non-joiner | Prevents a ligature or join | Correct rendering in Persian, Indic scripts |
| U+200D zero-width joiner | Joins the next character into the same glyph | Emoji sequences, Indic conjuncts |
| U+FEFF BOM | Ignored at the start of a file; a stray character elsewhere | Marks UTF-8 encoding for legacy readers |
| U+00A0 no-break space | A space that does not break at a line end | Typography: keep a number with its unit |
| U+00AD soft hyphen | Invisible unless the line breaks there | Optional hyphenation points in long words |
| U+202A–U+202E bidi controls | Change the direction of following text | Mixed left-to-right and right-to-left text |
Notice the pattern: almost every one of these is load-bearing for some text. That is why the fix is “strip them from fields where they cannot be meaningful” rather than “strip them everywhere”.
Where they come from
In practice, invisible characters arrive through three routes. Copying from a rendered document — a web page, a PDF, a Word file — carries whatever the layout engine inserted, including soft hyphens and no-break spaces. Copying from a UI that formats values — a dashboard that renders an email in a styled span may include a zero-width space for line breaking. Generated files — some exporters emit a BOM, and some libraries insert zero-width spaces to control wrapping.
A useful debugging instinct: when a value fails an exact match and looks correct, paste it into an inspector before checking the matching logic. The logic is usually right; the string is not what it appears to be.
Sanitising a field without breaking the text
There is no universal “remove invisibles” step, because the right set depends on what the field holds. A reasonable policy for an identifier field — username, email, a key — is to normalise to NFC and then remove everything in the format and invisible-operator categories, plus the no-break spaces. For prose, do the opposite: keep the joiners and bidi marks, and only normalise. The distinction is whether the value is meant to be read or compared.
# identifier policy, sketched
s = s.normalize("NFC")
.replace(/[\u200B-\u200F\u2060\uFEFF]/g, "") # zero-width + bidi marks
.replace(/[\u00A0\u202F\u2007\u2009]/g, " ") # exotic spaces -> normal space
.trim()Note that this turns no-break spaces into ordinary spaces rather than deleting them. Deleting them would join two words that a user separated; converting keeps the separation and discards the typography, which is usually what an identifier wants.
The BOM, specifically
A BOM is the one invisible character with an official role: at the very start of a UTF-8 file it identifies the encoding for readers that would otherwise guess. The correct handling is to strip exactly one occurrence at offset zero when reading, and to leave any other U+FEFF alone — a U+FEFF in the middle of a document is a zero-width no-break space, which is content.
This is why “strip BOM” implemented as a global replace is wrong: it deletes meaningful characters from the body. It is a small bug with a large blast radius, because it silently alters text that passed through the sanitizer.
Check it yourself
The whitespace cleaner removes the invisible and exotic whitespace characters and reports what it removed, without touching joiners or bidi marks. The UTF-8 inspector shows the bytes of every character, and the code point inspector names them.