Four kinds of damage
Copied text arrives with four independent problems, and each has a different fix. Invisible characters occupy no width but are real code points: a zero-width space, a byte-order mark, a soft hyphen. Whitespace variants look like a space and are not one: the non-breaking space that keeps a price and its currency together, the thin space used in typography. Line structure carries the source's wrapping: hard breaks from a PDF, CRLF from Windows, trailing spaces an editor left behind. Glyph substitutions change the character itself: curly quotes for straight ones, ligatures for letter pairs, full-width forms for ASCII.
A lookup fails when the stored value and the query differ in any of these. The visible text is not the data; the code points are. That is why the first diagnostic step is always the same: inspect the code points, not the rendering.
What JavaScript's trim() actually removes
String.prototype.trim() removes characters with the Unicode WhiteSpace property plus the
line terminators. That covers more than ASCII spaces — and less than people assume. Measured on the
strings below (one invisible character placed before an a):
| Character | Code point | UTF-8 bytes | trim() removes |
|---|---|---|---|
| Non-breaking space | U+00A0 | 2 | Yes |
| Zero-width space | U+200B | 3 | No |
| Zero-width non-joiner | U+200C | 3 | No |
| Zero-width joiner | U+200D | 3 | No |
| Soft hyphen | U+00AD | 2 | No |
| Byte-order mark | U+FEFF | 3 | Yes |
| Em space | U+2003 | 3 | Yes |
| Thin space | U+2009 | 3 | Yes |
Measured in Node 22. The result is the same in browsers because the property is specified by ECMAScript, not by the engine.
The practical reading: a trimmed string can still contain a zero-width space, because trim only touches the ends and only for whitespace-property characters. A value that looks trimmed and still fails an exact comparison is usually holding a U+200B or U+00AD in the middle.
Where the characters come from
The source predicts the damage. Web pages and CMS editors introduce non-breaking spaces to keep prices and names together, and zero-width spaces to allow breaks inside long identifiers. Word processors introduce curly quotes, ligatures and soft hyphens as part of typesetting. Windows tools and CSV exports introduce a byte-order mark and CRLF endings. PDF extraction introduces hard line breaks at the typeset margin and sometimes a space where a glyph was positioned. Knowing the source tells you which pass to run first, and it also tells you which characters are intentional.
A cleaning order that works
- Normalise line endings first. Convert CRLF and CR to LF. A file with mixed endings produces phantom empty lines in line tools and a stray carriage return in every field of a CSV.
- Remove only the invisibles you did not intend. U+200B, U+FEFF, U+00AD and non-standard spaces are safe to remove in most prose. Do not remove U+200D blindly: it is what holds an emoji sequence together, and deleting it turns one family emoji into four people.
- Collapse and trim whitespace. Runs of spaces become one, and each line loses its leading and trailing spaces. Do this after removing the invisible variants, because a non-breaking space will otherwise survive the collapse.
- Only then change case or build a slug. Case conversion and slugging are transformations of clean text; running them first makes the invisible characters part of the identifier you then have to fix.
The order matters because the operations are not commutative. Trimming before removing a zero-width space leaves the zero-width space; slugging before cleaning can turn an invisible character into a hyphen or drop it silently, producing two different slugs for two visually identical titles.
Case conversion is not one-to-one
Case mapping changes length and can change meaning. German ß uppercases to SS — two characters for
one — and the Turkish dotted İ lowercases to i plus a combining dot, not to a plain i. A slug built
from uppercased text can therefore differ in length from the input, and a case-insensitive comparison
that relies on toLowerCase() can disagree with a database collation. If a value must be
case-folded consistently, do it in one place and use the same locale everywhere.
Curly quotes are a separate substitution: “ ” ‘ ’ are different code points from " and ', so a
search for "exact" does not match “exact”. A cleaner can map them, but that
is a deliberate content change, not whitespace cleanup; do it when the destination is code or a
database key, not when the text is prose you want to preserve.
When not to clean
- Code and configuration. Whitespace is significant in Python, YAML and Markdown code fences; collapsing it changes the meaning.
- Right-to-left text. Directional marks (U+200E, U+200F, U+202A–U+202E) control display order; removing them can reorder a sentence.
- Emoji sequences. Keep U+200D; removing it breaks the sequence into its parts.
- Poetry, addresses and signatures. The line breaks are the content; use the line tools to sort or deduplicate, not the joiner.
The general rule is to remove a character only when you can say why it is there and why it should not be. “It is invisible” is not a reason — the zero-width joiner is invisible and load-bearing.
Diagnosing a mismatch in one minute
Paste both values into the Unicode inspector and compare the code point lists. If one contains U+00A0 where the other has U+0020, that is the difference. If both lists look identical but the values still compare unequal, check normalisation: é as U+00E9 and e + U+0301 are different data that render the same. If the lists differ only at the end, look for a trailing zero-width space or a carriage return. Almost every “identical strings” bug is one of those three.