Unicode Inspector (code points, categories, UTF-8)

What this tool does

Two strings can look identical on screen and still be different data. The classic causes are a precomposed letter versus a letter plus a combining mark, and invisible characters such as a zero-width space or a non-breaking space pasted in from a web page or a spreadsheet. Both break exact matching, unique constraints and search.

This inspector lists every code point in your text in order, with its hexadecimal value, the Unicode general category, and the UTF-8 byte sequence. Invisible and combining characters then become obvious rather than mysterious.

Method

The text is iterated by code point rather than by UTF-16 code unit, so characters outside the Basic Multilingual Plane — emoji and rare scripts — are counted as one item instead of two. Each character is encoded to UTF-8 to produce the byte column, and its category is taken from the Unicode general-category property (L letters, N numbers, M combining marks, Z separators, C control and format, P punctuation, S symbols) evaluated by the browser's regular-expression engine.

Example: why “identical” text fails to match

Text	Code points	Diagnosis
é (precomposed)	U+00E9	One character
é (decomposed)	U+0065 U+0301	e plus combining acute — not equal to the above
“a b” with a zero-width space	U+0061 U+200B U+0062	Invisible character breaks the word
๑	U+0E51 (3 bytes)	Thai digit, not the ASCII “1”

The four numbers behind “one character”

“How many characters is this?” has four defensible answers, and a database column, a password field and a word processor each pick a different one. This inspector shows the code points, which is the level the other three are built on:

Measure	What it counts	“👨‍👩‍👧‍👦”	Where it matters
Grapheme clusters	User-perceived characters	1	Word-processor counts, cursor movement
Code points	Unicode characters	7	What this page lists
UTF-16 code units	JavaScript string length	11	.length, .slice()
UTF-8 bytes	Storage and transport	25	Column limits, headers, bandwidth

The family emoji is one person on screen, seven code points (four people and three zero-width joiners), eleven UTF-16 code units because each of the four people is a surrogate pair, and 25 bytes in UTF-8. A validation rule that allows “20 characters” will accept a string of five family emoji as five graphemes, reject it as 35 code points, or store it as 125 bytes, depending on what the rule actually counts.

Invisible characters are not all the same

There are several distinct classes of invisible character, and the fix differs for each. This page labels them so you can tell which one you are looking at:

• Space separators (Zs) — the non-breaking space U+00A0, the narrow no-break space U+202F, the ideographic space U+3000. They take up width but do not break a line, and a regular space does not equal any of them.
• Format characters (Cf) — zero-width space U+200B, zero-width non-joiner U+200C, zero-width joiner U+200D (which is meaningful inside emoji), the byte-order mark U+FEFF, and the directional marks used in right-to-left text. These occupy no width at all.
• Control characters (Cc) — tab, carriage return and the rest of the C0 range. They are visible here as their code point rather than as an effect on the layout.
• Combining marks (M) — visible only in that they attach to the preceding character. A combining acute after e is not the same data as é, even though it renders the same.

Two consequences follow. First, a zero-width joiner is sometimes doing real work — removing it breaks an emoji sequence into separate people — so “delete every invisible character” is the wrong repair for a string that legitimately contains one. Second, the byte-order mark at the start of a file is invisible but shifts every offset by three bytes, which is why a CSV parser can report a malformed first column header that looks perfectly normal when you open the file in a text editor.

Reading the category column

The category is the Unicode general category, evaluated by the browser's own regular-expression engine rather than from a table shipped with the page, so it reflects the Unicode version the browser implements. A letter that is also uppercase reports Uppercase letter, a digit reports Decimal digit, and anything unassigned in that Unicode version falls through to Other. The category is why two characters that look alike can be treated differently by a validator: the ASCII 0 is a decimal digit, the circled ⓪ is a symbol, and a validation rule written as \d accepts the first and not the second.

Frequently asked questions

Go deeper

Longer explanations of the problems this tool solves:

Cleaning text that looks right and behaves wrongUnicode normalisation: why é can be one character or twoThe invisible characters that break lookupsPunycode and homograph attacks: what xn-- really meansWhy your character count is wrong: bytes vs characters

Sources

• Unicode Standard, code points, code point ranges and general categories.
• RFC 3629, UTF-8, a transformation format of ISO 10646.