Why your character count is wrong: bytes vs characters

Four numbers, one string

Take the family emoji 👨‍👩‍👧‍👦. Counted four different ways, it is four different numbers — and every one of them is correct in its own context:

The gap is not a bug in any of these systems. It is what happens when one glyph is built from several code points joined by invisible characters, and each layer counts at a different level.

Code points: the base unit

Unicode assigns every character a number — a code point — written U+1F468 for 👨, U+200D for the zero-width joiner. A string is a sequence of code points. Counting them is the most portable definition, and it is what this site's counter shows as “Characters”.

The family emoji is not one code point. It is five emoji code points — man, woman, girl, boy, and a fourth person — joined by four zero-width joiners:

👨  U+1F468
    U+200D  zero-width joiner
👩  U+1F469
    U+200D
👧  U+1F467
    U+200D
👦  U+1F466

Seven code points. The joiner tells the renderer “combine these into one glyph”, which is why you see one picture and the counter says seven. A skin-tone modifier works the same way: 👍🏽 is the thumbs-up plus a separate modifier code point, so it counts as two.

Bytes: what storage actually enforces

UTF-8 encodes each code point in one to four bytes. ASCII is one byte; accented Latin, Greek, Cyrillic and Hebrew are mostly two; CJK and emoji are three or four. So 日本 is two code points but six bytes, and the family emoji is seven code points but 25 bytes.

This is the number that matters when something rejects your input. A VARCHAR(50) column, an HTTP header limit, an S3 key length, a Redis value cap — these are almost always byte limits, not character limits. A 40-character name in Thai can be 120 bytes and fail a limit you thought you were well inside.

text          code points   UTF-8 bytes
Hello         5             5
café          4             5
日本          2             6
👨‍👩‍👧‍👦          7             25

UTF-16 units: why JavaScript says 11

JavaScript strings are sequences of UTF-16 code units, and '👨‍👩‍👧‍👦'.length returns 11, not 1 and not 7. Code points above U+FFFF — which includes most emoji — are stored as a surrogate pair, two units each. Five emoji code points become ten units, plus the four joiners, makes 11.

This is why naive validation fails. value.length <= 10 rejects a three-emoji string that looks like three characters. The correct fix is to count code points with [...value].length (the spread iterates by code point) or, if you want what the user sees, graphemes.

Graphemes: what the user sees

A grapheme cluster is the unit a human perceives as one character. It groups a base character with everything that modifies it: combining accents, skin-tone modifiers, joiner sequences, and regional-indicator pairs (flags). The standard way to count them in a browser is Intl.Segmenter:

new Intl.Segmenter('en', { granularity: 'grapheme' })
const count = [...segmenter.segment(text)].length;

This is the number to show when the label says “characters” and the user is looking at what is on screen — a word processor's count, a design tool's count. It is not what to validate against, because it is the least stable across fonts and platforms: a grapheme the user sees as one may render as two fallback glyphs on a device without the font.

Which one should you use?

The truncation bug this causes

Slicing a UTF-8 string by a byte count can split a multi-byte sequence, producing invalid UTF-8 — which most decoders render as �. Slicing by JS .length can split a surrogate pair, which produces a lone surrogate and, in JSON, an invalid string. Both look like corruption to the user and neither is caught by a length check that passed before truncation.

If you must truncate, iterate by grapheme or code point and stop before the limit, then trim any trailing whitespace. A short string that is valid is always better than a long one that is not.

Truncation, in detail

Truncation is where these four counts collide. Each unit implies a different slicing operation, and slicing at the wrong one produces invalid text rather than a shorter string.

If a hard limit is in bytes — a database column, an HTTP header — the correct procedure is to walk graphemes, accumulate their byte length, and stop before the limit is exceeded. A naive slice(0, 50) on a UTF-8 buffer can produce something no decoder will accept.

// safe truncation to a byte budget, walking graphemes
let out = "", used = 0;
for (const g of segmenter.segment(text)) {
  const n = new TextEncoder().encode(g.segment).length;
  if (used + n > budget) break;
  out += g.segment;
  used += n;
}

Emoji in particular

Emoji are the stress test for every one of these rules, because several mechanisms compose at once:

• Skin-tone modifiers — a second code point, so 👍🏽 is 2 code points and 8 bytes.
• Zero-width joiners — sequence several emoji into one glyph, as in the family example.
• Regional indicator pairs — a flag is two code points that render as one symbol; 🇬🇧 is 2 code points, 8 bytes, 1 grapheme.
• Keycap sequences — digit plus variation selector plus enclosing keycap, several code points for one visible character.

Practically: never assume an emoji is one character, never split a string on the assumption that adjacent code points are independent, and never use a byte limit as a proxy for “how long it looks”.

The database column problem

MySQL's older utf8 charset is really UTF-8 with a three-byte maximum, so it cannot store emoji — a four-byte character is rejected or truncated (and truncation at a byte boundary is exactly the invalid-sequence case above). utf8mb4 is the four-byte-capable charset and the one to use. This is worth checking before blaming the application: a column that silently rejects emoji is a configuration detail, not a bug in the text.

Even with utf8mb4, a VARCHAR(n) limit is measured in characters as the database defines them, not bytes — so the same limit can accept a different number of bytes depending on the content. If the application also enforces a byte cap, the two limits can disagree, and the smaller one wins at the least convenient moment.

Counting for a user interface

When a field shows “12 / 50”, decide explicitly which unit the 50 is and label it. If the server enforces code points and the UI counts graphemes, a user can fill the counter and still be rejected — or be allowed to submit something the server then truncates. The honest approach is to count in the unit that is actually enforced, and if that unit is invisible to the user, explain it in the helper text rather than letting them discover it on submit.

Check it yourself

Paste any string into the character counter to see code points, graphemes and UTF-8 bytes side by side. The UTF-8 inspector shows the bytes of each character individually, and the code point inspector names every character and flags the invisible ones.