← Lab
ASTERXOR / Lab

When one character is more than one thing

A visible symbol, a Unicode code point and a stored byte are different units. Counting text requires choosing which one you mean.

Separate gold marks and a smaller blue component form a connected abstract symbol.
A conceptual illustration of several stored components contributing to one displayed unit.
LabASTERXOR / LEON LEUNG

The same-looking accented letter can have two different lengths.

The letter é can be represented as one precomposed code point, or as an ordinary e followed by a combining acute accent. These sequences can display alike, yet Python counts one as length one and the other as length two. Neither answer is a failure of counting. The program is measuring a unit that does not always match a reader’s idea of one character.

Before counting text, name the unit.
THREE RULERS

Bytes, code points and visible units

A byte is a storage unit of eight bits. A Unicode code point is a numbered position in the character repertoire. An extended grapheme cluster is a standard text-segmentation approximation to a user-perceived character. Rendering then adds fonts, shaping and presentation. These layers cooperate, but their counts need not agree.

Python strings are sequences of Unicode code points, so len measures those positions. Encoding a string as UTF-8 produces bytes that can be counted separately. The experiment uses both operations on six fixed examples. It does not run a grapheme-segmentation engine or test whether every font draws each example as one visible symbol.

READ THE NUMBERS

One is not always one byte

The plain letter A uses one code point and one UTF-8 byte. Precomposed é uses one code point but two bytes. The decomposed e plus accent uses two code points and three bytes. The Korean syllable 한 uses one code point and three bytes in this fixture. The storage cost cannot be inferred from Python’s string length alone.

The thumbs-up with skin tone, 👍🏽, consists of two code points and eight UTF-8 bytes. The woman-technologist sequence, 👩‍💻, contains three code points and eleven bytes, including a zero-width joiner. A supporting renderer may combine those components visually. The extra component is meaningful even though it does not appear as a separate printed character.

Paired bars compare code-point and byte counts for letters and emoji sequences.
FIG 01Actual len and UTF-8 byte counts for the six literal fixtures. Labels describe examples without depending on emoji fonts.
CANONICAL FORMS

Two spellings can meet in the same form

Before normalization, the two versions of é are unequal as Python strings because their code-point sequences differ. Applying NFC normalization makes them equal in this example, producing the composed form. The script verifies that result rather than assuming visual resemblance means byte-for-byte identity.

Normalization is not a general instruction to erase distinctions. NFC handles canonical equivalence. Compatibility normalization can make additional changes with different consequences, and neither is a universal detector of lookalike names. A text system should choose a documented policy suited to its purpose, while preserving original input when that original form matters.

One and two code points before NFC both become one afterward.
FIG 02NFC maps these two canonically equivalent fixtures to the same composed representation.
THE CURSOR

A cut can split something the reader sees as whole

A byte limit, code-point limit and displayed-character limit can stop at different places. Cutting encoded UTF-8 in the middle of a multibyte sequence can leave invalid bytes. Cutting a string between an e and its combining accent, or inside an emoji sequence, may preserve valid code points while disrupting the reader’s intended symbol.

Applications therefore need an explicit rule for truncation, cursor movement and character limits. Unicode’s grapheme-boundary rules provide a common basis, with language and product context sometimes requiring further treatment. This article’s six fixtures illustrate why that layer exists; they are not a complete segmentation test suite or a promise about every editor’s behaviour.

AT THE BOUNDARY

A length limit is an interface promise

Suppose a form advertises a limit of twenty characters while a storage field allows only twenty bytes. Plain ASCII input may conceal the disagreement, but accented letters and emoji expose it quickly. A user can obey the visible rule and still encounter a storage rejection. The problem is a mismatched contract, not unusually difficult text.

A clear system specifies the unit, encodes and decodes deliberately, and tests representative scripts and combinations rather than only English letters. Equal-looking text also deserves careful handling in search and comparison. The helpful first step is modest: replace the vague question “How long is this string?” with a question about bytes, code points or the editing units a person expects.

THE THOUGHT TO TAKE AWAY

Text becomes less mysterious when every length comes with its own ruler.