UTF-8 — bytes, code points and visible characters

Why lengths disagree

UTF-8 encodes Unicode scalar values with one to four bytes. Python len(str) counts code points, not bytes or user-perceived grapheme clusters. A displayed accented character may consist of one precomposed code point or multiple combining code points, so careless byte slicing can corrupt text.

Executable demonstration (Python 3)

import unicodedata

text = "Aé🚀"
encoded = text.encode("utf-8")
assert len(text) == 3
assert len(encoded) == 7
assert encoded.decode("utf-8") == text

combined = "e\u0301"
precomposed = "é"
assert len(combined) == 2
assert len(precomposed) == 1
assert unicodedata.normalize("NFC", combined) == precomposed

A uses one byte, é uses two and the rocket emoji uses four: 1+2+4=7. NFC normalization can make canonically equivalent sequences comparable; it is not a universal fix for visually confusable characters or locale-aware sorting.

Real-world failure exercises

1. Take only the first two bytes of the rocket emoji and decode as strict UTF-8; observe UnicodeDecodeError.
2. Decode malformed bytes with errors="replace" and explain why this loses evidence compared with strict decoding.
3. Search Greek text with correct Unicode case handling; compare lower() with casefold() and normalization, and document language-specific caveats.
4. Store content as UTF-8 explicitly when opening files and verify round-trip bytes. Do not rely on a terminal's unspecified default encoding.
5. Test text containing a combining mark and a multi-code-point emoji sequence; explain why slicing by Python character count may still divide a visible symbol.

Completion and edge rules

Record Python version, exact input bytes and expected code points. A database column length, SMS limit and UI cursor movement may all apply different counting rules; name the unit instead of saying simply 'characters'.