Recipe: UTF-8 files with strict decoding and line endings

Intended use and acceptance criteria

Save a Greek and emoji note in a disposable directory, then prove that the bytes written equal its UTF-8 encoding and that strict decoding recovers the original string. Detect deliberately invalid UTF-8 rather than replacing unknown bytes with question marks. File names and file contents have independent Unicode handling.

Complete runnable example

Save this as 06_UTF8.py and execute with Python 3. The fixture is disposable and has no network dependency.

import tempfile
from pathlib import Path
with tempfile.TemporaryDirectory() as folder:
    item = Path(folder) / "σημείωση.txt"
    source = "Καλημέρα 🌍\nγραμμή 2"
    item.write_text(source, encoding="utf-8", newline="\n")
    raw = item.read_bytes()
    assert raw == source.encode("utf-8")
    assert item.read_text(encoding="utf-8") == source
    try:
        raw[:1] + b"\xff"
        (Path(folder)/"bad.txt").write_bytes(b"hello\xff")
        (Path(folder)/"bad.txt").read_text(encoding="utf-8")
    except UnicodeDecodeError:
        pass
    else:
        raise AssertionError("invalid UTF-8 silently decoded")
print("PASS strict UTF-8 roundtrip")

Guarantees and limitations

Path.write_text with encoding='utf-8' controls the content encoding explicitly. newline='\n' in supported Python versions requests consistent line endings; the bytes still depend on the exact text passed. A UnicodeDecodeError indicates invalid bytes for strict UTF-8, not simply a display-font problem. Passing errors='replace' may be useful for viewing corrupted data but destroys information and is unacceptable for verifying a round trip.

Adaptation lab

Insert a combining accent and compare NFC versus NFD byte sequences, without rewriting the original note. Add CRLF text and document whether newline conversion is desired for your file format. Catch malformed files separately from missing paths and permissions. Test the emoji on Android/Termux and confirm console font rendering is not mistaken for encoding validity.

Troubleshooting and release check

If a UTF-8 file looks broken in an editor, inspect the editor's encoding setting and raw hexadecimal bytes before changing source data. Avoid decoding an arbitrary truncated byte prefix. Use the exact error offset from UnicodeDecodeError to locate corrupt bytes instead of replacing the entire file.