Python

UnicodeDecodeError: invalid start byte

Written and reviewed by Sahil Srivastav

PythonText encodingInput validation
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 14: invalid start byte

What this error actually means

A file or socket yielded bytes, but code decoded them as UTF-8 and encountered a byte sequence that UTF-8 does not permit at that position. The byte is evidence about the producer, not a character Python can guess reliably.

The common trap is assuming the machine locale or filename tells you the encoding. Windows-1252, ISO-8859-1, UTF-16, compressed data, and a truncated multibyte sequence all produce different remedies. `errors="ignore"` removes evidence and can merge or alter identifiers.

Decode once at the boundary, preserve the original bytes when the source is untrusted, and make the chosen encoding part of the protocol.

Causes, most common first

  1. 1Producer emits a legacy encoding. Windows applications commonly emit cp1252 or UTF-16 rather than UTF-8.
  2. 2Compressed or binary data is treated as text. The first bytes are a format header, not encoded characters.
  3. 3Truncated multibyte sequence. A file or stream ended between UTF-8 continuation bytes.
  4. 4Incorrect HTTP charset declaration. The declared encoding does not match the response bytes.

When you see it

  • Only files from one exporter fail
  • The position changes with line endings or chunk size
  • Replacing errors makes names or keys silently change
  • A UTF-16 file reports an invalid UTF-8 byte near the start
  • The error appears after reading a response declared with the wrong charset

How to diagnose it

Step 1

Inspect bytes, not a rendered editor view

A hex dump shows BOMs and binary signatures that text editors may hide.

xxd -l 64 input.dat
file -bi input.dat

Step 2

Check the protocol metadata

Read HTTP Content-Type, file manifests, or producer settings before trying detectors.

curl -sI https://api.example.test/data | rg -i 'content-type|charset'

Step 3

Decode a sample explicitly

Try a documented candidate and inspect failures; do not accept a detector’s guess without validating known characters.

python - <<'PY'
b = open('input.dat','rb').read(256)
for enc in ('utf-8','cp1252','utf-16'):
    try: print(enc, b.decode(enc))
    except UnicodeDecodeError as e: print(enc, e)
PY

The fix

Fix the producer or pass the correct encoding explicitly to `open`, `TextIOWrapper`, or the HTTP client.

Use `utf-8-sig` when a UTF-8 BOM is an accepted export convention, and use `utf-16` only when metadata or BOM confirms it.

Reject or quarantine undecodable records with offsets and source identifiers; retain bytes for reprocessing.

Use `errors="replace"` only for display-only text where loss is acceptable and visible.

Separate binary parsing from text decoding; inspect magic bytes before calling `.decode()`.

with open(path, 'r', encoding='cp1252', newline='') as f:
    for line in f:
        consume(line)

# preserve bad input for later correction
text = raw.decode('utf-8', errors='strict')

How to stop it coming back

  • Specify encoding in every file and API contract
  • Add fixtures containing accents, emoji, BOMs, and malformed bytes
  • Log source and byte offset on rejection
  • Never use errors=ignore for identifiers or financial data
  • Keep raw payloads when the producer is outside your control

Practise production debugging in a real repository

Reading about a failure and reproducing one are different skills. Gronex ships broken backend repositories with failing test suites that encode the real invariant, so you debug from evidence instead of memorising symptoms.

FAQ

Is Latin-1 a safe fallback?

It maps every byte, so it never raises, but it can produce incorrect text. Use it only when the producer contract says ISO-8859-1.

Why does the same file work on my laptop?

Implicit locale defaults differ. Explicit encoding removes that machine-dependent behaviour.

Can UTF-8 contain byte 0x96?

Not as a standalone byte. In cp1252 it represents an en dash, which is a strong clue about the producer.

Related

Python requests ReadTimeout

Other errors engineers hit next to this one

Full error and symptom index →