TryBugHunt

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte in Python means python tried to read a file as UTF-8 text, but the file was saved in a different encoding or is not a text file at all.

PythonPattern: Type errors

How do I fix UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte?

UnicodeDecodeError: 'utf-8' codec can't decode byte means the file is not UTF-8 text. Often it was saved by Excel on Windows as cp1252, or as UTF-16 when byte 0xff appears at position 0. Pass the real encoding, like open(path, encoding="cp1252"). For images or PDFs, open with "rb" instead.

  1. 1. Check the file typeImages, PDFs and .xlsx files are not text. Open them with "rb" or a library made for that format.
  2. 2. Read the byte and position0xff or 0xfe at position 0 suggests UTF-16. A byte later in the file suggests cp1252.
  3. 3. Re-save as UTF-8Save the file as CSV UTF-8 from Excel or your editor, then open it with encoding="utf-8".

What does UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte mean?

Files store bytes. To turn bytes into text, Python needs to know the encoding (the rule that maps bytes to letters). UTF-8 is the common default, but files from Excel on Windows are often saved as cp1252, and some are UTF-16. When a byte does not fit the UTF-8 rules, you get this error. A byte of 0xff or 0xfe at position 0 usually means a UTF-16 file, or a binary file such as a JPEG image. On Windows the message may say 'charmap' codec instead, which is the same problem the other way round.

Why does UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte happen?

1. A file saved in another encoding

A CSV saved by Excel on Windows is often cp1252. Tell open() the real encoding.

Breaks

with open("marks.csv", encoding="utf-8") as f:
    text = f.read()

Works

with open("marks.csv", encoding="cp1252") as f:
    text = f.read()

2. A UTF-16 file

Files that start with the bytes 0xff 0xfe are UTF-16. Reading them as UTF-8 fails on the very first byte.

Breaks

open("export.txt", encoding="utf-8").read()

Works

open("export.txt", encoding="utf-16").read()

3. Opening a non-text file in text mode

Images, PDFs and .xlsx files are binary. They are not text in any encoding.

Breaks

data = open("photo.jpg").read()

Works

data = open("photo.jpg", "rb").read()   # bytes

How to find it in your own code

Checklist: 1) Check what kind of file it is; if it is not plain text, open it with "rb". 2) Re-save it from your editor or Excel as "CSV UTF-8", which is the cleanest fix. 3) Otherwise try encoding="utf-16" (0xff/0xfe at position 0) or encoding="cp1252". 4) As a last resort, errors="replace" reads the file and shows bad bytes as �, so you can see where the problem is. 5) Always pass encoding= when you open text files so your code works the same on every computer.

Practise type errors

None of the challenges raises this exact error yet. These practise the same kind of bug.

Still not sure why yours breaks?

Paste it into the visualizer and watch it run line by line, with every variable at every step. Free, and it runs in your browser.

Related error guides

View all type errors →