Reading a binary file in text mode

Question

Accepted Answer

Text mode doesn't just read bytes - it decodes them. On most modern systems the default encoding is UTF-8. UTF-8 is a variable-width encoding with strict rules: byte sequences above 127 must follow specific patterns, and arbitrary binary data almost certainly violates them. When the decoder hits a byte sequence it can't interpret, it raises UnicodeDecodeError: 'utf-8' codec can't decode byte 0x... in position ...: invalid start byte. You won't always get the error immediately. If the binary content happens to contain only valid ASCII bytes (0-127), the read succeeds but the data is still wrong once you hit a non-ASCII byte. This makes the bug intermittent and hard to reproduce in testing. Three ways to handle this: use 'rb' for binary files; pass errors='replace' to substitute a replacement character for bad bytes; or pass errors='ignore' to silently drop them. For data science pipelines reading CSVs with unknown encodings, errors='replace' plus a log message is usually the right call.