Character Encoding and Garbled Text
Character encoding is the agreement that maps stored bytes to text. Unicode defines the characters; encodings such as UTF-8 and UTF-16 define how their code points become bytes or code units.
Common Encoding Symptoms
| Symptom | Likely cause | First check |
|---|---|---|
é instead of é | UTF-8 decoded as a Western single-byte encoding | Response header and database connection charset |
| � replacement character | Invalid, missing, or undecodable bytes | Original byte source before conversion |
| Empty square or tofu glyph | Font lacks a glyph | Try a font with broader Unicode coverage |
| Question marks saved permanently | Destination encoding could not represent characters | Database column and connection encoding |
Keep the Entire Pipeline Consistent
The editor, source file, web server, HTML declaration, application runtime, database connection, database columns, APIs, and export files should agree on Unicode. One mismatched step can corrupt text even when every other component uses UTF-8.
Do not repeatedly encode already encoded text or decode bytes without knowing their source encoding. Preserve the original data while diagnosing a problem.