It's more common for programs that say they support UTF-8 to not really do so at all. It wasn't that long ago that "UTF-8" support was often just single byte, so it was little more than ASCII. Even now it's common for programs to choke on the optional BOM. Yes, it is redundant, congratulations. The spec still explicitly allows it. Three and four byte character support is still not the best, too.
> "UTF-8" support was often just single byte, so it was little more than ASCII
"Single byte UTF-8" is ASCII. That's one of its most important properties.
> Even now it's common for programs to choke on the optional BOM
And they should... BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing and should be abolished along with "extended" 8-bit ASCII encodings and UCS-2/UTF-16 (only UTF-32 makes sense, but should only be used at runtime to allow random access on UNICODE code points, but not for data exchange.
CJK characters is the one big weak point of UTF-8. It forces them to be 3 bytes large instead of the local codepage which is 2 bytes per character, or UTF-16 which is also 2 bytes per character.
IMHO a 30 percent size increase for sharing text data is really not a problem when most data sharing (by volume) happens via audio, video and image data which is infinitely larger. We obviously have both the bandwidth and storage volume to spare for those 30% increased size for text data which among all the other data would account for a tiny fraction of a percent.
Dwedit · · focus · HN ↗
sph · · focus · HN ↗
da_chicken · · focus · HN ↗
It's more common for programs that say they support UTF-8 to not really do so at all. It wasn't that long ago that "UTF-8" support was often just single byte, so it was little more than ASCII. Even now it's common for programs to choke on the optional BOM. Yes, it is redundant, congratulations. The spec still explicitly allows it. Three and four byte character support is still not the best, too.
flohofwoe · · focus · HN ↗
"Single byte UTF-8" is ASCII. That's one of its most important properties.
> Even now it's common for programs to choke on the optional BOM
And they should... BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing and should be abolished along with "extended" 8-bit ASCII encodings and UCS-2/UTF-16 (only UTF-32 makes sense, but should only be used at runtime to allow random access on UNICODE code points, but not for data exchange.
Dwedit · · focus · HN ↗
flohofwoe · · focus · HN ↗