Self-synchronization in UTF-8 is intuitively a great thing to have, yet I don’t remember actively relying on it ever. Does anyone have a good example of when it‘s useful?
Another related nice property that UTF-8 has: substring search reduces to bytestring substring search. I.e. given two Unicode strings in UTF-8 encoding, you can check if one is a substring of the other by just treating them as bytestrings and checking if one bytestring is a substring of the other bytestring. This is a stronger property than self-synchronization: UTF-8 has it, but UTF-8000 doesn’t.
> Does anyone have a good example of when it‘s useful?
It prevents vulnerabilities where an incorrect offset into a string could result in characters being read that aren’t in the original string (which could defeat a prior sanitization of the string).
lukasgelbmann · · focus · HN ↗
Another related nice property that UTF-8 has: substring search reduces to bytestring substring search. I.e. given two Unicode strings in UTF-8 encoding, you can check if one is a substring of the other by just treating them as bytestrings and checking if one bytestring is a substring of the other bytestring. This is a stronger property than self-synchronization: UTF-8 has it, but UTF-8000 doesn’t.
layer8 · · focus · HN ↗
It prevents vulnerabilities where an incorrect offset into a string could result in characters being read that aren’t in the original string (which could defeat a prior sanitization of the string).