Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".
We really ought to deprecate UTF-16 someday. The fact that it pretends to be a fixed-length encoding has caused all sorts of bugs over the years, with many people assuming n(UTF-16 codepoints) == n(characters) which breaks when the string contains non-BMP characters.
And also, for personal aesthetic reasons I hate that it limits the Unicode codepoint range to an awkward non-power-of-two number (now there are 0x110000 codepoints in total). UTF-8 and UTF-32's 2^31 feels much more natural.
I wonder what it would take. For things like Java, JSA, and c# there would have to be things like a parallel utf8 api (yes please), but the really hard parts is the things that are as old as time (windows).
I don't think it will ever happen, but one can dream.
I think that wouldn't change much. They would just make use of more grapheme clusters.
For emojies they already make heavy use of the Zero-Width-Joiner. So a woman firefighter is the woman emoji + ZWJ + fire engine. Sure the UTF-8000 approach is much better encoding size wise.
It's a joke based on the construction of emojis? Female plus fire engine obviously denotes a female fire engine but has apparently been repurposed as a female firefighter instead
I predict eventual convergence between UTF-whatever and most popular tokenizer for whatever LLM escapes to become world-ruling AGI.
I mean, if someone's seriously going to try encoding birdsong and dog barks, at this point they're basically reinventing tokens for multi-modal language models.
sph · · focus · HN ↗
Not until they decide to expand the emoji range, allocate space for all past and future fictional languages, as well as birdsong and dog barks.
Someone at the consortium is rubbing their hands with glee with all the newfound space.
But honestly, cool hack! If you invent a method to encode large numbers into bytes, why limit yourself to 24-bit numbers?
flohofwoe · · focus · HN ↗
Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".
throw0101a · · focus · HN ↗
For compatibility with UTF-16:
* <a href="https://datatracker.ietf.org/doc/html/rfc3629#section-12" rel="nofollow">https://datatracker.ietf.org/doc/html/rfc3629#section-12* <a href="https://en.wikipedia.org/wiki/UTF-16" rel="nofollow">https://en.wikipedia.org/wiki/UTF-16
The original spec had 31 bits (the UTF-32/UCS-4 range):
* <a href="https://datatracker.ietf.org/doc/html/rfc2279" rel="nofollow">https://datatracker.ietf.org/doc/html/rfc2279
* <a href="https://en.wikipedia.org/wiki/UTF-32" rel="nofollow">https://en.wikipedia.org/wiki/UTF-32
orangeboats · · focus · HN ↗
And also, for personal aesthetic reasons I hate that it limits the Unicode codepoint range to an awkward non-power-of-two number (now there are 0x110000 codepoints in total). UTF-8 and UTF-32's 2^31 feels much more natural.
bjoli · · focus · HN ↗
I don't think it will ever happen, but one can dream.
sharktheone · · focus · HN ↗
For emojies they already make heavy use of the Zero-Width-Joiner. So a woman firefighter is the woman emoji + ZWJ + fire engine. Sure the UTF-8000 approach is much better encoding size wise.
mitxela · · focus · HN ↗
Dylan16807 · · focus · HN ↗
sharktheone · · focus · HN ↗
mitxela · · focus · HN ↗
TeMPOraL · · focus · HN ↗
I mean, if someone's seriously going to try encoding birdsong and dog barks, at this point they're basically reinventing tokens for multi-modal language models.