‹ BackHN Continuity

Thread

UTF-8000: Unlimited UTF-8

134 points · 122 comments · vismit2000

  1. 2shortplanks · · focus · HN ↗
    On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.

    So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.

    1. flohofwoe · · focus · HN ↗
      OTH UTF-8 is just one variable-length stream encoding among many others (RLE, LBE128, etc...).
      1. saghm · · focus · HN ↗
        Unless I'm misremembering, even UTF-16 is variable. You need to bump up to UTF-32 to get fixed-width.
        1. cyphar · · focus · HN ↗
          Even better, it's arguably both -- surrogate characters are valid codepoint values so technically UTF-16 is fixed-width but programs need to have special handling for surrogate pairs meaning it is practically variable-width.

          Truly the worst of all worlds.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.