On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.
So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.
Without endorsing this specific idea, I think maybe after over half a century of C that this argument shouldn't get in the way of a new standard. Pretty much every other language has managed to solve this problem, and the people who write new projects in C/C++ have decided they're not concerned about buffer overflows, so if someone decides to start a new project using something like this (or go out of their way to add support for it to an existing project), that's kind of on them. The rest of computing shouldn't get stuck in 1972 forever.
I'm failing to see why an embedded environment would have any need for a new encoding format, which is kind of my point: the types of things that are going to be written in C are not the ones going to be adopting completely new backwards-incompatible standards anyhow. If we refuse to try something based on how it would interact of the ecosystem that would likely never consider adopting something like it in the first place, we're literally fixing our computing to constraints from half a century ago and counting. Nobody who is going to write C would stop because of something like a new Unicode scheme with much larger encoding widths, so why should that be an argument against it happening?
A reasonable person would assume you’re talking about UTF-8000. It’s not completely new: RFC2279, the original UTF-8 proposal, worked exactly like UTF-8000 for codepoints 31 bits or smaller in size. It’s not backwards-incompatible: UTF-8000 is exactly like UTF-8 for 1, 2, and 3-byte long codepoints, and like UTF-8 codepoints for 4-byte long codepoints with a value of 0x10_ffff or smaller (so all UTF-8 codepoints encoded with the first byte being 0b1111_00xx or starting with the bytes 0b1111_0100 0b1000_xxxx).
It’s a backwards compatible way of encoding numbers in UTF-8 larger than 0x10_ffff or (0x7fff_ffff with the original RFC2279 proposal).
I agree that C isn’t the best language to start a new programming project in. There are things I don’t like about Rust, mainly that there’s only one implementation of it, but if I were to start a new project needing the speed of a system programming language, it makes a lot of sense.
The original comment that I responded to said this:
> On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.
My response was that I don't see "it would be hard to write correct code for this in C" to be a particularly useful objection. I'm honestly a bit confused by the reaction to my comments because it seems that every new reply seems to have an orthogonal objection to the previous one, but none of them seem to be actually addressing the point I've been trying to make.
2shortplanks · · focus · HN ↗
So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.
saghm · · focus · HN ↗
Without endorsing this specific idea, I think maybe after over half a century of C that this argument shouldn't get in the way of a new standard. Pretty much every other language has managed to solve this problem, and the people who write new projects in C/C++ have decided they're not concerned about buffer overflows, so if someone decides to start a new project using something like this (or go out of their way to add support for it to an existing project), that's kind of on them. The rest of computing shouldn't get stuck in 1972 forever.
drfloyd51 · · focus · HN ↗
Sometimes the environment forces a language choice.
saghm · · focus · HN ↗
strenholme · · focus · HN ↗
A reasonable person would assume you’re talking about UTF-8000. It’s not completely new: RFC2279, the original UTF-8 proposal, worked exactly like UTF-8000 for codepoints 31 bits or smaller in size. It’s not backwards-incompatible: UTF-8000 is exactly like UTF-8 for 1, 2, and 3-byte long codepoints, and like UTF-8 codepoints for 4-byte long codepoints with a value of 0x10_ffff or smaller (so all UTF-8 codepoints encoded with the first byte being 0b1111_00xx or starting with the bytes 0b1111_0100 0b1000_xxxx).
It’s a backwards compatible way of encoding numbers in UTF-8 larger than 0x10_ffff or (0x7fff_ffff with the original RFC2279 proposal).
I agree that C isn’t the best language to start a new programming project in. There are things I don’t like about Rust, mainly that there’s only one implementation of it, but if I were to start a new project needing the speed of a system programming language, it makes a lot of sense.
saghm · · focus · HN ↗
> On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.
My response was that I don't see "it would be hard to write correct code for this in C" to be a particularly useful objection. I'm honestly a bit confused by the reaction to my comments because it seems that every new reply seems to have an orthogonal objection to the previous one, but none of them seem to be actually addressing the point I've been trying to make.