Maybe I'm not that deep into programming, but I don't understand the hype about Zig?
I programmed in rust a bit and can't say I'm an expert, but in my view rust mostly-solved the memory management problem at compile time and without a GC, and it works very well. The biggest con and cost I've always seen repeated so far is that "it's slow to compile", and I get that, if you're past 250 crates the final --release link tends to become noticeable, but there were improvements to incremental compilation.
On the other hand - looking at the syntax from this post - Zig feels a blend of javascript, python and golang syntax that still requires memory management. So a nicer-written C that inherits all the issues from C? From the post: no functional programming, data mutation, memory leak, double-free, memory corruption.
Personally I'd rather trade a couple minutes of final link every time when this is the other option.
As a long-time low-level programmer, and as someone working on a popular mainstream language, I find Zig fascinating, and I also think it addresses a long-standing problem in low-level programming. I'll get to the problem later, but the fascinating part is its use of partial evaluation (comptime) as a single coherent mechanism that replaces a myriad of other partial-evaluation mechanisms (macros, templates/generics, constexprs). That one mechanism is the core of the language, like macros are in lisps, and that design - whether you like it or not - is revolutionary. It's never been done before (other languages have partial evaluation mechanisms that are almost as general, but they're offered in addition to, not as a replacement of, other features).
> rust mostly-solved the memory management problem at compile time and without a GC
"Mostly" does a lot of work here because 1., if you look at the implementation of very efficient, possibly specialised data structures - the very thing you reach for a low-level language for - they typically require unsafe, and 2., it still suffers from the problem C++ has had for decades, which is that over time, as program changes and evolves over years, things tend to drift toward the more general mechanisms that rely on malloc/free on an individual objects, and the program gets slower and slower (huge runtimes like TCMalloc help, but not enough, because they can't move pointers). This problem, of programs that start out fast, but after five or ten years of evolution need to spend a lot of effort to remain fast, is one of the things moving collectors were designed to solve, but they require moving pointers, which doesn't work in low-level languages that are not meant to have an FFI layer between them and the hardware.
To compete with the performance of moving GCs, which allocate through bumping a pointer, like on the stack, and free memory in bulk, low-level languages need to rely on arenas (which work based on a similar principle), and Zig is the first language that makes arenas almost user-friendly and hopefully sufficiently composable to withstand program evolution. Of course, time will tell how well this works in practice.
> the very thing you reach for a low-level language for - they typically require unsafe
There's a formal proof asserting that if you keep up the safety invariants within an unsafe region then that will not infect other code, even in the presence of arbitrary other correctly-written unsafe blocks.
This means you can build abstractions on top of these low-level primitives to keep it contained, so consumer code never has to even think about or know there's unsafe blocks in it. The type system lets you build very powerful abstractions so these go a long way.
There's a lot of woo-woo scare quoting around how much you actually have to use unsafe code in Rust. It's fairly uncommon to actually have to reach for them in practice. Most of my usage ends up being things like converting a &[u8] to a &str when I know it's already valid UTF-8 so I want to skip the linear-time validity check. Very rarely do I have to build data structures with complicated pointer juggling, because there's often a library that already does what I need!
> which is that over time, as program changes and evolves over years, things tend to drift toward the more general mechanisms that rely on malloc/free on an individual objects, and the program gets slower and slower
What are you talking about? I've never encountered this and I've been using Rust for 10 years.
> There's a formal proof asserting that if you keep up the safety invariants within an unsafe region then that will not infect other code, even in the presence of arbitrary other correctly-written unsafe blocks.
In general "unsafe" does not compose.
"if you keep up the safety invariants within an unsafe region"
Here's an article about the research on it which lays out the properties in simple terms: <a href="https://smallcultfollowing.com/babysteps/blog/2016/10/02/observational-equivalence-and-unsafe-code/" rel="nofollow">https://smallcultfollowing.com/babysteps/blog/2016/10/02/obs...
I'm curious why you think that statement is doing heavy lifting. It's much easier to write and verify that a few lines of code are correct than it is to write and verify that an entire program is correct. But that's the norm in C and Zig, and historically people haven't been very good at it. That's why we try to do it as little as possible.
Many more C programs have been verified than Rust programs. Also, Zig's spatial and memory safety is as good as Rust's, so it's not really similar to C at all.
The reason it's not "the norm" is that (especially with spatial safety taken care of), not every line is equally dangerous at all. Still, there's no doubt that more guarantees help, but that is only when all other things are equal. If you pick a low-level language for mostly low-level things, so Rust doesn't offer safety for the trickiest code, and furthermore it makes certain things harder to see because the language is more complicated, then things become much less clear. Obviously, when the vast majority of the trickiest, most important code doesn't need to be low-level, Rust would probably be safer on the whole, but in such situations I see no reason to choose either Rust or Zig. You need to choose a low-level language if the core of what you're doing needs to be low-level.
That's because Zig offers spatial memory safety (e.g. buffer overflows and index out of bounds), but no temporal memory safety (e.g. use-after-free). I suppose the "and" before "memory safety" is a typo.
You must reason about the invariants in unsafe code on a global level. In particular, you could have unsafe code in crate A, whose data are then used by crate B. It could be fine. But then crate B changes its implementation which now violates the invariant expectations of crate A.
This is true. In Java, we have a notion we call "integrity", which is a generalisation of memory safety and includes a host of properties guaranteed by the platform. It includes memory safety, but also things like "a non-public method cannot be called or a non-public field cannot be accessed (even reflectively) by code in another module".
To address the problem that once integrity can be violated anywhere, only global analysis can prove that nothing bad happens, we've done two things:
1. We require the application to explicitly permit any integrity violation by a module; i.e. a library can't allow itself to violate integrity. This is a principle we call "Integrity by Default" (<a href="https://openjdk.org/jeps/8305968" rel="nofollow">https://openjdk.org/jeps/8305968).
2. We try to minimise the need for potential integrity violations (this is very different from Rust, which requires unsafe even for things like benign write/write races, which are fairly common, and various basic data structures). Over the years we've offered safe replacements for things that used to require Unsafe. In other words, clearly demarcating unsafe code isn't enough if it's needed at all in many situations.
It isn't perfect, of course, as some libraries do require unsafe operations for direct interaction with native code or with memory, but their number has been greatly reduced, and they cannot do this without the application's explicit approval. Interestingly, this has annoyed library authors who want to do unsafe things but don't want to application authors to be alarmed because "we know what we're doing," and it's also annoyed some application authors who want to use such libraries and are forced to explicitly add permissions. But I think that the community, as a whole, has eventually accepted this because the harm done to those who don't care is small (they just need to add the permissions), to those who do care it helps a lot, and because fewer and fewer libraries require "integrity-busting" permissions, many applications need to do absolutely nothing and get important guarantees for free.
> this is very different from Rust, which requires unsafe even for things like benign write/write races, which are fairly common, and various basic data structures
I know this paper [0] is quite old at this point, but the mention of benign data races reminded me of it. Would you happen to know how applicable it is to modern memory models?
Benign write/write races (when multiple threads do unordered writes of the same value to the same address) are quite common and useful, both in parallel algorithms and in lazy initialisation. Useful benign read/write races are far more rare to the point I'd say it's ok to assume they don't (or shouldn't) exist.
However, in C and C++ (and Rust) benign non-atomic write/write races are UB (indeed, LLVM also treats them as potential causes of UB). In C# and in Java they are safe (although Java currently only has non-atomic writes on 32-bit machines, but soon they'll be more common when value types are enhanced). LLVM even has a specific construct to support the Java-style memory model (<a href="https://llvm.org/docs/Atomics.html#unordered" rel="nofollow">https://llvm.org/docs/Atomics.html#unordered), and Zig lets you use it (<a href="https://ziglang.org/documentation/master/#atomicStore" rel="nofollow">https://ziglang.org/documentation/master/#atomicStore).
> In particular, you could have unsafe code in crate A, whose data are then used by crate B.
Is this backwards? If B consumes data from A then to me that does not imply that A depends on anything from B; for a more concrete example that sentence reads to me like A is basically "throwing data over the wall" to B and whatever B does with said data is of no relevance to A. As a result, if B changes that shouldn't affect A.
Also for what it's worth I get the impression you and treyd might be talking about slightly different things when talking about whether unsafe code composes. I believe treyd is referring to the RustBelt series of papers [0, 1], for which the statement "unsafe code composes" means (at a high level) that adding a module with a memory-safe API to a memory-safe system will result in a memory-safe system as long as the implementation upholds the safe semantics. Yes, the last bit can be a rather significant caveat, as you said.
What you're talking about seems more along the lines of needing to look beyond the boundaries of unsafe blocks to prove that the unsafe block upholds its invariants, which is also true. I think you only need to check within whatever safe encapsulation boundary is relevant, though, rather than globally.
> Is this backwards? If B consumes data from A then to me that does not imply that A depends on anything from B; for a more concrete example that sentence reads to me like A is basically "throwing data over the wall" to B and whatever B does with said data is of no relevance to A. As a result, if B changes that shouldn't affect A.
This is a specifically crafted bad idea, but you could have module A use unsafe to craft a Vec<u8> that is safe to use to read or write, but not to grow or shrink. You declare an invariant that the receiver shalt not grow or shrink the Vec.
If B only reads and write, you're good. But if a future B breaks the invariant, bad things happen. As I said, specifically a bad idea; there's a much better type to use if the thing can't grow or shrink...
No real world example, because I don't think we've run into memory safety issues with unsafe in the Rust code base I work in... but we only use unsafe where it's required (syscalls and other FFI).
Fair. I suppose that even in such a scenario you shouldn't need truly global analysis to prove safety - in principle an analysis of A should reveal the soundness precondition on a safe API - though that's probably easier said than done.
plqbfbv · · focus · HN ↗
I programmed in rust a bit and can't say I'm an expert, but in my view rust mostly-solved the memory management problem at compile time and without a GC, and it works very well. The biggest con and cost I've always seen repeated so far is that "it's slow to compile", and I get that, if you're past 250 crates the final --release link tends to become noticeable, but there were improvements to incremental compilation.
On the other hand - looking at the syntax from this post - Zig feels a blend of javascript, python and golang syntax that still requires memory management. So a nicer-written C that inherits all the issues from C? From the post: no functional programming, data mutation, memory leak, double-free, memory corruption.
Personally I'd rather trade a couple minutes of final link every time when this is the other option.
pron · · focus · HN ↗
As a long-time low-level programmer, and as someone working on a popular mainstream language, I find Zig fascinating, and I also think it addresses a long-standing problem in low-level programming. I'll get to the problem later, but the fascinating part is its use of partial evaluation (comptime) as a single coherent mechanism that replaces a myriad of other partial-evaluation mechanisms (macros, templates/generics, constexprs). That one mechanism is the core of the language, like macros are in lisps, and that design - whether you like it or not - is revolutionary. It's never been done before (other languages have partial evaluation mechanisms that are almost as general, but they're offered in addition to, not as a replacement of, other features).
> rust mostly-solved the memory management problem at compile time and without a GC
"Mostly" does a lot of work here because 1., if you look at the implementation of very efficient, possibly specialised data structures - the very thing you reach for a low-level language for - they typically require unsafe, and 2., it still suffers from the problem C++ has had for decades, which is that over time, as program changes and evolves over years, things tend to drift toward the more general mechanisms that rely on malloc/free on an individual objects, and the program gets slower and slower (huge runtimes like TCMalloc help, but not enough, because they can't move pointers). This problem, of programs that start out fast, but after five or ten years of evolution need to spend a lot of effort to remain fast, is one of the things moving collectors were designed to solve, but they require moving pointers, which doesn't work in low-level languages that are not meant to have an FFI layer between them and the hardware.
To compete with the performance of moving GCs, which allocate through bumping a pointer, like on the stack, and free memory in bulk, low-level languages need to rely on arenas (which work based on a similar principle), and Zig is the first language that makes arenas almost user-friendly and hopefully sufficiently composable to withstand program evolution. Of course, time will tell how well this works in practice.
treyd · · focus · HN ↗
There's a formal proof asserting that if you keep up the safety invariants within an unsafe region then that will not infect other code, even in the presence of arbitrary other correctly-written unsafe blocks.
This means you can build abstractions on top of these low-level primitives to keep it contained, so consumer code never has to even think about or know there's unsafe blocks in it. The type system lets you build very powerful abstractions so these go a long way.
There's a lot of woo-woo scare quoting around how much you actually have to use unsafe code in Rust. It's fairly uncommon to actually have to reach for them in practice. Most of my usage ends up being things like converting a &[u8] to a &str when I know it's already valid UTF-8 so I want to skip the linear-time validity check. Very rarely do I have to build data structures with complicated pointer juggling, because there's often a library that already does what I need!
> which is that over time, as program changes and evolves over years, things tend to drift toward the more general mechanisms that rely on malloc/free on an individual objects, and the program gets slower and slower
What are you talking about? I've never encountered this and I've been using Rust for 10 years.
dnautics · · focus · HN ↗
In general "unsafe" does not compose.
"if you keep up the safety invariants within an unsafe region"
This condition is doing a lot of heavy lifting.
treyd · · focus · HN ↗
I'm curious why you think that statement is doing heavy lifting. It's much easier to write and verify that a few lines of code are correct than it is to write and verify that an entire program is correct. But that's the norm in C and Zig, and historically people haven't been very good at it. That's why we try to do it as little as possible.
pron · · focus · HN ↗
The reason it's not "the norm" is that (especially with spatial safety taken care of), not every line is equally dangerous at all. Still, there's no doubt that more guarantees help, but that is only when all other things are equal. If you pick a low-level language for mostly low-level things, so Rust doesn't offer safety for the trickiest code, and furthermore it makes certain things harder to see because the language is more complicated, then things become much less clear. Obviously, when the vast majority of the trickiest, most important code doesn't need to be low-level, Rust would probably be safer on the whole, but in such situations I see no reason to choose either Rust or Zig. You need to choose a low-level language if the core of what you're doing needs to be low-level.
aw1621107 · · focus · HN ↗
Is there a word missing before "memory"? Seems odd to specifically call out spatial memory safety when memory safety subsumes it.
steveklabnik · · focus · HN ↗
ngrilly · · focus · HN ↗
pron · · focus · HN ↗
dnautics · · focus · HN ↗
You must reason about the invariants in unsafe code on a global level. In particular, you could have unsafe code in crate A, whose data are then used by crate B. It could be fine. But then crate B changes its implementation which now violates the invariant expectations of crate A.
pron · · focus · HN ↗
To address the problem that once integrity can be violated anywhere, only global analysis can prove that nothing bad happens, we've done two things:
1. We require the application to explicitly permit any integrity violation by a module; i.e. a library can't allow itself to violate integrity. This is a principle we call "Integrity by Default" (<a href="https://openjdk.org/jeps/8305968" rel="nofollow">https://openjdk.org/jeps/8305968).
2. We try to minimise the need for potential integrity violations (this is very different from Rust, which requires unsafe even for things like benign write/write races, which are fairly common, and various basic data structures). Over the years we've offered safe replacements for things that used to require Unsafe. In other words, clearly demarcating unsafe code isn't enough if it's needed at all in many situations.
It isn't perfect, of course, as some libraries do require unsafe operations for direct interaction with native code or with memory, but their number has been greatly reduced, and they cannot do this without the application's explicit approval. Interestingly, this has annoyed library authors who want to do unsafe things but don't want to application authors to be alarmed because "we know what we're doing," and it's also annoyed some application authors who want to use such libraries and are forced to explicitly add permissions. But I think that the community, as a whole, has eventually accepted this because the harm done to those who don't care is small (they just need to add the permissions), to those who do care it helps a lot, and because fewer and fewer libraries require "integrity-busting" permissions, many applications need to do absolutely nothing and get important guarantees for free.
aw1621107 · · focus · HN ↗
I know this paper [0] is quite old at this point, but the mention of benign data races reminded me of it. Would you happen to know how applicable it is to modern memory models?
[0]: <a href="https://www.usenix.org/legacy/event/hotpar11/tech/final_files/Boehm.pdf" rel="nofollow">https://www.usenix.org/legacy/event/hotpar11/tech/final_file...
pron · · focus · HN ↗
However, in C and C++ (and Rust) benign non-atomic write/write races are UB (indeed, LLVM also treats them as potential causes of UB). In C# and in Java they are safe (although Java currently only has non-atomic writes on 32-bit machines, but soon they'll be more common when value types are enhanced). LLVM even has a specific construct to support the Java-style memory model (<a href="https://llvm.org/docs/Atomics.html#unordered" rel="nofollow">https://llvm.org/docs/Atomics.html#unordered), and Zig lets you use it (<a href="https://ziglang.org/documentation/master/#atomicStore" rel="nofollow">https://ziglang.org/documentation/master/#atomicStore).
aw1621107 · · focus · HN ↗
Is this backwards? If B consumes data from A then to me that does not imply that A depends on anything from B; for a more concrete example that sentence reads to me like A is basically "throwing data over the wall" to B and whatever B does with said data is of no relevance to A. As a result, if B changes that shouldn't affect A.
Also for what it's worth I get the impression you and treyd might be talking about slightly different things when talking about whether unsafe code composes. I believe treyd is referring to the RustBelt series of papers [0, 1], for which the statement "unsafe code composes" means (at a high level) that adding a module with a memory-safe API to a memory-safe system will result in a memory-safe system as long as the implementation upholds the safe semantics. Yes, the last bit can be a rather significant caveat, as you said.
What you're talking about seems more along the lines of needing to look beyond the boundaries of unsafe blocks to prove that the unsafe block upholds its invariants, which is also true. I think you only need to check within whatever safe encapsulation boundary is relevant, though, rather than globally.
[0]: <a href="https://people.mpi-sws.org/~dreyer/papers/rustbelt/paper.pdf" rel="nofollow">https://people.mpi-sws.org/~dreyer/papers/rustbelt/paper.pdf
[1]: <a href="https://plv.mpi-sws.org/rustbelt/rbrlx/paper.pdf" rel="nofollow">https://plv.mpi-sws.org/rustbelt/rbrlx/paper.pdf
toast0 · · focus · HN ↗
This is a specifically crafted bad idea, but you could have module A use unsafe to craft a Vec<u8> that is safe to use to read or write, but not to grow or shrink. You declare an invariant that the receiver shalt not grow or shrink the Vec.
If B only reads and write, you're good. But if a future B breaks the invariant, bad things happen. As I said, specifically a bad idea; there's a much better type to use if the thing can't grow or shrink...
No real world example, because I don't think we've run into memory safety issues with unsafe in the Rust code base I work in... but we only use unsafe where it's required (syscalls and other FFI).
aw1621107 · · focus · HN ↗
toast0 · · focus · HN ↗
aw1621107 · · focus · HN ↗