Very little of the article is actually focused on stochasticity. I'd also say drawing an equivalent between the non-determinism of a person and an LLM is not that accurate either.
It's for example impossible to have a discussion with an LLM where you both learn something which you can apply tomorrow. The LLM doesn't learn until the next model is released and by then your discussion is just a tiny fraction of the training data (if present at all). AGENTS.md, skills and so on are just a proxy for what we actually want, an agent that listens and understands. A proxy mind you, that requires constant tweaking with no sign of generalisation in sight.
Does humans being non-deterministic make coding solved? I'm not sure how this relates to the main point.
I'm also not sure what humans being non-deterministic even means here. The point is if you're comparing results with NFR, pure agentic coding falls short.
I'll spell it out... This guys is saying "LLMs are non-deterministic and therefore it's a bad idea to use them for coding", but humans are also non-deterministic and yet we somehow manage to get by with humans coding.
The fact is there's no way of coding in a deterministic way, so it's irrelevant that LLMs are non-deterministic.
You ignored my comment about NFR. But, if you really wanna harp on the non-determinism. I don't think humans and LLMs are actually non-deterministic in the same way?
Humans are non-deterministic when compared between humans. But if you give a human the same task they can be pretty consistent. Models quality varies much more than a single human does and human's reasoning don't reset with their context window. Otherwise tenure wouldn't matter, companies wouldn't try to retain specific talent, and pay bands wouldn't be so wide. But thanks for spelling it out.
You didn't seem to have a reputable point to address there. Just an assertion.
> I don't think humans and LLMs are actually non-deterministic in the same way?
Perhaps, but the author didn't make that argument. It's definitely easier to quickly vary skill in a model (by switching models) than it is to vary it in humans (by hiring/firing).
I don't see why that's bad. If anything it's better for AI - way easier to say "always use at least Astra/Opus 5.5" than it is to fire your crap employees.
I think some people have had the fortune to never actually work with poor programmers. They exist!
> You didn't seem to have a reputable point to address there. Just an assertion
I mean in my anecdotal evidence, AI has not been meeting bigger picture requirements and needs hand holding. Even on the newest models. So are you not experiencing that?
> I don't see why that's bad. If anything it's better for AI - way easier to say "always use at least Astra/Opus 5.5" than it is to fire your crap employees.
So you're saying you'd rather have AI make a consistently bad engineer inconsistently mid and a consistently good engineer inconsistently mid over what they usually are?
I much prefer consistency in my engineers. Especially when investing in them helps them improve.
And the reason I think it makes good engineers inconsistently mid is because look at the drop in quality as a customer from big tech which hires top talent. I had to cancel my Spotify subscription because it's become so sloppy.
> AI has not been meeting bigger picture requirements and needs hand holding. Even on the newest models. So are you not experiencing that?
Ah yeah in that case I agree. We're not quite at the point where completely hands-off vibe coding gives good results. But it seems very unlikely that that will still be true in a few years.
I mean I really hope that isn't true and AI stops improving or has some fundamental weakness that means it can never make long term maintainable code. Then I get to have a job! But I'm not going to believe something just because my job depends on it...
IshKebab · · focus · HN ↗
Doesn't matter what you think about AI, "it isn't perfect" is clearly a nonsense reason not to object to it.
vincent-uden · · focus · HN ↗
It's for example impossible to have a discussion with an LLM where you both learn something which you can apply tomorrow. The LLM doesn't learn until the next model is released and by then your discussion is just a tiny fraction of the training data (if present at all). AGENTS.md, skills and so on are just a proxy for what we actually want, an agent that listens and understands. A proxy mind you, that requires constant tweaking with no sign of generalisation in sight.
gyesxnuibh · · focus · HN ↗
I'm also not sure what humans being non-deterministic even means here. The point is if you're comparing results with NFR, pure agentic coding falls short.
IshKebab · · focus · HN ↗
The fact is there's no way of coding in a deterministic way, so it's irrelevant that LLMs are non-deterministic.
gyesxnuibh · · focus · HN ↗
Humans are non-deterministic when compared between humans. But if you give a human the same task they can be pretty consistent. Models quality varies much more than a single human does and human's reasoning don't reset with their context window. Otherwise tenure wouldn't matter, companies wouldn't try to retain specific talent, and pay bands wouldn't be so wide. But thanks for spelling it out.
IshKebab · · focus · HN ↗
You didn't seem to have a reputable point to address there. Just an assertion.
> I don't think humans and LLMs are actually non-deterministic in the same way?
Perhaps, but the author didn't make that argument. It's definitely easier to quickly vary skill in a model (by switching models) than it is to vary it in humans (by hiring/firing).
I don't see why that's bad. If anything it's better for AI - way easier to say "always use at least Astra/Opus 5.5" than it is to fire your crap employees.
I think some people have had the fortune to never actually work with poor programmers. They exist!
gyesxnuibh · · focus · HN ↗
I mean in my anecdotal evidence, AI has not been meeting bigger picture requirements and needs hand holding. Even on the newest models. So are you not experiencing that?
> I don't see why that's bad. If anything it's better for AI - way easier to say "always use at least Astra/Opus 5.5" than it is to fire your crap employees.
So you're saying you'd rather have AI make a consistently bad engineer inconsistently mid and a consistently good engineer inconsistently mid over what they usually are?
I much prefer consistency in my engineers. Especially when investing in them helps them improve.
And the reason I think it makes good engineers inconsistently mid is because look at the drop in quality as a customer from big tech which hires top talent. I had to cancel my Spotify subscription because it's become so sloppy.
IshKebab · · focus · HN ↗
Ah yeah in that case I agree. We're not quite at the point where completely hands-off vibe coding gives good results. But it seems very unlikely that that will still be true in a few years.
I mean I really hope that isn't true and AI stops improving or has some fundamental weakness that means it can never make long term maintainable code. Then I get to have a job! But I'm not going to believe something just because my job depends on it...
idz · · focus · HN ↗
IshKebab · · focus · HN ↗