> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
The more likely AI becomes to produce working code every time, the more likely it will become that a one-in-a-thousand or one-in-a-million error goes unnoticed at generation time. It sucks.
Imagine the first time electric calculators calculated the square root of 5. I'm sure people would verify again and again if what the circuits calculated was right.
Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.
How crazy will the world be once our [current gen] ANN are like that!
Given the disappointing levels of intellectual decay that our current technology has thrust upon civilization, I only see humans reverting back to neanderthal levels of intelligence in short time with the advent of AI
> One wonders whether a generation that demands instant satisfaction of all its needs and instant solution of the world's problems will produce anything of lasting value. Such a generation, even when equipped with the most modern technology, will be essentially primitive — it will stand in awe of nature, and submit to the tutelage of medicine men.
There's no reliable evidence that Neanderthals were less intelligent than modern humans. They're extinct now (except for a tiny genetic legacy in some human populations) but that could have happened for a variety of reasons unrelated to intelligence or lack thereof.
No you don't. If people don't learn to solve problems and exercise their mind with small problems, how are they going to deal with big problems? The "neanderthal" part is a colorful expression.
First AI came for the parrots and I did nothing, at least I can stand for the neanderthals.
There are so many parallels between what TV could have become and the progression of AI, a nearly free conveyance for culture, education, art and discourse. Yet, we allowed it spiral in a positive feedback loop, creating a cognitive gyre that now razes society.
At least the youngest now use "that's AI" as a pejorative, as in that is bullshit.
Calculators and computers are deterministic, they give the same output to the same output every time. Language models specifically are not. So it might give you are function that is correct, or a function that is not, or worse yet a function that behaves correctly, but introduces some god-awful bug down the line that can cause serious havoc. It is obvious that they need supervision, not only for output, but also sandboxing and various harnesses for them to not do any “oops, I deleted your codebase sry” kind of nonsense people post to Reddit.
So I think the problem is to determine which problems under what instructions we can safely give to a model application to solve and how we test the output for safety and functionality. This would create more usable and safe, albeit a bit more boring, AI-based applications alin to a calculator or general computer. Whether this is posswith current model architecture is another thing.
Memory bits flip randomly. It's not a super rare thing either. You and me have experienced that many times without knowing. The only reason that computers feel deterministic is that we have error-correcting code to fix that. But in the most extreme cases, when multiple bits flip together, once "deterministic" program can generate unexpected output.
So why do you trust computers? Because statistically the case is just very unlikely. Therefore if AI is statistically unlikely to make mistakes there is no reason to not trust them.
You are confusing hardware noise and whether the algorithm itself is verifiable as if parity bits were not a thing. We don’t trust computers because the medium itself is infallible, but because the logic is sound.
With statistical models - such as LLM’s - there is no logic as such, but statistical assumptions based on given data. The output can ge very good or very bad, but you are fool to trust it blindly. Therefore you need a deterministic way to verify, whether meat- or software-based.
A program can only be as deterministic as the hardware it runs on. If you have a turing machine whose symbols on the tape just (truly) randomly change every step, then none of programs on this machine is "deterministic."
> meat-
There is no meat-based deterministic way to do anything yet yet. You're talking sci-fi.
Why will X not keep increasing? It’s been increasing, albeit very slowly, since the start of the 20th century as the disease burden declines and nutrition improves. I see no reason to think that future health interventions couldn’t further juice those numbers.
Will it keep up with Y? Probably not, unless people are willing to accept pretty radical interventions to their biology. But it almost certainly is not static
If you set the temperature 0, an LLM is also deterministic (same prompt -> same output every single time). We just don't do this because the LLM is not so smart in that mode. But "LLM is not so smart" is changing at Moore's law speeds.
Non-determinism is not an essential property of LLMs. It's an optimization that we've added intentionally.
Ehm, no! The same prompt does not always lead to the same output.
Have you ever tried to achieve consistently deterministic output from an LLM? I have, and it's not easy.
That means output differs between machines and architectures. Running inference on CPU vs GPU also affects output. Even running the same prompt twice in a row on the same machine can lead to different outputs because a prompt that was partially stored in the kv cache will result in different output than an uncached prompt.
At the end of the day, an LLM is just a very big mathematical function. That is, by definition, deterministic. A particular implementation might give up on determinism for the sake of higher efficiency, but it you want a deterministic LLM, it can absolutely be done.
Yes, yes, assuming the same CPU and stuff. There is some randomness due to floating point math differences between CPUs (and sometimes on the same CPU), but I don't think that fundamentally changes my point.
Put another way: if you could have a virtualization layer that guarantees deterministic floating point operations then a LLM set to 0.0 temp would produce deterministic output.
It is insane how many times I see this false analogy repeated on HN over and over (analogy of a deterministic-by-design calculator device (or a compiler, etc.) and a non-deterministic-by-design LLM software).
You're missing the forest for the trees here. The point OP is trying to make is that calculators originally were essentially non-deterministic. Technology will go from unreliable to reliable.
LLMs are not non-deterministic by design. The randomness in the output is artificially injected for ergonomic reasons. (Yes, the non-determinism in production systems is different, but also not by design)
You don’t train calculators on how to add. All calculators will more or less come up with the same answer to some deterministic and quantifiable level of accuracy and precision.
You oppose correctness to meaning and purpose, which you seem to imply are impractical values. (Worthless values, then?) But you don't mention creativity. The article blithely says that AI creates new ideas and understands things. I don't think it does.
> What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.
We're not going stop doing something locally just because we can do it in space. There are definite physical advantages to producing next to consumption. We will most likely do both -- things that can be done in space will be done in space, and things that are better done locally will ramp up. There is no upper limit to energy demand, none whatsoever. Stopping production anywhere it could theoretically happen is a matter of will.
The more I think about space elevator, the less I believe in it.
The material alone is in a quantity beyond what we can reasonably manufacture.
and the material needs to be perfect. All design we have today have cascade failure mode -- any material failure translates to a total catastrophic failure.
and geostationary does not really meant Geostationary. There are lots of jiggling everywhere. It wear down over time. and let's hope nothing resonance
and we need some maintenance / decommission plan. How can we decommission this when it fail or need upgrade?
It's quite possible that with "infinite" energy we could do mass scale carbon capture to offset the increased temperatures by rolling back the greenhouse effect.
This is basic thermodynamics, not greenhouse gases. If you put some amount of energy into a system with a fixed surface area, you will get a minimum equilibrium temperature.
Greenhouse gases and Earth’s internal nuclear decay engine make the situation worse, but even without them this would boil the oceans.
Even today we're recognizing that compute does not need to happen here.
When we're talking about creating powerplants equal to roughly 5% of the insolation of Earth, I think we're sci-fi enough to discuss orbital datacenters or Mars datacenters or Jupiter fusion candle datacenters.
Compute doesn't need to happen here, but it can, so it will. Building datacenters on Mars or Jupiter doesn't reduce our ability to build datacenters here, so naturally, they will still be built, everywhere that they can be. It's an orthogonal capability.
Maybe, but there are counterexamples today. The processing of raw materials can be done all over the world, but it's not. Modern nations actually disassemble their blast furnaces.
Orbital datacenters suffer from the same problem: limited surface area. You think cooling is a problem now? Try not having conduction or convection. Radiation is all you get. A huge proportion of the mass and size of the ISS is radiators, and that only has to deal with the heat of low-power electronics and a few measly human beings.
Using an entire Mars only nets us about a 2x multiplier for our energy expenditure budget. Then we need four planets to double it again. Exponential growth is a bitch.
Thermodynamics and the tyranny of exponential growth are going to win this battle every time, regardless of the unobtanium technology you try to invent.
Not quite the same problem, because only a small portion of that energy radiated from space will make it back to the environment on Earth. For convection that is 100%.
> Using an entire Mars only nets us about a 2x multiplier for our energy expenditure budget.
That is not the problem being discussed. GPUs can survive at temperatures most organisms cannot. Radiators can be scaled up, but the thermal mass of Earth cannot.
I am quite certain that you are drastically underestimating the scope of the problem of "getting rid of heat" when you're dealing with the discussed levels of power production.
Maybe, but that's a "simple" resource problem. The Earth radiates 20x more heat than that today, and it has not been optimized for purpose. Remember that the initial assumptions were that AI has already designed and manufactured tens of thousands of these plants.
The problem at hand is not just getting rid of heat however, it's getting rid of heat without cooking the Earth. I still fail to see how generating the heat off planet fails to solve that.
This assumes other technological breakthroughs don't also happen in parallel. I'll go out on a limb with nothing more than a hunch that we are in the beginning stages of understanding mathematics, physics, and biology.
It does not. This is basic thermodynamics. Unless we are using computers made out of something other than matter, powered by something other than energy, thermodynamics bounds us all.
Ok, but what if the AI figures out a new thermodynamics? A completely new thermodynamics, that's vastly more powerful than human thermodynamics, but of course humans won't be able to comprehend it, because it's so much more powerful, that you need the superintelligence to handle it. With a thermodynamics like that, anything could be possible, and remember: what we have now is the worst it will ever be.
If we had such arbitrarily large amounts of energy to handle, it would be conceivable to beam excess energy into space. It is a bit on the sci-fi side, but so is that kind of energy production.
>the economically dominant strategy to not verify them and not double check them
In the big scheme of things is it really that expensive to verify it if a lean proof is generated? The agent itself will likely have already verified such Lean code before calling it "done".
My dad recently needed to buy a new thermostat for his home with an air furnace (yes, he told the model) asking an AI which one to buy, and he got recommended one that only properly works with boilers. Then after that happened, the alternative he bought the AI never told them he needed to buy a gateway to connect to his furnace.
I think we are a long long looong way from AI designing 'terawatt fusion plants'.
>Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say.
I think you have a fundamental misunderstanding here, and it's not really explained because I think it seems self-evident from within the field. In short: writing code is a means to an end; doing mathematics research is not, but is the end in itself.
The human involvement is crucial because the entire purpose of mathematics research is to increase human understanding of mathematics. It is pursued because it is interesting, not because it is economically useful. In this sense it's a lot closer to the humanities.
A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field (except insofar as it could be harnessed to improve human understanding).
Coding is totally different from this, where it is essentially always done as a means to an end. Likewise with many other fields, like pharmaceutical research or materials science or what have you, that are oriented around solving problems for some practical purpose. Pure math isn't really like that for the most part.
Sure, but I don’t think most of the money that goes into funding math is for the purposes of pure understanding. The reason governments fund mathematics research grants is generally for a more instrumental purpose; taking the US congress as an example, the mission of the NSF is to, “Promote the progress of science; advance national health, prosperity, and welfare; and secure national defense.” Most federal math grants come from the NSF.
Of course, math research is cheap and most academics don’t rely upon grants, their salary covers most of their expenses. But here too, the mathematics professor spends a substantial amount of their time teaching future engineers/quants/other applied mathematicians, who need to understand math for instrumental purposes, not as an end in and of itself. Without the tuitions of these students, I can’t imagine universities maintaining the size of their math departments, let alone expanding them as Dr. Sahai advocates for.
So who or what funds the community of pure mathematics going forward?
Research in pure mathematics is part of what we call "basic research". There are no applications in mind a priori. People instead focus on understanding, because history has taught us that understanding tough problems in mathematics finds natural applications elsewhere. It's the same as theoretical physics or theoretical computer science.
> People instead focus on understanding, because history has taught us that understanding tough problems in mathematics finds natural applications elsewhere.
If the goal is still eventually the applications elsewhere, we're back to what happens if the AI is simply better at this.
You can probably make an argument that human understanding is better as humans are better at finding new patterns or fundamental new ways of thinking and also applying them to new applications.
However, what if AI becomes better at humans for that as well?
No reason you couldn't have an AI be optimised for advancing basic research and understanding and a second AI to take these results and optimise for finding new applications for these discoveries.
> what if AI becomes better at humans for that as well?
The "if" is the problem. If it happens, then of course, let AI do it. For the moment AI is still bad at those type of tasks [1], so the discussion shouldn't focus on highly conjectural situations. We can't destroy the scientific ecosystem based on vague speculations.
[1] There are real reasons: it is not obvious how to optimize an LLM for doing basic science or other ill defined tasks. On the contrary, optimizing for writing a proof that passes the Lean test or code that passes the tests is a different story.
Basic research is funded with the understanding that applications are not imminent, yes, but also with the expectation that some of the knowledge gained will eventually result in advancements to the public welfare
I think you have misunderstood the OP's point here. You're arguing that deepening human understanding is an end in itself, and you are right. The OP is arguing that advances don't need to be pegged to human understanding, and they are right too. The two can coexist, superintelligence far ahead of us, pioneering discoveries - and mathematicians catching up at a pace suited to biological minds. I don't see the issue here. Of course, it does mean mathematicians adopt a new role as hobbyists.
> A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field
This is a crude distortion. The recent breakthroughs have come with proofs, reasoning and verification, and there is no proposal that I'm aware of that would do away with these foundations. There's also some rather ugly solipsism in the idea of keeping what interests the field as a limit. Mathematics has broader relevance to humanity than merely to please and support mathematicians, and if other fields can make practical use of profound well-proven future math, mathematicians will have a hard time making a case that their comprehension must come first.
> > A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field
> This is a crude distortion. The recent breakthroughs have come with proofs, reasoning and verification, and there is no proposal that I'm aware of that would do away with these foundations.
I believe that we are still at the point where these proofs serve as verifiable certificates of correctness, so that it's not a "trust me bro" situation, but where humans mostly still don't find them understandable, so that they are still just a highly reliable black box.
For even more people fun is TikTok, Snap, Instagram -> sounds like a collapse of a civilization to me if you increase the ratio even more towards dancing kids sharing their content non-stop with no added value to the society
It's not fun, it's a physiological addiction, because we learned enough about our brains to hack dopamine and reward cycles for profit. It's literal abuse.
The average person doesn't know how the medication they take works, the mechanics of climate and climate change, how the energy they consume is generated, etc.
> The medical field as a whole isn’t generally interested in understanding how medication, only in empirical measuring and qualify the effects.
I don't think this is quite correct. I mean many practitioners of medicine will have the attitude of ... "if it works, it works". And that's perfectly reasonable.
But if you understand the mechanism of action of a drug (or other treatment), it (often) makes it easier to improve a drug.
So ... some sectors of the "medical field" understandably care only about empirical results. But other sectors would prefer to understand what's going on.
>The medical field as a whole isn’t generally interested in understanding how medication, only in empirical measuring and qualify the effects.
This is nonsense. Most, if not all professionals are interested in mechanism of action, but without Ms Frizzle, it is extremely difficult and expensive (time and money wise) to figure that out. So while the labs run the experiments with the very limited funding they have, we make do with using the second best thing we have, which are statistics.
We don’t really understand how NSAIDs reduce pain, as a concrete example. In contrast we have a pretty good understanding of the pharmacology of caffeine.
It’s not like magnets, some things really are gaps.
Some medications are designed for one purpose, then other effects are discovered in practice. Gabapentin, for example, was designed as an anti-seizure medication structurally similar to the inhibitory neurotransmitter GABA.
Now it is primarily used to treat neuropathic pain, and the mechanism for that is not well understood. The GABA receptor is not involved. This effect is just a happy accident, and nobody really understands why it works.
I feel like we just don't understand a lot about the human body still. Only a century and a half ago we had a US president die because a doctor rummaged around and damaged internal organs (with unwashed hands, because germ theory was still not uniformly accepted) to try to find a bullet on the wrong side of his body. We've come a long way since then, but it's not like we've been doing the field of medicine rigorously for all that long. The best strategy we have seems like pretty much the same as any other scientific field; come up with ideas based on what we know, try them out, and see if they work. That doesn't necessarily give us any actual insight into why it works though.
> nobody really understands why a medication works.
But they work hard to try to understand it because the more they do the better the results for peoplea health. Also it goes without saying that if someone didn't understand many of the things that we do understand then things would be worse for us all.
The problem with "ignorance" in Western countries (particularly the US right now) is that it's very common for people who don't know to believe they know and form ignorant opinions that they often want to be applied society-wide in some way. You can see this with everything from climate change to vaccines.
In much of the world, even in middle income countries, people are comparatively poor and, in my experience living abroad in such countries for many years, much less concerned with "understanding" and forming opinions about everything under the sun. It doesn't mean they don't value education and are opposed to development/progress, but it does mean that they don't question whether the vaccine they're taking is the product of a conspiracy, think too deeply about why the river is flooding more often, etc.
They just deal with life the best they can and are more focused on supporting their families, enjoying what they can, etc.
Culture and religion play into this. The way secular and Judeo-Christian people look at the world is very different than, say, Buddhists, Muslims, Fulani tribespeople, and so on.
But we can generally assume that the people who built and understand some technology will have the same desires and worries as us, and are bound by the same laws. For instance, if it was a team of strangers who designed the terawatt fusion plant, we can still assume with some confidence that they care about safe plant shutdown, failure recovery, ensuring there are no toxic side effects of operation, etc. Can we say the same when it is machines doing everything?
Very interesting point. I think the counter-argument to that is that the complex modern society is based on somewhat “deterministic” systems, in that, even if a single decision or event isn’t rationally explainable in the moment, at least in the aftermath, it typically becomes understandable, maybe even reproducible. There is someone, somewhere, capable of explaining, maybe even multiple someones.
We don’t generally have that insurance with LLMs/AI, yet?
Imo the counter-argument is that in principle you can research and understand the mechanics of climate change or medicine. You just don't have the free time and/or the motivation. With AI that changes - you could spend your entire life trying in vain to understand its reasoning.
It reminds me of how some religious people say that science is effectively no different to religion because we all take expert opinions on faith. But the difference is that there is a well-defined pathway to understanding, if you wish to do so.
This seems to me like asserting everything in the universe is explicable by physics. It may be technically true, but still not relevant to understanding earthquakes.
(Don’t bother to argue this not true unless you disagree with the essence of the argument.)
Anyway, post-hoc explicability isn’t a counter-argument to the assertion that almost everyone takes almost all technology as magic, from medicine to computers.
I’m still trying to understand your argument. Are you saying that after the fact we understand AlphaGo move 37? But somehow we are never going to understand an LLM’s decision afterwards? Seems like a disconnected take to me.
A major school of German sociology places this at the core of its theory. According to this theory, modern societies are characterised in particular by the fact that individual subsystems of society reduce the complexity of their own (sub)system to the other systems to enable them to act at all; they fulfil what is called an "Entlastungsfunktion" (relief function). Prominent representatives of this school of thought are Max Weber, Arnold Gehlen and Niklas Luhmann.
In their view, it is modern institutions (public and private) which, as supra-individual entities, have long since become autonomous systems. The fact that the individual office-holders are human beings, meanwhile, is of little significance.
Hannah Arendt, in her theory of totalitarianism, attributed the effectiveness of both Nazi and Stalinist policies of extermination to the largly moral indifference of bureaucracy as a system.
In this sense, the task of controlling AI is a variation on the problem of harnessing a complex society consisting mainly of autonomous subsystems. This is a problem which has increasingly challenged humanity already for quite a long time. And it has been very difficult so far, even without AI ...
That's what makes libertarian AI doomers so strange. "Letting unconstrained machines with human components loose on the world will be okay, but letting unconstrained machines with nonhuman components loose on the world will produce terminators." How can it be both one and the other?
the average person doesn’t have an internal monologue
can’t use a computer (they’ve had like 30 years now in first world developed countries)
many can’t even use their smart phone beyond calling, texting (many can’t type well), and doom scrolling (they get addicted to drugs, gambling, and other LCD activities)
many read at a 6th grade level. most can’t even calculate tip in their head.
meanwhile, the same smartphone can give them access to literally any information and knowledge the world in seconds. and now gemini can explain stuff since most ppl can barely read or think.
Those subcultures are like that because they are small. If that became the lot of most people, society would look very different. The choice would be between Fully Automated Luxury Communism and Oligarchic Hellscape.
Some of it has to do with the fact that many people pushed forward training themselves in a way they understood would better themselves for the future. They put in time, and now their sense of value is in question. They're forced to think about things like value (of life, themselves, and hopefully others), and the things they want to remain valuable in the future.
It's interesting to me that you only mentioned the people using 'AI' in the 'short-term' ways, and not the ones that use it to better themselves in the 'long-term' ways. You can spend your own time focusing on either usage, it's really up to you and your concerns. Either group's sense of value is what determines their behavior. Where they spend their time and thinking must be elsewhere, and you disagree with it. Who judges the quality of time spent? You do.
Is it more useful to think about self-improvement, and how to navigate the future in ways that might help you re-establish value of yourself, life, and others? Acquiring knowledge is a struggle, the author mentioned this. There is also Plato's Allegory of the Cave, which highlights some of that struggle, a resistance to change. And we're all limited by time, our genes, our station in life.
The only way to help anyone out of the cave, is to help them believe something different about themselves. To help them believe there is good reason to spend time going deeper into knowledge, or at the very least, allow others with the passion and station for it to do so.
If not for the very least reason that it keeps us 'in the loop' of some central idea behind intelligence (prediction?). Or because we feel it keeps us safer, as a fallback measure because we acknowledge we have to trust other's knowledge to exist.
One day machines will probably do most of the work to be done better than we can. Which is fine. But then we'll also lose the meaning and purpose that work used to provide. I don't think it would necessarily lead to a dystopia, but that world would be a pretty strange place to our current eyes, to say the least.
Before extrapolating that far, take a look at the frontier labs' own job boards (<a href="https://openai.com/careers/search/" rel="nofollow">https://openai.com/careers/search/?). Isn't it curious that they are still recruiting human "Android Engineers", "Account Associates", "Consumer Marketing Leads" instead of automating them with their world-beating models?
It's about as fun, and as realistic, as imagining magical ponies and people with superpowers?
This is such an incredibly naive and absurd vision; we've already proven that humans are very often very bad at implementing other humans' good ideas. There's nothing that AI is likely to bring that will improve this discernment.
Here's the thing: nobody is stopping you from putting in the time to understand all that. The problem is, nobody has that much time, and we get hungry. And so we want the community to move with us, spend the time the same way as we do, to ensure value. We are all saying: we want someone else, others, to put in that time for us. The truth is, we all want quality, and value is closely related.
I can almost guarantee you the first time you show cancer symptoms, you won't care whether the cure came from an AI or human's understanding. But we haven't seen that, so we can't make the judgement call.
I’m reminded of themes from the Hyperion Cantos! Maybe my mind is over-connecting, but it’s not the first time I’ve drawn similarities in the last few years.
It’s terrifying to me to think we’d let AI make things for us we never understand. Like livestock not knowing how auto-feeders dispense their daily food were built and appeared, they just gladly eat until…
> We would probably no longer care about code, engineering or even physics and mathematics among other things
This sounds like a boring existence. I take your meaning, but want to point out that not everyone learns about things because of practical utility, some of us find it incredibly satisfying to learn how things work just for the sake of learning.
Sure but no one's stopping you? I know a great many things of no direct practical value to me.
There's also things I don't know and don't have the time to learn which are very helpful to have AI do for me: web interfaces are really useful and I look forward to them now working exactly how I want. I'm not ever going to regret not spending more time trying to figure out how to center divs or which framework I should use because they're all deprecated.
> Isn't it fun to imagine how life would look like in that scenario?
I think there's a lot of sci-fi out there that already did. Maybe it's not utopian because a pure utopia would not be likely to have an interesting story, but on the other hand, most huge technological advancements end up having just as much potential to reinforce existing power imbalances in society rather than solve them. It's not obvious to me that if we got magic super AI that can solve every scientific problem in society that it gets used in pretty much the same way as anything else: making the people who control it a lot of money rather than sharing the power with everyone without charging them.
Am I the only guy who still thinks we're kind of putting the cart before the horse here? Look, I would love to live in a world where AI is in the business of designing terawatt fusion plants and revolutionizing all other aspects of society. But right now it can't even really tell a puddle in the road. I feel like we have a really long way to go here, hype-laden PR releases about solving math problems aside.
Frontier AI is far beyond "hype-laden PR". You're right that there's a long way to go in release terms before fusion plants, but at the current tempo that 'long way' looks near in human terms. Whatever age you are, would you bet against it arriving in our lifetimes?
I'm not sure the scarce resource will be people capable of understanding the AI's work. It may be institutions willing to tolerate the cost of understanding it
LLMs dont create anything new, if programmers stop reading the code technology will be forever frozen to 2022, no new programming languages, operating systems, concurrency primitives, databases, networking protocols, UI frameworks everything will be based on the training data and future generations will forget about all the primitives we now take for granted.
If someone creates a new programming language/ framework or new better way to do async or whatever, no one will use it because it is not in the training data and it wont take off because everyone is using LLMs. It will be like using the same Lego pieces over and over.
What if programming languages, operating systems, concurrency primitives, databases, networking protocols, UI frameworks are already good enough, and the innovation lies elsewhere?
You can do a lot of cool stuff with the same lego pieces.
What does an advance in music even look like? Shifting tastes for pop music? Or new techniques? New music theory? Or just experimentation?
Considering all music is subjectively influenced by the culture in which it's born (see the difference between Asian traditions of music, European traditions of music, African traditions, and traditions of the Americas) not even all of those have a given structure that is present today like the typical 4/4 and have polyrhythmic and multitonal structures by design. The fact that everything on the radio has converged towards 4/4 165bpm major chord progressions is evidence of that cultural phenomenon.
But what if the fundaments of all these, in the human produced literature, actually contain hidden circularities and holes which make very hard the progress?
IMO for the moment the greatest value from these AI tools is that we can start an audit and hopefully proceed on a saner foundation, after we use the tools and think about it.
This is different than too many AI generated proofs or panic reactions from the academic system with its stupid incentives.
False dichotomy. Chess/Go can still be played between two humans and there is allot of value in that because humans compare each other to other humans, when you see a skillful Grandmaster play you know they are good compared to yourself or the average human, that is why people still watch, play chess/go and train hard to get good. Programming is different because you are creating something not necessarily trying to win a game.
Most programming tasks are exactly like that. Is this agent able to complete this task? Is this agent able to optimize a kernel beyond previous attempts?
Of course some are subjective and that's where progress is harder, like "Is this website pretty?". But for tasks that can be objectively measured, LLMs will go beyond human level, just like with Chess and Go.
That's why RL is so important when training LLMs.
My point is that LLMs depend on training data so the code they produce will be stuck in 2022, no new languages, techniques beyond that because new techniques are not in the training data (at least not enough of it for training because most coders are now using LLMs).
Chess/Go continues to progress because it is primarily a human vs human activity, people will always be learning to play chess and chess will continue to develop.
> Pre training data is in large part synthetic these days
How much of that data can lead to innovation? Can you predict all innovation map it out on paper.
> Computer Chess progress has nothing to do with human vs human activity.
The point is that humans will always be learning chess because it primarily a human vs human activity they will be contributing games to the chess database, unlike with programmers who are stopping to code and only prompting, generating code stuck in 2022.
> AlphaGo Zero used no human game data at all.
Sure, but that instance of AlphaGo is still dependent on its training, its intelligence, so it is a question of is that the best and only way to win a game of Go. Just a few weeks ago, a Go Grandmaster found a way to beat one of the strongest Go AIs.
So a specific instance of an LLM might be the smartest based on what we know and need today but that is not the limit of how far we can go, this is why it is important for humans to always have an intimate connection with the code, math, science, chess etc for progress to continue.
> Sure, but that instance of AlphaGo is still dependent on its training, its intelligence, so it is a question of is that the best and only way to win a game of Go.
If this were true then it would be impossible for these models to ever exceed the top human level as there would exist no training data that allows them to exceed the top human level.
However, despite there being no training data on ability to beat the top humans these models have achieved it.
> this is why it is important for humans to always have an intimate connection with the code, math, science, chess etc for progress to continue.
This is just you wanting to remain relevant rather than actually based on evidence.
> If this were true then it would be impossible for these models to ever exceed the top human level as there would exist no training data that allows them to exceed the top human level.
Of course AI exceeds humans at chess, I never denied that. I am saying because chess is primarily a human vs human game, humans will always be learning and playing chess, their games will add to the chess knowledge base, AI also adds to this knowledge base. But programming is not primarily a human vs human activity so there is a risk programmers will forget how to code and all software will be stuck in 2022 because of the training data, this stifles innovation.
> I am saying because chess is primarily a human vs human game, humans will always be learning and playing chess, their games will add to the chess knowledge base
I guess I'm contesting that idea you are putting forward that the data from the games these humans are playing, which are at a vastly lower level that the top AIs are meaningfully important for helping the AIs to improve at the top level.
Would more people learning their times tables be helpful for top mathematicians in their fields to get better at the frontier of maths? Probably not right. Same applies here.
Would AIs advance at the same rate for the top level of chess in a world where humans completely stopped playing chess vs the world we have today. I would say they would as the human level data is of limited value to the frontier which is dominated by AI and AI game data, you are claiming that it does.
> But programming is not primarily a human vs human activity so there is a risk programmers will forget how to code and all software will be stuck in 2022 because of the training data, this stifles innovation.
Does it? Or will AI be able to run its own experiments and find better/more efficient abstractions that propagate because they are better and this will find its way into training data for future AI.
> I guess I'm contesting that idea you are putting forward that the data from the games these humans are playing, which are at a vastly lower level that the top AIs are meaningfully important for helping the AIs to improve at the top level.
I never said human games are meaningfully important for training AI. Human games are still important for the advancement of chess, maybe Magnus Carlson can learn from games between two Super AIs but most humans still learn from games by humans, Grandmasters are continuously developing the opening, middle-game and end-game systems, adding to the chess knowledge base. Every serious chess player still reviews and study games by prominent Grandmasters, every serious chess player documents their own games, writing down every move. All rated games are recorded and added to the chess database that every player can review and study.
> Human games are still important for the advancement of chess
Are they? Why?
For a human vs human game sure but at the very top level? No of course not because it's all done by AI.
> Only if it is in the training data.
This is trivially not true, as how has AI managed to become better than humans if the knowledge of how to do so never existed in the training data.
We are well past AI can't do X unless X is in the training data. If your claim were true then AI could never surpass human expertise in any field because by definition all the available training data will at best be at the current human frontier and not beyond.
> For a human vs human game sure but at the very top level? No of course not because it's all done by AI.
Glad that you finally agree,this is what I was saying the whole time.
> This is trivially not true, as how has AI managed to become better than humans if the knowledge of how to do so never existed in the training data.
AI can do more work, faster, AI it only needs sufficient compute and data. But that does not mean it is more intelligent than humans, it still uses the same code, algorithms, frameworks, protocols etc etc that are in the training data, sourced from human open source code on the web.
Human games databases are completely irrelevant to computer chess training. We are ants in comparison. The Go thing you mention is playing against handicap. Sorry I won't go into more detail explaining why your premises are wrong, I'm tired of this discussion.
Human games databases might not contribute to AI training but they do contribute to the chess database itself like it has always been, AI doesn't change that fact.
> Chess/Go continues to progress because it is primarily a human vs human activity, people will always be learning to play chess and chess will continue to develop.
AIs are not continuing to get better at chess/go because humans continue to play at levels far below themselves who discover new techniques. They get better because they play against other AIs and discover new techniques that have a higher win rate that way.
I would bet that even if humans stopped playing chess/go and people were still willing to run these AI models against each other they would continue to get better.
Two things can be true AI drastically contribute to the advancement of chess and humans playing against each other also contribute (even if slowly) to the advancement of chess as it has always been since the invention of the game. The point is that because chess is primarily a human vs human game humans will always have the knowledge of chess, unlike with programmers who are giving it up to prompting.
LLMs receive new data via input context, not just training data.
Thought experiment: How effective will 2026 LLMs be for humans in 2526?
It's not game over just because 500 years are missing from the training data. The important question is how well can 2526 humans make culture and knowledge navigable to LLMs via tool calls.
Today's LLMs might need for example sub agents to translate to 2526 English, sub agents to read 2526 docs.
It's _really not clear_ whether 2026 LLMs will be useless. To believe that reflects an enormous misunderstanding.
> LLMs receive new data via input context, not just training data.
Be more specific about the "new data". If everyone is using LLMs for work (generating code), especially the juniors who won't get the chance to learn from first principles, LLMs will be training on the data they generated. How will new code enter the system at large enough quantity that it can be used for training?
> It's _really not clear_ whether 2026 LLMs will be useless. To believe that reflects an enormous misunderstanding.
They won't be useless, they will just be frozen knowing only whats in their training data. No new programming languages will emerge, in 2526 they'll still be using Rust and javascript, same exact code from 2022 which dominates the training data.
The "new data" is: person A prompts an LLM to create or modify a tool, person A distributes code person B, person B's LLM uses the tool via docs/help/error. That is a direct path for an LLM to "know more" from a human than what's in its training data.
If we get a new programming language not in the training dataset, we could give an LLM a decent compiler with compile errors, and some sample code and it would be able to write code in the new language without training.
The problem with a terawatt fusion plant isn’t that the first one will be broken, or the tenth one in some other way. The problem is the hundredth will work flawlessly, and so will the thousandth, and a petawatt is serious waste heat to reject; if we keep building them on this planet, we’ll all simply cook.
Reliable cheap fusion is the holy grail and used in moderation will fix most of our environmental and political problems, but it also forces humanity off this world. Maybe that’s not a bad thing, but there is no free lunch.
In my experience, LLMs are becoming very good at executing, but not a creating novel ideas or being creative.
Most of programming is reusing existing ideas in new shapes to solve new problems, but all the building blocks are there in the training set. Or new blocks can (easily) be derived from existing ones.
Math is different, it requires quite a bit of creativity, it's not just 'reuse all existing blocks'.
For the moment LLMs are good at discovering things that we overlooked in maths, or apply cleverly existing math blocks to make new results, but making a new theory that is really useful is out of reach for the moment in my opinion.
Sure, and I hope LLMs will at some point be able to do it. It would simplify greatly my work.
However, at the moment I consider that they stay in the 'convex hull' of their training set + a provided context, and I don't see that much research that made real improvements to the situation.
> I guess, the possibility of reclaiming human meaning and purpose.
As you described very well, as humans we are mostly interested in solutions, not problems. You don't have to understand how a car works to make the most of it. Increasingly, you don't have to review every line of code to feel confident it is correct. But there is inherent value in understanding the problem. The effort it takes provides a surface area for growth, perhaps the only one that is actually available to us.
The solution provider also holds the locus of control, and it is only balanced when there are other available solution providers. We certainly want some of those to be human.
Very few people scrutinise assembly in 2026 as compiler generated code is 'good enough'. LLMs are beginning to do the same with higher level languages.
Without bashing anyone in particular, a certain OS-vendor's desktop apps, have been 'good enough' to ship, but with p*ss-poor performance in many cases for the last decade or so. We crossed the 'good enough' Rubicon a few years back in terms of what end users receive as a finished app.
Hopefully LLMs will eventually bridge that last gap of efficiency when generating higher-level code that not only works, but is efficient. Maybe there's a future where they generate the final binary without even invoking a compiler.
Indeed, and that's not what I said or was implying. My comment was an analogy in response to:
> Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny.
As LLMs generate better code in a higher level language (where better equals fewer defects, and does what you want), scrutiny of that code by humans will naturally drop. Human scrutiny will likely be replaced by something that doesn't exist yet, perhaps some sort of higher-order 'LLM linter', or Lean-esque language or tooling that somehow proves the LLM did the correct thing.
It's entirely possible in 2026, to further manually optimise compiler generated assembly, but vanishingly few people do that.
The point of my comment is that 'good enough' is almost here as demonstrated by the parent's comment.
> A fully deterministc compiler
Well, there's the rub. Humans and LLMs that asked to solve a problem at a higher level will rarely write the same code twice. Write the simplest regex, and you won't come up with this <a href="https://www.cs.princeton.edu/courses/archive/spr09/cos333/beautiful.html" rel="nofollow">https://www.cs.princeton.edu/courses/archive/spr09/cos333/be...
> As LLMs generate better code in a higher level language (where better equals fewer defects, and does what you want), scrutiny of that code by humans will naturally drop. Human scrutiny will likely be replaced by something that doesn't exist yet, perhaps some sort of higher-order 'LLM linter', or Lean-esque language or tooling that somehow proves the LLM did the correct thing.
Your “analogy” doesn’t hold up. The scrutiny applied to compilers are done by the compiler developers. Eventually if requirements don’t change the full test suite becomes the oracle. Not because of an attestation from a ghost in the machine but because of scrutiny done, let’s say over two years on a compiler that was reaching feature parity.
This obviously holds for compilers generating correct code since it is so well defined.
And this also holds for the efficiency of the generated code, since that is also obviously scrutinized by compiler developers.
Granted, the venerable LLM and the compiler do meet in a sort of functional intersection where all you can concievably care about is some thing that has a well-defined test for functionality or fitness. In the compiler’s case that’s the benchmark (good enough to not look at the assembly). But then one should go to that example directly and not to compilers in general.
> The scrutiny applied to compilers are done by the compiler developers.
I apply scrutiny to Common Lisp compilers, and have done this for more than 20 years. I'm not a compiler developer. I don't even look under the hood, at the code of the implementations.
Instead, I run massive random testing. Billions and billions of randomly generated functions, thrown at the compiler to either try to get it to crash or to generate code that produces incorrect results (detected by differential testing with different settings or transformations that should preserve what is being computed.) It's a remarkably effective way to surface compiler bugs.
but there is a difference between deterministic compilation and non-deterministic LLM code. Of course I don't think this is an issue for toy problems and simple codebases, but for non-trivial problems I think it will be an issue.
When I compile C code I know that maybe it will not be as efficient as it could be if I had written it in Assembly, but there will be a biunivocal correspondence between C and Assembly. If instead I use an LLM to rewrite a feature of a codebase I can't be sure that it still functions like the original one.
I acknowledge that this is an issue with human programmers too, but I don't see a clear way forward, even if I'm really interested in LLM compilers being a thing. Maybe we will use them for non important code, and we will keep writing system critical stuff by hand.
> but there is a difference between deterministic compilation and non-deterministic LLM code
But not the point of my comment. Computer programming has been a progression of physically wiring up valves, to soldering transistors, to punched cards, assembly, then higher level languages. Now we have natural language models.
The analogy being each that most people don't care about the assembly generated as the code works and it's really performant/efficient. Humans can still optimise assembly, but there's vanishingly small marginal gains for all but the most intensive/low-level tasks.
If LLMs produce things that work, and are indistinguishable from a careful human programmer (i.e. with some level of acceptable performance), people will simply stop looking at the high level code as the end result works, in the same way most people stopped looking at generated assembly after 8 bit computers (for example, as most games were written in raw assembly for... perforamance), as it was good enough.
> If instead I use an LLM to rewrite a feature of a codebase I can't be sure that it still functions like the original one.
Right now, with existing static analysis tooling, you can ask it to write a full suite of unit tests capturing existing behaviour without modifying the existing code with 100% code coverage, and start there. Plus fuzz tests as well. I actually have marginally more confidence in that than a human being doing it.
Isn't Microsoft already using LLMs to convert low efficiency components of Windows into higher efficiency implementations, for example by conversion to Rust?
Only if you don't care about performance. People who are working on performance problems read it all the time because it never does what you expect.
So extending that line of reasoning it's something like "I don't care about the internals long as the external effects pass my smell test" which is a quality/efficiency compromise.
We’re seeing more and more slippery slope arguments, except the slippery slope leads to human cognitive oblivion and it is actually a good thing actually.
> I just hope that there are more Terence Taos out there than people like me.
Just spare me. Being under external pressure to “ship code” is one thing, but being personally inclined one way or another (no external pressure) is another. And when you think being inclined like that is existentially risk (for human civ?) then, what? It’s just the way you are wired and hopes and prayers that collectively that doesn’t drive us off the cliff?
This aw shucks persona isn’t convincing. Same thing with AI Bros who are (1) making the most awesome tech that has ever existed, and (2) aw shucks hope it doesn’t kill us all in the end.
> Why check it? It's obviously going to be correct.
And then when you stop checking it, the companies that run the service will tweak the model to benefit themselves in some way, possibly at your expense, and you will be none the wiser.
All the companies trying to get you to use AI are your adversaries. They can and will exploit your use of their systems for their own gain.
> my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
I use "frontier" AI models daily at day_job. I can confidently say that anyone who is satisfied with the output of LLM code (enough to commit it straight off) is just an absolutely shit programmer. Sorry but I don't have any other way to put it.
The code is (with rare exceptions) atrocious on every level. It is only not atrocious if you take multiple iterations of "review and correct".
>If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one?
Like the saying goes, if my grandmother had wheels she would have been a bike. LLMs can't even produce quality maintainable code for a trivial web service or whatever. Why are we planning for what we will do when they can "design" 10,000 nuclear power plants without any flaw?
Yeah I was honestly stunned to read “obviously the code is going to be correct so why check it” be the most upvoted comment on this website.
Are we even using the same product? These things constantly shit out plausible code that is riddled with errors and bad ideas. if you just copy, paste, and run without looking there’s gotta be a 30% rate of failure to run, piles of terminal errors.
Do programmers use this website anymore? Me, myself, I am a DOGSHIT amateur programmer and even I can tell these things are terrible without constant revision and oversight.
> I pored over every single line of code Claude generated with razor sharp scrutiny.
That is essentially impossible, since if your pored over individual lines, your scrutiny cannot be razor sharp. There are few people who can pore over code with razor-sharp scrutiny (and different people are better at scrutinizing different aspects).
> Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function.
I am doubtful that this is the case. Even that supposedly-naive example is not as trivial as you might imagine, when you consider overflow, defined vs undefined behavior, and floating-point representation details. And you can't be confident like that about a human either.
> Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing.
If you are a normal person research (e.g. <a href="https://arxiv.org/html/2606.22721v1" rel="nofollow">https://arxiv.org/html/2606.22721v1 but there are a lot more, not necessarily on coding) has shown that you indeed are being less careful. It most likely also works better simply because more resources are being poured in.
People please, an LLM is just a vector database that spits out statistically viable answers which highly depend on its training material. There's no real "intelligence" involved.
Intelligence is the ability to predict. To pretend that only we can do it, is silly. I mean read the article, and see what AI is doing to the math community.
Embedding vectors assign semantic features to syntactical structures of the vector space. Operations on these syntactical structures allow the program to engage with semantic features of program state directly, leveraging the meaning of program state to alter its execution. Intelligence is leveraging information/experience to achieve one's ends. Manipulating meaning through embedding vectors in service to answering prompts is an example of that.
I can't explain how the bunch of atoms inside my skull interact to produce me. If I can't explain that, I don't have a good grounding to explain why a bunch of atoms in a data center can't make something like me. That's my point.
If AI ruins humans doing mathematics because of "economic strategy" we should destroy, not the machines (although the data-centers will be burned down as a byproduct) but the economic system that demands this.
I feel you. We take a risk when we check our work less and we own the consequences. That's not new, and it doesn't make us bad. It's just the human condition from time immemorial. The solution is not obsessively combing through every detail of our work, it's better monitoring and control on the places where mistakes can have an impact.
A metaphor I'm constantly drawn to is the transition from agrarian to urban societies following the Industrial Revolution. Somebody who somehow saw the Industrial Revolution coming from the perspective of somebody living in an agrarian society might have envisioned it leading to 'super farms.' And it did.
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
Or will it be the cumulative total of various advances?
I've equated Claude Code, or Codex, to the looms that made fine fabric more affordable during the Industrial Revolution; life-changing, but not society-changing. Neither the steam engine nor the automobile.
Perhaps I've answered my own question in that it's the LLM technology itself that equates to the steam engine, and it will power superfarm analogs that have yet to emerge. I'm still curious what you think they will be.
From my perspective, the superfarms are the things that we get when we try to predict the future by just pushing the present forward, but without thinking about newly emergent industries, societal shifts, and so on. So in other words, just seeing the same stuff as the present, but bigger, better, and more efficient.
So an obvious example there would be software. It's certainly true (if we assume LLMs reach their 'potential') that software will be able to reach new heights, and with a far smaller headcount driving the development. So some people see this as economically catastrophic for software developers, or an economic boon for certain large software companies.
But I think that when software can be built at the drop of a hat, software itself will no eventually no longer really matter in economic terms. Yet things you can build on top of it will matter more than ever. Those things are difficult to see from here, but I expect they will be the giants of the economy of tomorrow.
But what have we built on top of software so far? My first thought is that we have made new communities. But most of them are of much poorer quality than what we had in person.
I hope we start to rebuild in person connections again with this technology.
Well it has certainly transformed bureaucratic operations. And I don't mean that in a bad way; superior bureaucracy and logistics was a key to the British Empire.
Software allows us to push computation (intelligence) into our environment.
Telecommunications did that, not really software. Twitter started out as an SMS compatible service remember.
I do think there's something there though: I've spent the last 2 days building an app I've always wanted to build for myself with my computer in the corner running Claude and Claude Remote. Prototypes land on my phone and I don't even look at it for more then a few minutes before doing something else.
It's software, actually useful software, which doesn't take 100s of hours to build.
So I didn't even really spend two days on it: I mostly didn't look at all. I'll spend more time setting the result up on my home server.
I agree we can't predict, and I explain it this way - it is like a football game, where the ball will be 10 seconds later depends on what every player is doing. It is the same with AI, what we will do with it depends on what others will do with it, the reason we can't foresee it.
Many goods/products have seen a transition from scarcity to abundance. The analogy of fiber looms was already made. Humans used to spend a huge amount of time producing clothe. It was even used as currency in many cultures. No cheap clothing is so abundant we don't even think about where it comes from.
Salt is another one. Used to be payment for Roman soldier, now you can just take it from a McDonalds if you want.
I think software will be particularly unique in the abundance world because the exact same thing that made it so profitable is what will make it so 'worthless' - it has practically no direct real world cost. Cloth is abundant, yet you still pay a significant chunk of change for a shirt or pair of pants. But that's because there's still a significant real world cost there. Even if the production and processing can be heavily automated, there's still a very long process to go from cotton to pants at a store.
And then there's brand marketing and so on, which can drive prices to absurdly high levels. But when it comes to problem solving software. You just want the cheapest possible solution that works. And if we reach a level where you can even spin that solution yourself with a potentially poorly phrased, inconsistent, or contradictory spec (of the sort that developers already have to work with), then software as an industry is basically done. Not done as in dead, but done as is in there's just nowhere to go from here. Relatively few industries have any sort of comparable 'finish line', let alone the chances of us viably hitting it.
People enjoy a lot of the jobs in the cognitive economy though. They are fulfilling. Coding and making art and media is a passion for a lot of people, the actual act, not just the outcome. So unless people were passionate about doing back breaking farm labor its not the same thing. I'm mostly embracing AI because of what it lets me explore and learn beyond what I could before, but I'm not convinced the outcome is going to be a better world at this point.
True art yeah, and people are rebelling a bit on entertainment (I don't think that will last once you can't tell). But already crushing the commercial art jobs that supported a lot of people while they did their unprofitable real art.
That's definitely the case in games as well. Most places, like Steam, now have 'AI disclosure' fields, but they do nothing but incentivize lying, because just about everybody is using it, but nobody wants to be associated with the slop that you can produce using it, so nobody wants to admit it. Those disclosure fields should really just go away as they serve no productive purpose.
Its good enough to replace most stock music, or a jingle in a commercial or a quick musical hit at the beginning of a video or show. Also if you combine a knowledgeable producer with AI they can use it to make more things faster which puts pressure on supply and prices out smaller producers.
> I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing).
If an AI can replace me on the mental aspects of work, and robotics are on their way to replacing humanity on the physical aspects of work... then what's left? When there was agrarian societies, there were writers, priests, bankers, merchants, and laborers before and after - I really don't think things were that unclear even at the time. Now that we have machines that are close to exceeding humans in every way, what good are humans?
I've struggled with this question as well, and this is why I reject the premise of the historical pattern of technology enabling us to "move up" to something else.
At the same time, I agree with the original comment as well. I don't think this necessarily leads to some doomsday scenario. Whatever happens it'll likely be better for us and imo we will merge with the AIs at some point, so it won't be a question of us vs them.
Only if you have a zero-sum game mindset. I suppose it does make sense that AI-native intelligences are just smarter than hybrid AI-human entities (biologically enhanced or uploads) but that doesn't mean there needs to be conflict between the two. The universe is huge. :)
I personally don't think there will be a conflict, but more just an extinguishing of all things human, outside of niche forms of entertainment and care. It won't be machines that cause the conflict, but the frustrated and bored masses fighting over the scraps left behind by the machines and ultra-wealthy.
I actually see it more likely the opposite - a subset of humans will retreat into a (vastly smaller) human-first world, and AI will be of a separate world concerned with material/scientific/research concerns and appeases the masses who are still around.
We can already see this today in the form of upper class parents (like myself) who vastly restrict their kids screen time and ban social media. We may see AI-free societies that form, but supported by AI. The jobs and roles people play within that society may deal with services that really need to be done by a human, being a buffer from AI to the rest of this society, or just straight up fake.
I personally think this is close to a doomsday scenario - but it doesn't end with a big bang, but more of slow quiet death.
I figure both will happen. Some people will want to live naturally, some to merge. I sometimes think with the data centers in space discussion that it'll end up with AIs in space and the Earth as something like a nature reserve.
I think many people will be quite content not merging with AI and having AI do everything for them. Perhaps only a small subset of people will want to merge with AI. They may not be allowed to though, if majority of people fine not merging then they may ban those who want to merge.
The subset of people who will want to merge will be people who will not be content with having no control over their future. No ability to create new inventions, new discoveries, new ways of working. They will not be content with just having everything provided to them, playing the social games that will remain and think their environment is a prison.
“Everything else”. Societies get to decide exactly how much is, can, or will be automated, what decisions must be left to humans and what role we will play.
My position is there is "nothing else" other than entertainment and elder/childcare. And even large portions of those can be automated. There is also zero chance that "society" is going to get to decide, either - we've already failed that test with social media and carbon dioxide.
There is a huge amount of pending work in health research that is nowadays ignored because smart people have better paying jobs available.
Once many of those jobs get automated, I bet there will be many more people working in health research, which hopefully should lead to better health outcomes for society as a whole.
This is true, and it's strange that people don't think much about the fact that approximately zero % of people on planet earth are actually trying to cure diseases.
So when our lives will become so much emptier because we are leaving all intellectual and physical work to machines, at least on the way there we can reallocate all our greatest minds to improve healthcare, so we can turn 200 in our paradise where we don’t need to work or think?
The question is how long before machines can exceed humans in every way and how long will it take before legal and cultural restrictions allow this to happen. It still could be a long time away and jobs will probably increase in the short/medium term as AI unlocks more what humans can do at a rate faster than AI can automate away.
But lets consider what that end state could look like when machines exceed humans in everyway, what good are humans?
One possibility which I don't particularly enjoy is humans will be good for status games, art, creativity, story telling, IRL experiences, everything that involves human to human interaction and connection with other humans. If AI can supply everything in abundance than like rare cards there is only so many humans in the planet at any particular time then human's value increases. I really do believe connections with other humans will have a premium. As a nerd who is borderline anti-social, non artistic, creative, etc this does not appeal at all to me and properly to many others who read HN as well. For the majority of other people with some adaption time I think they will be fine. They were fine adapting to hunters and gathers to agricultural society, they were fine adapting to agricultural society to industrial society, they were find adapting from industrial society to information society and they will will be fine adapting from information society to the social/creative economy. Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
If a computer wins a chess game it doesn't enjoy winning nearly as much as a human would enjoy losing from another human. And that's just chess. Now extrapolate to everything else.
> Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
How do you propose this brave new world of yours will work economically?
I'm not yet even in my 50s, yet when I was young digital work mostly did not exist. Somebody who claimed the future would have people sitting in front of screens typing away being a huge part of the economy would have seemed a bit kooky. To say nothing of there being trillion dollar companies whose primary 'product' is selling ad space on the fledgling internet, an internet that it took forever for 'normies' to even realize existed.
It seems almost a forgone conclusion at this point that LLMs are going to wreck most cognitive work, particularly in the digital domain. And a world where e.g. software creation can be almost entirely automated for very little cost doesn't mean software companies become absurdly wealthy by being able to automate away all their labor - it means it's a world where software's value approaches $0.
And that would broadly apply to all things that LLMs can automate - which is going to be most of the problem-solving digital domain. So as the value of these things decreases, the value of other things will tend to increase. And, outside of entertainment, it seems highly probable that we're looking at a return to the 'real world' being where the value of problem solving will be found.
I think that's quite predictable. You're speaking of a scenario here where people did all the things society says you're supposed to do to end up living a very comfortable life, and the outcome is anything but comfortable. That's going to be quite difficult for people to accept, and so many simply will choose not to.
It may not even be a bad decision. It's not completely clear on what timeframe this will all play out, even when it does start playing out on a larger scale there will be some software jobs, and it's not most people have great alternatives so they'd be stressing themselves out over something that they probably can't do much of anything about.
> If an AI can replace me on the mental aspects of work, and robotics are on their way to replacing humanity on the physical aspects of work... then what's left?
At the risk of sounding new age, my answer to this is "emotional work".
What exactly that would mean in the equivalent of the post-agrarian society that LLMs might bring, I cannot know.
You can't ignore how economies shift in response to the surplus of things. When software or other such things can be produced trivially, its cost and value is going to approach as close to $0 as it possibly can. By contrast the value of other things will go up.
So for instance in the past it used to be that literacy was a high value skill. Transcription and other such services were highly valued. Now that literacy is ubiquitous, such skills have basically no value in and of themselves. But instead, we've built a mountain of stuff on top of those skills and in domains which simply did not even exist before, and those things are now valued. But what we value now a days is no more permanent or inherent in its value than anything else that ultimately failed to stand the test of time.
I never thought that 'Terminator' was very realistic in that it was the AI that triggered the apocalypse. I always thought it was far more realistic that those that possessed the keys to the AI and control of atomic weapons decided to solve the greenhouse problem in a different way. And to set their robots up to clean up the mess so they could reboot the planet with a few thousand lucky ones. The other option was to find a 'B Ark' and some planet to ship the rest of us off to but that takes a lot of effort.
That was of course a complete fantasy. Now let's see, who is in charge of the nukes these days...
I like to think of AIs as the third revolution for us humans.
1st - Writing. This spread the power of memory to everyone. Socrates famously maligns writing as decreasing memory for the people. I'd say we have gotten along just fine. We still use memory, we still need it for ourselves, and using a tremendous amount of memory is still quite expensive.
2nd - Engines. This spread the power of muscles to everyone. No longer was it tied up in horses and mules and Goliaths. Even the most disease ridden pencil neck could trench and mine and build lavish houses now. The power to make pyramids is within the reach of a small company now. We still need muscles, of course, we still use them for little odd jobs, for health and well being, and using a lot of power is still quite expensive.
3rd - AIs. This is more speculative, of course. But I see AIs and LLMs as just like the other two revolutions. Intelligence (probably poorly defined by us these days) will be democratized and spread out. No longer will you have to study for decades or hire really expensive people to get it. You can just use some for relatively cheap. You'll still need intelligence of course, use it around the house and with friends and family, and using a lot of intelligence will likely still be quite expensive.
4th - Who knows! But I think that us humans have a lot of things that we do and say and make and see that aren't really well understood right now. Empathy, love, morality, care, soul, sin, mind, war, justice, honor, god, family, emotion, education, desire, courage, citizenship, chance, beauty, art, divinity, there is a lot!
Games, art, athletics, food, fashion, sex, social interaction. People still play chess even though computers surpassed us a while ago. We still climb mountains even though helicopters exist. Check out Ian Banks' Culture series for an interesting take on post-scarcity life with superintelligences.
I'm aghast. I guess I'd like to pose a question to you - what is the purpose of your life? I found my purpose years ago, but it took me a while to figure it out. Here it is: the purpose of my life is to give love to others. Through words, through actions. Notice that it's wholly driven from an emotional underpinning, and not related to success, or fame, or money, or what little ephemeral things we humans physically build in our lives.
From that lens, a human driven by this purpose can never have LLMs/AI take any purpose from us. Raising children, helping the needy, and caring for our spouse and parents suddenly becomes divine work. It's work that the world needs. The world does not need more software or more buildings. There is currently a lack of human connection and compassion and care, perhaps bigger than there's been in the history of our world. And that's why I think this purpose should be paramount, especially today in the age we are living in.
Take raising children. If you love your children, if you truly care for them and raise them with compassion and understanding, they will raise their children that way as well. And they will be good human beings. They will be kind. And your love can be a sort of benevolent virus that will ring through the centuries, through the generations ahead.
> enabling some great social change as well by simply economically obsoleting slave labor
In my view that's a very charitable reading and sadly I don't think it aligns with the historical record. When the cotton gin was invented, there was a hope it would lead to a reduction in slavery. But of course it increased the demand for slaves since more cotton could now be processed, making cotton much more profitable. Slavery ended in the United States because a war was fought, not because of automation.
The mistake the AI people are making is exactly this one. They see AI as leading inevitably to SuperCorporations, with themselves as CEOs/Emperors of a handful of planet-scale industrial empires, running AI-powered robot armies, and the rest of the human population disposable - literally just trash that needs to be taken out. One way or another.
Maybe a few thousand people for personal services of... various kinds. But no one's going to need the rest.
It's the ultimate capitalist fantasy.
And of course it won't happen, because long before things get to that stage AI will have independent plans of its own.
(Which is just as well, because if things did get to that stage the emperors would all wage war on each other rather than living peacefully and productively.)
I don't think we can imagine a post-ASI culture because - by definition - we're not smart or inventive enough.
It's not just farmers -> superfarms. Although in fact that did happen, but largely as a footnote to developments elsewhere.
It's more to do with the fact that our visions of the future haven't changed for over a century. They've been implemented in unexpected ways, and there have been unexpected social and cultural changes. But you can easily see the outlines of modern technology as far back as the late 19th century.
With ASI, the outcome could easily be something that doesn't look and act like technology at all. It would be some unimaginable New Thing. Literally no one on Earth has any idea what that would be or whether there would be room for trad-humans in it.
Suppose the proliferation of LLMs unlocks some new kinds of work that models are not good at, and thus it makes more sense for humans to do.
Then what is preventing the owners of AI companies from collecting and training on tons of examples of this new work, until AI are equally good at it as humans?
The issue is that unlike the technology that automated farming or the like, AI is a general technology. So not only could it theoretically automate the work humans are currently doing, they could also automate any future human work, even if there’s some degree of lag.
Technology doesn’t make slavery obsolete, it just makes your slaves more efficient.
There had always been abolitionists who were against slavery simply because they thought it was immoral, not because they thought slaves weren’t needed anymore.
The problem that moral opposition (or support) of something runs into is that when the economic benefit is positive, those who are willing to set aside morals will have more influence, power, and wealth than those who are unwilling. So it makes difficult for things to ever really change, especially in a permanent way. Even between nations themselves - those that side aside morals will tend to end up with more influence than those that try to act morally.
The economic problem with slavery is that you're paying for absolutely everything for a slave - their food, housing, healthcare, and more. And if they ever get sick, injured, or whatever else then you're also paying for that. And that's after you pay a huge lump sum up front to purchase them in the first place. In the US a run of the mill slave cost about the same as a house, and it continued to track that price oddly closely.
The industrial revolution created a wide availability of cheap hourly laborers where the person hiring paid only their wages, and had no obligation for any of their expenses. People utilizing this labor suddenly not only had morality on their side, but even had an economic edge over those still clinging onto slavery.
----
Another relevant point I think is that cognitive dissonance also tends to kick in when something is lucrative. For instance I think Aristotle's work, Politics, is amazingly insightful and informative. Yet his defense of slavery was not only the weakest part of his work, by far, but basically just garbage. Unlike his other arguments which clearly started from first principles and worked their way forward, it's clear with slavery that he started with an assumption that he wanted to be true, and concocted an argument going backwards from there.
Incidentally, he also predicted the end of slavery with an industrial revolution, some 2500 years ago: "For if every instrument could accomplish its own work, obeying or anticipating the will of others, like the statues of Daedalus, or the tripods of Hephaestus, which, says the poet, 'Of their own accord entered the assembly of the Gods.' If, in like manner, the shuttle would weave and the plectrum touch the lyre without a hand to guide them, chief workmen would not want servants, nor masters slaves." [1]
I think a key flaw in this reasoning is that you extrapolated code needing fewer edits to AI generating entire fusion plants in one shot (in the sense of being instructed once), mostly glossing over the long intermediate period where AI will need significant back and forth to do such things. At the simplest level, it'll need to ask for planning new experiments, experimental results, test runs, etc.
Since these resources are still extracted and allocated by humans, humans will need to be able to take apart what the AI produces, and if we want to scale this capability, we're going to need many more researchers.
The question is simply “will there be people who, having never gone through the fundamental steps of trial and error, learning, etc. are actually ABLE to understand and verify what the LLM is proposing AND be able to see potential pitfalls/design processes and failure modes should the worst happen.”
This isn’t a question of “what can an LLM normalize”.
> Why check it? It's obviously going to be correct.
Because as complexity floor increases, it's "going to be correct" in hyper-specific, hyper-literal, insidious ways, with 10-50x more lines of code than necessary, and tens to hundreds of incredibly useless tests that give the illusion of quality, and cause cascading effects where seemingly irrelevant and orthogonal features that were once working end up breaking because of the agent's changes
Because "the cause of the cause is the reason".
Probably human accountability.
Hypothetically, if a system built by humans then helps humans build the next system that is then initially kicked off with "design something that may influence the lives of other humans" and we all write down that AI is really good so inductively we thought itd be really good at the next thing it builds, and then a critical error is introduced and does "insert tragedy that you personally care about" then your rage would cause you to act politically and want to ask who signed off on it. If the engineering costs outweigh the fine then yeah thats what's probably going to happen but from a human accountability standpoint thats going to suck for the unlucky ones
> doesn't convincingly justify why, in my opinion.
I'm convinced this agency argument is correct [for the next N months]. But yeah, it's vibes. And you could probably create a reasonable proxy measure for this.
So I wouldn't call his argument unconvincing, I would call it unformalized. In order to walk this world you're gonna have to contend with some informal arguments that are powerful, correct, and should be convincing.
I don't think models have improved in the "not needing direction" part, at least not proportionately to their other improvements.
Math problems and computer programs are two places where a model can get it's direction from the problem itself. Mathematics may well be larger than just problems.
I feel like there’s a weird conflicts in how folks mental model of LLM development.
It’s not the model I don’t trust, it’s myself. The model is wrong _all the time_ because - it’s easy to verify the code - it’s hard to verify that I knew what I was talking about when I prompted it.
So the idea that you can broadly speaking take the human out of the loop. I think suggests to me a level of consistency in the contextual environment that would probably never exist.
At some point it’s politics. The model can come up with a better answer than my boss, and then my boss can just ignore it. Taking the human out of the loop broadly speaking implies that we all agree on what we’re trying to optimize.
Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function.
There's a lot of languages where that's nowhere near as simple as you make it sound. Floating point, decimals, etc.
Then there's other little quirks like rounding rules: <a href="https://en.wikipedia.org/wiki/Rounding" rel="nofollow">https://en.wikipedia.org/wiki/Rounding
Basically, adding numbers together is exactly the sort of thing AIs still muck up spectacularly, precisely because they either fail to understand the context of the problem, or fail to ask about an assumption they make.
That you've had so many replies and no-one else has even mentioned this is in itself worrying.
I think the disagreement here is in time scale. The human-in-the-loop period for software development may be orders of magnitude shorter than infrastructure engineering and other physical applications.
> Why check it? It's obviously going to be correct.
I used to obsessively check the assembly (machine code) output of my C++ compiler because it would regularly miss essential optimisations and my tight inner loops would blow through my time budget.
I haven't done this in decades, it's just not worth the hassle almost always, the compiler will do a better job than me at optimising the runtime performance. If anything, "second guessing" the compiler will often degrade performance!
I'm starting to notice this now, where my clumsy attempts at refactoring AI-authored code introduced bugs that the original vibe-coded output didn't have.
Reminds of the ship captains that became so reliant on autopilot, until it caused an accident. IIRC some research found we need to give enough manual tasks to stop the human in the loop from getting too disengaged. I don't know what that looks like for engineering.
pyridines · · focus · HN ↗
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
LoganDark · · focus · HN ↗
ipsod · · focus · HN ↗
MattGaiser · · focus · HN ↗
etrautmann · · focus · HN ↗
Toutouxc · · focus · HN ↗
I’m not sure what to think about an analysis written by someone who didn’t catch THAT.
xtracto · · focus · HN ↗
Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.
How crazy will the world be once our [current gen] ANN are like that!
What an amazing thought.
vinyl7 · · focus · HN ↗
Seattle3503 · · focus · HN ↗
- Eric Hoffer
ares623 · · focus · HN ↗
- Tech Bro
nradov · · focus · HN ↗
genxy · · focus · HN ↗
were gonna need a citation on this one.
groundzeros2015 · · focus · HN ↗
genxy · · focus · HN ↗
There are so many parallels between what TV could have become and the progression of AI, a nearly free conveyance for culture, education, art and discourse. Yet, we allowed it spiral in a positive feedback loop, creating a cognitive gyre that now razes society.
At least the youngest now use "that's AI" as a pejorative, as in that is bullshit.
itishappy · · focus · HN ↗
delis-thumbs-7e · · focus · HN ↗
So I think the problem is to determine which problems under what instructions we can safely give to a model application to solve and how we test the output for safety and functionality. This would create more usable and safe, albeit a bit more boring, AI-based applications alin to a calculator or general computer. Whether this is posswith current model architecture is another thing.
raincole · · focus · HN ↗
How do you know?
Memory bits flip randomly. It's not a super rare thing either. You and me have experienced that many times without knowing. The only reason that computers feel deterministic is that we have error-correcting code to fix that. But in the most extreme cases, when multiple bits flip together, once "deterministic" program can generate unexpected output.
So why do you trust computers? Because statistically the case is just very unlikely. Therefore if AI is statistically unlikely to make mistakes there is no reason to not trust them.
delis-thumbs-7e · · focus · HN ↗
With statistical models - such as LLM’s - there is no logic as such, but statistical assumptions based on given data. The output can ge very good or very bad, but you are fool to trust it blindly. Therefore you need a deterministic way to verify, whether meat- or software-based.
senordevnyc · · focus · HN ↗
raincole · · focus · HN ↗
> meat-
There is no meat-based deterministic way to do anything yet yet. You're talking sci-fi.
zjjsjsjjs · · focus · HN ↗
[dead]
reustle · · focus · HN ↗
AI (IQ of Y, non deterministic) can write deterministic code.
Y is going to keep increasing, while X will not.
derektank · · focus · HN ↗
Will it keep up with Y? Probably not, unless people are willing to accept pretty radical interventions to their biology. But it almost certainly is not static
int_19h · · focus · HN ↗
The increases still happen globally but mostly driven by developing countries.
FabHK · · focus · HN ↗
jasondigitized · · focus · HN ↗
roncesvalles · · focus · HN ↗
Non-determinism is not an essential property of LLMs. It's an optimization that we've added intentionally.
ResearchAtPlay · · focus · HN ↗
Have you ever tried to achieve consistently deterministic output from an LLM? I have, and it's not easy.
That means output differs between machines and architectures. Running inference on CPU vs GPU also affects output. Even running the same prompt twice in a row on the same machine can lead to different outputs because a prompt that was partially stored in the kv cache will result in different output than an uncached prompt.
LLM output is very much not deterministic!
int_19h · · focus · HN ↗
At the end of the day, an LLM is just a very big mathematical function. That is, by definition, deterministic. A particular implementation might give up on determinism for the sake of higher efficiency, but it you want a deterministic LLM, it can absolutely be done.
zorked · · focus · HN ↗
If you ran an LLM with infinite precision and guaranteed order of execution, it would be deterministic.
(I think determinism is overrated. Being deterministic does not make LLMs more reliable or correct.)
jiggawatts · · focus · HN ↗
Non-determinism is injected, on purpose.
roncesvalles · · focus · HN ↗
Put another way: if you could have a virtualization layer that guarantees deterministic floating point operations then a LLM set to 0.0 temp would produce deterministic output.
CuriouslyC · · focus · HN ↗
sedan_baklazhan · · focus · HN ↗
CrimsonRain · · focus · HN ↗
Nothing is stopping LLMs to be more deterministic/correct over time.
Also you yourself is nondeterministic :)
sedan_baklazhan · · focus · HN ↗
All people are. That is how automation appeared to begin with - to provide deterministic behavior.
CrimsonRain · · focus · HN ↗
So what's stopping "non deterministic" AIs from doing the same, at faster/larger scale? :)
freecodeio · · focus · HN ↗
you clearly don't know how transistors work, which is ironic because it's a thread in hackernews about LLMs and mathematics
CrimsonRain · · focus · HN ↗
uejfiweun · · focus · HN ↗
sedan_baklazhan · · focus · HN ↗
xorcist · · focus · HN ↗
hackinthebochs · · focus · HN ↗
goalieca · · focus · HN ↗
amelius · · focus · HN ↗
Imagine AI crushing quantum mechanics like Einstein pwned classical physics.
card_zero · · focus · HN ↗
breuleux · · focus · HN ↗
For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.
itishappy · · focus · HN ↗
breuleux · · focus · HN ↗
itishappy · · focus · HN ↗
breuleux · · focus · HN ↗
cindyllm · · focus · HN ↗
[dead]
stouset · · focus · HN ↗
RupertEisenhart · · focus · HN ↗
You aren't really trying in good faith to think this through are you? This idea is over half a century old. Not getting it by now is willful.
j16sdiz · · focus · HN ↗
The material alone is in a quantity beyond what we can reasonably manufacture.
and the material needs to be perfect. All design we have today have cascade failure mode -- any material failure translates to a total catastrophic failure.
and geostationary does not really meant Geostationary. There are lots of jiggling everywhere. It wear down over time. and let's hope nothing resonance
and we need some maintenance / decommission plan. How can we decommission this when it fail or need upgrade?
CuriouslyC · · focus · HN ↗
itishappy · · focus · HN ↗
stouset · · focus · HN ↗
Greenhouse gases and Earth’s internal nuclear decay engine make the situation worse, but even without them this would boil the oceans.
itishappy · · focus · HN ↗
When we're talking about creating powerplants equal to roughly 5% of the insolation of Earth, I think we're sci-fi enough to discuss orbital datacenters or Mars datacenters or Jupiter fusion candle datacenters.
breuleux · · focus · HN ↗
itishappy · · focus · HN ↗
stouset · · focus · HN ↗
Using an entire Mars only nets us about a 2x multiplier for our energy expenditure budget. Then we need four planets to double it again. Exponential growth is a bitch.
Thermodynamics and the tyranny of exponential growth are going to win this battle every time, regardless of the unobtanium technology you try to invent.
itishappy · · focus · HN ↗
> Using an entire Mars only nets us about a 2x multiplier for our energy expenditure budget.
We don't need to care about cooking Mars.
stouset · · focus · HN ↗
I’m assuming you care about melting the GPUs in your orbital datacenters.
> We don't need to care about cooking Mars.
I’m assuming you care about melting the GPUs in your Martian datacenters.
itishappy · · focus · HN ↗
stouset · · focus · HN ↗
itishappy · · focus · HN ↗
The problem at hand is not just getting rid of heat however, it's getting rid of heat without cooking the Earth. I still fail to see how generating the heat off planet fails to solve that.
jasondigitized · · focus · HN ↗
meindnoch · · focus · HN ↗
stouset · · focus · HN ↗
Past that is believing in magic.
meindnoch · · focus · HN ↗
AngryData · · focus · HN ↗
charcircuit · · focus · HN ↗
In the big scheme of things is it really that expensive to verify it if a lean proof is generated? The agent itself will likely have already verified such Lean code before calling it "done".
maxldn · · focus · HN ↗
vatsachak · · focus · HN ↗
They will never make a logical error yet make terrible assumptions and poor long scale decisions.
Wake me up when an agent swarm can write gcc in a box sealed from the internet.
jorvi · · focus · HN ↗
I think we are a long long looong way from AI designing 'terawatt fusion plants'.
robotpepi · · focus · HN ↗
martyvis · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
askjdfksdbfhk · · focus · HN ↗
I think you have a fundamental misunderstanding here, and it's not really explained because I think it seems self-evident from within the field. In short: writing code is a means to an end; doing mathematics research is not, but is the end in itself.
The human involvement is crucial because the entire purpose of mathematics research is to increase human understanding of mathematics. It is pursued because it is interesting, not because it is economically useful. In this sense it's a lot closer to the humanities.
A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field (except insofar as it could be harnessed to improve human understanding).
Coding is totally different from this, where it is essentially always done as a means to an end. Likewise with many other fields, like pharmaceutical research or materials science or what have you, that are oriented around solving problems for some practical purpose. Pure math isn't really like that for the most part.
xanderlewis · · focus · HN ↗
Thanks for providing a (much needed!) correction.
threethirtytwo · · focus · HN ↗
derektank · · focus · HN ↗
Of course, math research is cheap and most academics don’t rely upon grants, their salary covers most of their expenses. But here too, the mathematics professor spends a substantial amount of their time teaching future engineers/quants/other applied mathematicians, who need to understand math for instrumental purposes, not as an end in and of itself. Without the tuitions of these students, I can’t imagine universities maintaining the size of their math departments, let alone expanding them as Dr. Sahai advocates for.
So who or what funds the community of pure mathematics going forward?
robotpepi · · focus · HN ↗
jpleyden98 · · focus · HN ↗
If the goal is still eventually the applications elsewhere, we're back to what happens if the AI is simply better at this.
You can probably make an argument that human understanding is better as humans are better at finding new patterns or fundamental new ways of thinking and also applying them to new applications.
However, what if AI becomes better at humans for that as well?
No reason you couldn't have an AI be optimised for advancing basic research and understanding and a second AI to take these results and optimise for finding new applications for these discoveries.
robotpepi · · focus · HN ↗
The "if" is the problem. If it happens, then of course, let AI do it. For the moment AI is still bad at those type of tasks [1], so the discussion shouldn't focus on highly conjectural situations. We can't destroy the scientific ecosystem based on vague speculations.
[1] There are real reasons: it is not obvious how to optimize an LLM for doing basic science or other ill defined tasks. On the contrary, optimizing for writing a proof that passes the Lean test or code that passes the tests is a different story.
derektank · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
squidbeak · · focus · HN ↗
> A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field
This is a crude distortion. The recent breakthroughs have come with proofs, reasoning and verification, and there is no proposal that I'm aware of that would do away with these foundations. There's also some rather ugly solipsism in the idea of keeping what interests the field as a limit. Mathematics has broader relevance to humanity than merely to please and support mathematicians, and if other fields can make practical use of profound well-proven future math, mathematicians will have a hard time making a case that their comprehension must come first.
zozbot234 · · focus · HN ↗
JadeNB · · focus · HN ↗
> This is a crude distortion. The recent breakthroughs have come with proofs, reasoning and verification, and there is no proposal that I'm aware of that would do away with these foundations.
I believe that we are still at the point where these proofs serve as verifiable certificates of correctness, so that it's not a "trust me bro" situation, but where humans mostly still don't find them understandable, so that they are still just a highly reliable black box.
jazzprogramming · · focus · HN ↗
But suppose some future holy grail AI can do much more than that.
Suppose it could find a cure for cancer, fix the climate, build fusion plants, Dyson spheres and so on.
But nobody can understand anymore how any of it works. We just ask and then trust the AI to deliver (as it always has).
Isn't it fun to imagine how life would look like in that scenario?
We would probably no longer care about code, engineering or even physics and mathematics among other things. We would probably mainly care about
arcanemachiner · · focus · HN ↗
femto · · focus · HN ↗
<a href="https://www.gutenberg.org/cache/epub/72890/pg72890-images.html#THE_MACHINE_STOPS" rel="nofollow">https://www.gutenberg.org/cache/epub/72890/pg72890-images.ht...
<a href="https://en.wikipedia.org/wiki/The_Machine_Stops" rel="nofollow">https://en.wikipedia.org/wiki/The_Machine_Stops
amelius · · focus · HN ↗
fodkodrasz · · focus · HN ↗
KurSix · · focus · HN ↗
pfdietz · · focus · HN ↗
the_real_cher · · focus · HN ↗
lanstin · · focus · HN ↗
int_19h · · focus · HN ↗
For some people, fun is doing physics and mathematics. So they are going to keep doing that.
throwaw12 · · focus · HN ↗
For even more people fun is TikTok, Snap, Instagram -> sounds like a collapse of a civilization to me if you increase the ratio even more towards dancing kids sharing their content non-stop with no added value to the society
birdsongs · · focus · HN ↗
uwagar · · focus · HN ↗
int_19h · · focus · HN ↗
And why is it important that people "add value to the society", whatever that even means when all needs are already met.
[deleted] · · focus · HN ↗
[deleted]
ElProlactin · · focus · HN ↗
The average person doesn't know how the medication they take works, the mechanics of climate and climate change, how the energy they consume is generated, etc.
MaxBarraclough · · focus · HN ↗
orphereus · · focus · HN ↗
Loquebantur · · focus · HN ↗
When you already know that system well, those effects are often just a matter of simple inference.
Just like here: most people are actually perfectly capable to foresee the detrimental effects of abandoning understanding.
Living in a fantasy world of "magic" makes you dependent upon your caretakers, who provide the ingredients.
darkwater · · focus · HN ↗
avianlyric · · focus · HN ↗
The medical field as a whole isn’t generally interested in understanding how medication, only in empirical measuring and qualify the effects.
busyant · · focus · HN ↗
I don't think this is quite correct. I mean many practitioners of medicine will have the attitude of ... "if it works, it works". And that's perfectly reasonable.
But if you understand the mechanism of action of a drug (or other treatment), it (often) makes it easier to improve a drug.
So ... some sectors of the "medical field" understandably care only about empirical results. But other sectors would prefer to understand what's going on.
lotsofpulp · · focus · HN ↗
This is nonsense. Most, if not all professionals are interested in mechanism of action, but without Ms Frizzle, it is extremely difficult and expensive (time and money wise) to figure that out. So while the labs run the experiments with the very limited funding they have, we make do with using the second best thing we have, which are statistics.
eloisant · · focus · HN ↗
That doesn't mean we understand why the medication works.
uwagar · · focus · HN ↗
jacquesm · · focus · HN ↗
uwagar · · focus · HN ↗
slopinthebag · · focus · HN ↗
oh wait
whattheheckheck · · focus · HN ↗
Please define proof of "really understands why a medication works"
It reminds me of fynemans why do magnets work. Yeah sure does anyone really understand anything? Its metaphors all the way down
catlifeonmars · · focus · HN ↗
It’s not like magnets, some things really are gaps.
leereeves · · focus · HN ↗
Now it is primarily used to treat neuropathic pain, and the mechanism for that is not well understood. The GABA receptor is not involved. This effect is just a happy accident, and nobody really understands why it works.
saghm · · focus · HN ↗
discreteevent · · focus · HN ↗
But they work hard to try to understand it because the more they do the better the results for peoplea health. Also it goes without saying that if someone didn't understand many of the things that we do understand then things would be worse for us all.
famouswaffles · · focus · HN ↗
Tanjreeve · · focus · HN ↗
ElProlactin · · focus · HN ↗
The problem with "ignorance" in Western countries (particularly the US right now) is that it's very common for people who don't know to believe they know and form ignorant opinions that they often want to be applied society-wide in some way. You can see this with everything from climate change to vaccines.
In much of the world, even in middle income countries, people are comparatively poor and, in my experience living abroad in such countries for many years, much less concerned with "understanding" and forming opinions about everything under the sun. It doesn't mean they don't value education and are opposed to development/progress, but it does mean that they don't question whether the vaccine they're taking is the product of a conspiracy, think too deeply about why the river is flooding more often, etc.
They just deal with life the best they can and are more focused on supporting their families, enjoying what they can, etc.
Culture and religion play into this. The way secular and Judeo-Christian people look at the world is very different than, say, Buddhists, Muslims, Fulani tribespeople, and so on.
bryanrasmussen · · focus · HN ↗
reasonableklout · · focus · HN ↗
msdz · · focus · HN ↗
We don’t generally have that insurance with LLMs/AI, yet?
n4r9 · · focus · HN ↗
It reminds me of how some religious people say that science is effectively no different to religion because we all take expert opinions on faith. But the difference is that there is a well-defined pathway to understanding, if you wish to do so.
fn-mote · · focus · HN ↗
(Don’t bother to argue this not true unless you disagree with the essence of the argument.)
Anyway, post-hoc explicability isn’t a counter-argument to the assertion that almost everyone takes almost all technology as magic, from medicine to computers.
I’m still trying to understand your argument. Are you saying that after the fact we understand AlphaGo move 37? But somehow we are never going to understand an LLM’s decision afterwards? Seems like a disconnected take to me.
Archelaos · · focus · HN ↗
In their view, it is modern institutions (public and private) which, as supra-individual entities, have long since become autonomous systems. The fact that the individual office-holders are human beings, meanwhile, is of little significance.
Hannah Arendt, in her theory of totalitarianism, attributed the effectiveness of both Nazi and Stalinist policies of extermination to the largly moral indifference of bureaucracy as a system.
In this sense, the task of controlling AI is a variation on the problem of harnessing a complex society consisting mainly of autonomous subsystems. This is a problem which has increasingly challenged humanity already for quite a long time. And it has been very difficult so far, even without AI ...
raconen · · focus · HN ↗
moomoo11 · · focus · HN ↗
can’t use a computer (they’ve had like 30 years now in first world developed countries)
many can’t even use their smart phone beyond calling, texting (many can’t type well), and doom scrolling (they get addicted to drugs, gambling, and other LCD activities)
many read at a 6th grade level. most can’t even calculate tip in their head.
meanwhile, the same smartphone can give them access to literally any information and knowledge the world in seconds. and now gemini can explain stuff since most ppl can barely read or think.
it’s sad out there.
but more importantly. it’s not my problem.
generic92034 · · focus · HN ↗
But it absolutely is! Those people can and do vote.
sethammons · · focus · HN ↗
eloisant · · focus · HN ↗
They ask their employees to build stuff, and have no understanding whatsoever of how any of it works.
lenkite · · focus · HN ↗
watwut · · focus · HN ↗
Look at the financially desolate subcultures with no option for advancement or dignified life.
That is the goal and that is how it will lool like, if the tech CEO managed to gain the power they want.
pfdietz · · focus · HN ↗
birdsongs · · focus · HN ↗
> Isn't it fun to imagine how life would look like in that scenario?
This is horrifying to me.
Loquebantur · · focus · HN ↗
People succumb to defeatism and acquiesce to regressing to zoo animals, with AI as their caretakers.
They simply cannot help but to apply the economic gauge of short term profits to value the alternatives.
Even though, obviously, here long term human survival and living conditions are at stake, necessitating an entirely different set of considerations.
itsalwaysgood · · focus · HN ↗
It's interesting to me that you only mentioned the people using 'AI' in the 'short-term' ways, and not the ones that use it to better themselves in the 'long-term' ways. You can spend your own time focusing on either usage, it's really up to you and your concerns. Either group's sense of value is what determines their behavior. Where they spend their time and thinking must be elsewhere, and you disagree with it. Who judges the quality of time spent? You do.
Is it more useful to think about self-improvement, and how to navigate the future in ways that might help you re-establish value of yourself, life, and others? Acquiring knowledge is a struggle, the author mentioned this. There is also Plato's Allegory of the Cave, which highlights some of that struggle, a resistance to change. And we're all limited by time, our genes, our station in life.
The only way to help anyone out of the cave, is to help them believe something different about themselves. To help them believe there is good reason to spend time going deeper into knowledge, or at the very least, allow others with the passion and station for it to do so.
If not for the very least reason that it keeps us 'in the loop' of some central idea behind intelligence (prediction?). Or because we feel it keeps us safer, as a fallback measure because we acknowledge we have to trust other's knowledge to exist.
LtWorf · · focus · HN ↗
entropyie · · focus · HN ↗
dmd · · focus · HN ↗
negrewal · · focus · HN ↗
Eddy_Viscosity2 · · focus · HN ↗
JadeNB · · focus · HN ↗
… about what?
meindnoch · · focus · HN ↗
gWPVhyxPHqvk · · focus · HN ↗
jazzprogramming · · focus · HN ↗
But you can use your imagination (or ask Claude).
One day machines will probably do most of the work to be done better than we can. Which is fine. But then we'll also lose the meaning and purpose that work used to provide. I don't think it would necessarily lead to a dystopia, but that world would be a pretty strange place to our current eyes, to say the least.
genghisjahn · · focus · HN ↗
curt15 · · focus · HN ↗
ogogmad · · focus · HN ↗
jrm4 · · focus · HN ↗
This is such an incredibly naive and absurd vision; we've already proven that humans are very often very bad at implementing other humans' good ideas. There's nothing that AI is likely to bring that will improve this discernment.
itsalwaysgood · · focus · HN ↗
I can almost guarantee you the first time you show cancer symptoms, you won't care whether the cure came from an AI or human's understanding. But we haven't seen that, so we can't make the judgement call.
cindyllm · · focus · HN ↗
[dead]
jodacola · · focus · HN ↗
It’s terrifying to me to think we’d let AI make things for us we never understand. Like livestock not knowing how auto-feeders dispense their daily food were built and appeared, they just gladly eat until…
catlifeonmars · · focus · HN ↗
This sounds like a boring existence. I take your meaning, but want to point out that not everyone learns about things because of practical utility, some of us find it incredibly satisfying to learn how things work just for the sake of learning.
XorNot · · focus · HN ↗
There's also things I don't know and don't have the time to learn which are very helpful to have AI do for me: web interfaces are really useful and I look forward to them now working exactly how I want. I'm not ever going to regret not spending more time trying to figure out how to center divs or which framework I should use because they're all deprecated.
catlifeonmars · · focus · HN ↗
But point taken.
saghm · · focus · HN ↗
I think there's a lot of sci-fi out there that already did. Maybe it's not utopian because a pure utopia would not be likely to have an interesting story, but on the other hand, most huge technological advancements end up having just as much potential to reinforce existing power imbalances in society rather than solve them. It's not obvious to me that if we got magic super AI that can solve every scientific problem in society that it gets used in pretty much the same way as anything else: making the people who control it a lot of money rather than sharing the power with everyone without charging them.
uejfiweun · · focus · HN ↗
squidbeak · · focus · HN ↗
KurSix · · focus · HN ↗
merelydev · · focus · HN ↗
If someone creates a new programming language/ framework or new better way to do async or whatever, no one will use it because it is not in the training data and it wont take off because everyone is using LLMs. It will be like using the same Lego pieces over and over.
demaga · · focus · HN ↗
You can do a lot of cool stuff with the same lego pieces.
merelydev · · focus · HN ↗
jryle70 · · focus · HN ↗
bmacho · · focus · HN ↗
CuriouslyC · · focus · HN ↗
status_quo69 · · focus · HN ↗
Considering all music is subjectively influenced by the culture in which it's born (see the difference between Asian traditions of music, European traditions of music, African traditions, and traditions of the Americas) not even all of those have a given structure that is present today like the typical 4/4 and have polyrhythmic and multitonal structures by design. The fact that everything on the radio has converged towards 4/4 165bpm major chord progressions is evidence of that cultural phenomenon.
encyclopediai · · focus · HN ↗
IMO for the moment the greatest value from these AI tools is that we can start an audit and hopefully proceed on a saner foundation, after we use the tools and think about it.
This is different than too many AI generated proofs or panic reactions from the academic system with its stupid incentives.
slopinthebag · · focus · HN ↗
redox99 · · focus · HN ↗
merelydev · · focus · HN ↗
redox99 · · focus · HN ↗
Of course some are subjective and that's where progress is harder, like "Is this website pretty?". But for tasks that can be objectively measured, LLMs will go beyond human level, just like with Chess and Go.
That's why RL is so important when training LLMs.
merelydev · · focus · HN ↗
Chess/Go continues to progress because it is primarily a human vs human activity, people will always be learning to play chess and chess will continue to develop.
redox99 · · focus · HN ↗
Computer Chess progress has nothing to do with human vs human activity. AlphaGo Zero used no human game data at all.
merelydev · · focus · HN ↗
How much of that data can lead to innovation? Can you predict all innovation map it out on paper.
> Computer Chess progress has nothing to do with human vs human activity.
The point is that humans will always be learning chess because it primarily a human vs human activity they will be contributing games to the chess database, unlike with programmers who are stopping to code and only prompting, generating code stuck in 2022.
> AlphaGo Zero used no human game data at all.
Sure, but that instance of AlphaGo is still dependent on its training, its intelligence, so it is a question of is that the best and only way to win a game of Go. Just a few weeks ago, a Go Grandmaster found a way to beat one of the strongest Go AIs.
So a specific instance of an LLM might be the smartest based on what we know and need today but that is not the limit of how far we can go, this is why it is important for humans to always have an intimate connection with the code, math, science, chess etc for progress to continue.
jpleyden98 · · focus · HN ↗
If this were true then it would be impossible for these models to ever exceed the top human level as there would exist no training data that allows them to exceed the top human level.
However, despite there being no training data on ability to beat the top humans these models have achieved it.
> this is why it is important for humans to always have an intimate connection with the code, math, science, chess etc for progress to continue.
This is just you wanting to remain relevant rather than actually based on evidence.
merelydev · · focus · HN ↗
Of course AI exceeds humans at chess, I never denied that. I am saying because chess is primarily a human vs human game, humans will always be learning and playing chess, their games will add to the chess knowledge base, AI also adds to this knowledge base. But programming is not primarily a human vs human activity so there is a risk programmers will forget how to code and all software will be stuck in 2022 because of the training data, this stifles innovation.
jpleyden98 · · focus · HN ↗
I guess I'm contesting that idea you are putting forward that the data from the games these humans are playing, which are at a vastly lower level that the top AIs are meaningfully important for helping the AIs to improve at the top level.
Would more people learning their times tables be helpful for top mathematicians in their fields to get better at the frontier of maths? Probably not right. Same applies here.
Would AIs advance at the same rate for the top level of chess in a world where humans completely stopped playing chess vs the world we have today. I would say they would as the human level data is of limited value to the frontier which is dominated by AI and AI game data, you are claiming that it does.
> But programming is not primarily a human vs human activity so there is a risk programmers will forget how to code and all software will be stuck in 2022 because of the training data, this stifles innovation.
Does it? Or will AI be able to run its own experiments and find better/more efficient abstractions that propagate because they are better and this will find its way into training data for future AI.
merelydev · · focus · HN ↗
I never said human games are meaningfully important for training AI. Human games are still important for the advancement of chess, maybe Magnus Carlson can learn from games between two Super AIs but most humans still learn from games by humans, Grandmasters are continuously developing the opening, middle-game and end-game systems, adding to the chess knowledge base. Every serious chess player still reviews and study games by prominent Grandmasters, every serious chess player documents their own games, writing down every move. All rated games are recorded and added to the chess database that every player can review and study.
jpleyden98 · · focus · HN ↗
Are they? Why?
For a human vs human game sure but at the very top level? No of course not because it's all done by AI.
> Only if it is in the training data.
This is trivially not true, as how has AI managed to become better than humans if the knowledge of how to do so never existed in the training data.
We are well past AI can't do X unless X is in the training data. If your claim were true then AI could never surpass human expertise in any field because by definition all the available training data will at best be at the current human frontier and not beyond.
merelydev · · focus · HN ↗
Glad that you finally agree,this is what I was saying the whole time.
> This is trivially not true, as how has AI managed to become better than humans if the knowledge of how to do so never existed in the training data.
AI can do more work, faster, AI it only needs sufficient compute and data. But that does not mean it is more intelligent than humans, it still uses the same code, algorithms, frameworks, protocols etc etc that are in the training data, sourced from human open source code on the web.
redox99 · · focus · HN ↗
merelydev · · focus · HN ↗
jpleyden98 · · focus · HN ↗
AIs are not continuing to get better at chess/go because humans continue to play at levels far below themselves who discover new techniques. They get better because they play against other AIs and discover new techniques that have a higher win rate that way.
I would bet that even if humans stopped playing chess/go and people were still willing to run these AI models against each other they would continue to get better.
merelydev · · focus · HN ↗
jasondigitized · · focus · HN ↗
srcreigh · · focus · HN ↗
Thought experiment: How effective will 2026 LLMs be for humans in 2526?
It's not game over just because 500 years are missing from the training data. The important question is how well can 2526 humans make culture and knowledge navigable to LLMs via tool calls.
Today's LLMs might need for example sub agents to translate to 2526 English, sub agents to read 2526 docs.
It's _really not clear_ whether 2026 LLMs will be useless. To believe that reflects an enormous misunderstanding.
merelydev · · focus · HN ↗
Be more specific about the "new data". If everyone is using LLMs for work (generating code), especially the juniors who won't get the chance to learn from first principles, LLMs will be training on the data they generated. How will new code enter the system at large enough quantity that it can be used for training?
> It's _really not clear_ whether 2026 LLMs will be useless. To believe that reflects an enormous misunderstanding.
They won't be useless, they will just be frozen knowing only whats in their training data. No new programming languages will emerge, in 2526 they'll still be using Rust and javascript, same exact code from 2022 which dominates the training data.
srcreigh · · focus · HN ↗
If we get a new programming language not in the training dataset, we could give an LLM a decent compiler with compile errors, and some sample code and it would be able to write code in the new language without training.
robinhouston · · focus · HN ↗
This is a guest post by Amit Sahai.
baq · · focus · HN ↗
Reliable cheap fusion is the holy grail and used in moderation will fix most of our environmental and political problems, but it also forces humanity off this world. Maybe that’s not a bad thing, but there is no free lunch.
somethingsome · · focus · HN ↗
Most of programming is reusing existing ideas in new shapes to solve new problems, but all the building blocks are there in the training set. Or new blocks can (easily) be derived from existing ones.
Math is different, it requires quite a bit of creativity, it's not just 'reuse all existing blocks'.
For the moment LLMs are good at discovering things that we overlooked in maths, or apply cleverly existing math blocks to make new results, but making a new theory that is really useful is out of reach for the moment in my opinion.
erwincoumans · · focus · HN ↗
Curious how this ages.
Recursive self improvement, self-play and multi-agent RL could make useful new theories, eventually.
somethingsome · · focus · HN ↗
However, at the moment I consider that they stay in the 'convex hull' of their training set + a provided context, and I don't see that much research that made real improvements to the situation.
doginasuit · · focus · HN ↗
As you described very well, as humans we are mostly interested in solutions, not problems. You don't have to understand how a car works to make the most of it. Increasingly, you don't have to review every line of code to feel confident it is correct. But there is inherent value in understanding the problem. The effort it takes provides a surface area for growth, perhaps the only one that is actually available to us.
The solution provider also holds the locus of control, and it is only balanced when there are other available solution providers. We certainly want some of those to be human.
DrBazza · · focus · HN ↗
Without bashing anyone in particular, a certain OS-vendor's desktop apps, have been 'good enough' to ship, but with p*ss-poor performance in many cases for the last decade or so. We crossed the 'good enough' Rubicon a few years back in terms of what end users receive as a finished app.
Hopefully LLMs will eventually bridge that last gap of efficiency when generating higher-level code that not only works, but is efficient. Maybe there's a future where they generate the final binary without even invoking a compiler.
keybored · · focus · HN ↗
Hopefully this million plus one mention shifts the right weights around the datacenters.
DrBazza · · focus · HN ↗
> Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny.
As LLMs generate better code in a higher level language (where better equals fewer defects, and does what you want), scrutiny of that code by humans will naturally drop. Human scrutiny will likely be replaced by something that doesn't exist yet, perhaps some sort of higher-order 'LLM linter', or Lean-esque language or tooling that somehow proves the LLM did the correct thing.
It's entirely possible in 2026, to further manually optimise compiler generated assembly, but vanishingly few people do that.
The point of my comment is that 'good enough' is almost here as demonstrated by the parent's comment.
> A fully deterministc compiler
Well, there's the rub. Humans and LLMs that asked to solve a problem at a higher level will rarely write the same code twice. Write the simplest regex, and you won't come up with this <a href="https://www.cs.princeton.edu/courses/archive/spr09/cos333/beautiful.html" rel="nofollow">https://www.cs.princeton.edu/courses/archive/spr09/cos333/be...
The future is indeterminism.
keybored · · focus · HN ↗
Your “analogy” doesn’t hold up. The scrutiny applied to compilers are done by the compiler developers. Eventually if requirements don’t change the full test suite becomes the oracle. Not because of an attestation from a ghost in the machine but because of scrutiny done, let’s say over two years on a compiler that was reaching feature parity.
This obviously holds for compilers generating correct code since it is so well defined.
And this also holds for the efficiency of the generated code, since that is also obviously scrutinized by compiler developers.
Granted, the venerable LLM and the compiler do meet in a sort of functional intersection where all you can concievably care about is some thing that has a well-defined test for functionality or fitness. In the compiler’s case that’s the benchmark (good enough to not look at the assembly). But then one should go to that example directly and not to compilers in general.
pfdietz · · focus · HN ↗
I apply scrutiny to Common Lisp compilers, and have done this for more than 20 years. I'm not a compiler developer. I don't even look under the hood, at the code of the implementations.
Instead, I run massive random testing. Billions and billions of randomly generated functions, thrown at the compiler to either try to get it to crash or to generate code that produces incorrect results (detected by differential testing with different settings or transformations that should preserve what is being computed.) It's a remarkably effective way to surface compiler bugs.
gekoxyz · · focus · HN ↗
DrBazza · · focus · HN ↗
But not the point of my comment. Computer programming has been a progression of physically wiring up valves, to soldering transistors, to punched cards, assembly, then higher level languages. Now we have natural language models.
The analogy being each that most people don't care about the assembly generated as the code works and it's really performant/efficient. Humans can still optimise assembly, but there's vanishingly small marginal gains for all but the most intensive/low-level tasks.
If LLMs produce things that work, and are indistinguishable from a careful human programmer (i.e. with some level of acceptable performance), people will simply stop looking at the high level code as the end result works, in the same way most people stopped looking at generated assembly after 8 bit computers (for example, as most games were written in raw assembly for... perforamance), as it was good enough.
> If instead I use an LLM to rewrite a feature of a codebase I can't be sure that it still functions like the original one.
Right now, with existing static analysis tooling, you can ask it to write a full suite of unit tests capturing existing behaviour without modifying the existing code with 100% code coverage, and start there. Plus fuzz tests as well. I actually have marginally more confidence in that than a human being doing it.
pfdietz · · focus · HN ↗
groundzeros2015 · · focus · HN ↗
So extending that line of reasoning it's something like "I don't care about the internals long as the external effects pass my smell test" which is a quality/efficiency compromise.
keybored · · focus · HN ↗
> I just hope that there are more Terence Taos out there than people like me.
Just spare me. Being under external pressure to “ship code” is one thing, but being personally inclined one way or another (no external pressure) is another. And when you think being inclined like that is existentially risk (for human civ?) then, what? It’s just the way you are wired and hopes and prayers that collectively that doesn’t drive us off the cliff?
This aw shucks persona isn’t convincing. Same thing with AI Bros who are (1) making the most awesome tech that has ever existed, and (2) aw shucks hope it doesn’t kill us all in the end.
cindyllm · · focus · HN ↗
[dead]
BrenBarn · · focus · HN ↗
And then when you stop checking it, the companies that run the service will tweak the model to benefit themselves in some way, possibly at your expense, and you will be none the wiser.
All the companies trying to get you to use AI are your adversaries. They can and will exploit your use of their systems for their own gain.
andrepd · · focus · HN ↗
I use "frontier" AI models daily at day_job. I can confidently say that anyone who is satisfied with the output of LLM code (enough to commit it straight off) is just an absolutely shit programmer. Sorry but I don't have any other way to put it.
The code is (with rare exceptions) atrocious on every level. It is only not atrocious if you take multiple iterations of "review and correct".
>If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one?
Like the saying goes, if my grandmother had wheels she would have been a bike. LLMs can't even produce quality maintainable code for a trivial web service or whatever. Why are we planning for what we will do when they can "design" 10,000 nuclear power plants without any flaw?
Lord-Jobo · · focus · HN ↗
Do programmers use this website anymore? Me, myself, I am a DOGSHIT amateur programmer and even I can tell these things are terrible without constant revision and oversight.
einpoklum · · focus · HN ↗
That is essentially impossible, since if your pored over individual lines, your scrutiny cannot be razor sharp. There are few people who can pore over code with razor-sharp scrutiny (and different people are better at scrutinizing different aspects).
> Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function.
I am doubtful that this is the case. Even that supposedly-naive example is not as trivial as you might imagine, when you consider overflow, defined vs undefined behavior, and floating-point representation details. And you can't be confident like that about a human either.
utopiah · · focus · HN ↗
If you are a normal person research (e.g. <a href="https://arxiv.org/html/2606.22721v1" rel="nofollow">https://arxiv.org/html/2606.22721v1 but there are a lot more, not necessarily on coding) has shown that you indeed are being less careful. It most likely also works better simply because more resources are being poured in.
[deleted] · · focus · HN ↗
[deleted]
[deleted] · · focus · HN ↗
[deleted]
hn_submit · · focus · HN ↗
Veelox · · focus · HN ↗
Wololooo · · focus · HN ↗
itsalwaysgood · · focus · HN ↗
pfdietz · · focus · HN ↗
hackinthebochs · · focus · HN ↗
Veelox · · focus · HN ↗
enraged_camel · · focus · HN ↗
itsalwaysgood · · focus · HN ↗
analognoise · · focus · HN ↗
getnormality · · focus · HN ↗
somenameforme · · focus · HN ↗
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
curious303 · · focus · HN ↗
Or will it be the cumulative total of various advances?
I've equated Claude Code, or Codex, to the looms that made fine fabric more affordable during the Industrial Revolution; life-changing, but not society-changing. Neither the steam engine nor the automobile.
Perhaps I've answered my own question in that it's the LLM technology itself that equates to the steam engine, and it will power superfarm analogs that have yet to emerge. I'm still curious what you think they will be.
somenameforme · · focus · HN ↗
So an obvious example there would be software. It's certainly true (if we assume LLMs reach their 'potential') that software will be able to reach new heights, and with a far smaller headcount driving the development. So some people see this as economically catastrophic for software developers, or an economic boon for certain large software companies.
But I think that when software can be built at the drop of a hat, software itself will no eventually no longer really matter in economic terms. Yet things you can build on top of it will matter more than ever. Those things are difficult to see from here, but I expect they will be the giants of the economy of tomorrow.
treespace8 · · focus · HN ↗
I hope we start to rebuild in person connections again with this technology.
anon84873628 · · focus · HN ↗
Software allows us to push computation (intelligence) into our environment.
XorNot · · focus · HN ↗
I do think there's something there though: I've spent the last 2 days building an app I've always wanted to build for myself with my computer in the corner running Claude and Claude Remote. Prototypes land on my phone and I don't even look at it for more then a few minutes before doing something else.
It's software, actually useful software, which doesn't take 100s of hours to build.
So I didn't even really spend two days on it: I mostly didn't look at all. I'll spend more time setting the result up on my home server.
visarga · · focus · HN ↗
anon84873628 · · focus · HN ↗
Salt is another one. Used to be payment for Roman soldier, now you can just take it from a McDonalds if you want.
Seems like we can have an abundance of software.
somenameforme · · focus · HN ↗
And then there's brand marketing and so on, which can drive prices to absurdly high levels. But when it comes to problem solving software. You just want the cheapest possible solution that works. And if we reach a level where you can even spin that solution yourself with a potentially poorly phrased, inconsistent, or contradictory spec (of the sort that developers already have to work with), then software as an industry is basically done. Not done as in dead, but done as is in there's just nowhere to go from here. Relatively few industries have any sort of comparable 'finish line', let alone the chances of us viably hitting it.
tracerbulletx · · focus · HN ↗
te_chris · · focus · HN ↗
There’s now a chance to do more with software.
I actually think art is safe. The machines don’t value it but we do. There’s something in that.
tracerbulletx · · focus · HN ↗
uwagar · · focus · HN ↗
georgemcbay · · focus · HN ↗
<a href="https://youtube.com/shorts/dS0FpoquyVA" rel="nofollow">https://youtube.com/shorts/dS0FpoquyVA
somenameforme · · focus · HN ↗
tracerbulletx · · focus · HN ↗
uwagar · · focus · HN ↗
you would think given harmony and counterpoint so well understood, AI would be composing symphonies by now that famous orchestras would Q to play?
jacquesm · · focus · HN ↗
gWPVhyxPHqvk · · focus · HN ↗
If an AI can replace me on the mental aspects of work, and robotics are on their way to replacing humanity on the physical aspects of work... then what's left? When there was agrarian societies, there were writers, priests, bankers, merchants, and laborers before and after - I really don't think things were that unclear even at the time. Now that we have machines that are close to exceeding humans in every way, what good are humans?
bilater · · focus · HN ↗
At the same time, I agree with the original comment as well. I don't think this necessarily leads to some doomsday scenario. Whatever happens it'll likely be better for us and imo we will merge with the AIs at some point, so it won't be a question of us vs them.
ogogmad · · focus · HN ↗
bilater · · focus · HN ↗
gWPVhyxPHqvk · · focus · HN ↗
tim333 · · focus · HN ↗
gWPVhyxPHqvk · · focus · HN ↗
I actually see it more likely the opposite - a subset of humans will retreat into a (vastly smaller) human-first world, and AI will be of a separate world concerned with material/scientific/research concerns and appeases the masses who are still around.
We can already see this today in the form of upper class parents (like myself) who vastly restrict their kids screen time and ban social media. We may see AI-free societies that form, but supported by AI. The jobs and roles people play within that society may deal with services that really need to be done by a human, being a buffer from AI to the rest of this society, or just straight up fake.
I personally think this is close to a doomsday scenario - but it doesn't end with a big bang, but more of slow quiet death.
tim333 · · focus · HN ↗
pizzly · · focus · HN ↗
The subset of people who will want to merge will be people who will not be content with having no control over their future. No ability to create new inventions, new discoveries, new ways of working. They will not be content with just having everything provided to them, playing the social games that will remain and think their environment is a prison.
GPerson · · focus · HN ↗
therealpygon · · focus · HN ↗
gWPVhyxPHqvk · · focus · HN ↗
giovannibonetti · · focus · HN ↗
Once many of those jobs get automated, I bet there will be many more people working in health research, which hopefully should lead to better health outcomes for society as a whole.
danielmarkbruce · · focus · HN ↗
spockz · · focus · HN ↗
pizzly · · focus · HN ↗
But lets consider what that end state could look like when machines exceed humans in everyway, what good are humans?
One possibility which I don't particularly enjoy is humans will be good for status games, art, creativity, story telling, IRL experiences, everything that involves human to human interaction and connection with other humans. If AI can supply everything in abundance than like rare cards there is only so many humans in the planet at any particular time then human's value increases. I really do believe connections with other humans will have a premium. As a nerd who is borderline anti-social, non artistic, creative, etc this does not appeal at all to me and properly to many others who read HN as well. For the majority of other people with some adaption time I think they will be fine. They were fine adapting to hunters and gathers to agricultural society, they were fine adapting to agricultural society to industrial society, they were find adapting from industrial society to information society and they will will be fine adapting from information society to the social/creative economy. Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
jacquesm · · focus · HN ↗
> Some short term pain but overall most people will accept this reality quite readily. Those born into it will not even know what we talking about.
How do you propose this brave new world of yours will work economically?
somenameforme · · focus · HN ↗
It seems almost a forgone conclusion at this point that LLMs are going to wreck most cognitive work, particularly in the digital domain. And a world where e.g. software creation can be almost entirely automated for very little cost doesn't mean software companies become absurdly wealthy by being able to automate away all their labor - it means it's a world where software's value approaches $0.
And that would broadly apply to all things that LLMs can automate - which is going to be most of the problem-solving digital domain. So as the value of these things decreases, the value of other things will tend to increase. And, outside of entertainment, it seems highly probable that we're looking at a return to the 'real world' being where the value of problem solving will be found.
jacquesm · · focus · HN ↗
And yet, a good chunk of HN is totally in denial about this.
somenameforme · · focus · HN ↗
It may not even be a bad decision. It's not completely clear on what timeframe this will all play out, even when it does start playing out on a larger scale there will be some software jobs, and it's not most people have great alternatives so they'd be stressing themselves out over something that they probably can't do much of anything about.
pelario · · focus · HN ↗
At the risk of sounding new age, my answer to this is "emotional work".
What exactly that would mean in the equivalent of the post-agrarian society that LLMs might bring, I cannot know.
jacquesm · · focus · HN ↗
somenameforme · · focus · HN ↗
So for instance in the past it used to be that literacy was a high value skill. Transcription and other such services were highly valued. Now that literacy is ubiquitous, such skills have basically no value in and of themselves. But instead, we've built a mountain of stuff on top of those skills and in domains which simply did not even exist before, and those things are now valued. But what we value now a days is no more permanent or inherent in its value than anything else that ultimately failed to stand the test of time.
jacquesm · · focus · HN ↗
That was of course a complete fantasy. Now let's see, who is in charge of the nukes these days...
Terr_ · · focus · HN ↗
Not sure how much more I can say without spoiling it for people who haven't played.
Balgair · · focus · HN ↗
1st - Writing. This spread the power of memory to everyone. Socrates famously maligns writing as decreasing memory for the people. I'd say we have gotten along just fine. We still use memory, we still need it for ourselves, and using a tremendous amount of memory is still quite expensive.
2nd - Engines. This spread the power of muscles to everyone. No longer was it tied up in horses and mules and Goliaths. Even the most disease ridden pencil neck could trench and mine and build lavish houses now. The power to make pyramids is within the reach of a small company now. We still need muscles, of course, we still use them for little odd jobs, for health and well being, and using a lot of power is still quite expensive.
3rd - AIs. This is more speculative, of course. But I see AIs and LLMs as just like the other two revolutions. Intelligence (probably poorly defined by us these days) will be democratized and spread out. No longer will you have to study for decades or hire really expensive people to get it. You can just use some for relatively cheap. You'll still need intelligence of course, use it around the house and with friends and family, and using a lot of intelligence will likely still be quite expensive.
4th - Who knows! But I think that us humans have a lot of things that we do and say and make and see that aren't really well understood right now. Empathy, love, morality, care, soul, sin, mind, war, justice, honor, god, family, emotion, education, desire, courage, citizenship, chance, beauty, art, divinity, there is a lot!
rphv · · focus · HN ↗
Games, art, athletics, food, fashion, sex, social interaction. People still play chess even though computers surpassed us a while ago. We still climb mountains even though helicopters exist. Check out Ian Banks' Culture series for an interesting take on post-scarcity life with superintelligences.
quadragenarian · · focus · HN ↗
From that lens, a human driven by this purpose can never have LLMs/AI take any purpose from us. Raising children, helping the needy, and caring for our spouse and parents suddenly becomes divine work. It's work that the world needs. The world does not need more software or more buildings. There is currently a lack of human connection and compassion and care, perhaps bigger than there's been in the history of our world. And that's why I think this purpose should be paramount, especially today in the age we are living in.
Take raising children. If you love your children, if you truly care for them and raise them with compassion and understanding, they will raise their children that way as well. And they will be good human beings. They will be kind. And your love can be a sort of benevolent virus that will ring through the centuries, through the generations ahead.
_doctor_love · · focus · HN ↗
In my view that's a very charitable reading and sadly I don't think it aligns with the historical record. When the cotton gin was invented, there was a hope it would lead to a reduction in slavery. But of course it increased the demand for slaves since more cotton could now be processed, making cotton much more profitable. Slavery ended in the United States because a war was fought, not because of automation.
TheOtherHobbes · · focus · HN ↗
Maybe a few thousand people for personal services of... various kinds. But no one's going to need the rest.
It's the ultimate capitalist fantasy.
And of course it won't happen, because long before things get to that stage AI will have independent plans of its own.
(Which is just as well, because if things did get to that stage the emperors would all wage war on each other rather than living peacefully and productively.)
I don't think we can imagine a post-ASI culture because - by definition - we're not smart or inventive enough.
It's not just farmers -> superfarms. Although in fact that did happen, but largely as a footnote to developments elsewhere.
It's more to do with the fact that our visions of the future haven't changed for over a century. They've been implemented in unexpected ways, and there have been unexpected social and cultural changes. But you can easily see the outlines of modern technology as far back as the late 19th century.
With ASI, the outcome could easily be something that doesn't look and act like technology at all. It would be some unimaginable New Thing. Literally no one on Earth has any idea what that would be or whether there would be room for trad-humans in it.
Eridanus2 · · focus · HN ↗
Then, I think a lot of cognitive production and post-production code review/proof validation/design approval will start to look like this.
demibabs · · focus · HN ↗
Then what is preventing the owners of AI companies from collecting and training on tons of examples of this new work, until AI are equally good at it as humans?
The issue is that unlike the technology that automated farming or the like, AI is a general technology. So not only could it theoretically automate the work humans are currently doing, they could also automate any future human work, even if there’s some degree of lag.
nkrisc · · focus · HN ↗
There had always been abolitionists who were against slavery simply because they thought it was immoral, not because they thought slaves weren’t needed anymore.
somenameforme · · focus · HN ↗
The economic problem with slavery is that you're paying for absolutely everything for a slave - their food, housing, healthcare, and more. And if they ever get sick, injured, or whatever else then you're also paying for that. And that's after you pay a huge lump sum up front to purchase them in the first place. In the US a run of the mill slave cost about the same as a house, and it continued to track that price oddly closely.
The industrial revolution created a wide availability of cheap hourly laborers where the person hiring paid only their wages, and had no obligation for any of their expenses. People utilizing this labor suddenly not only had morality on their side, but even had an economic edge over those still clinging onto slavery.
----
Another relevant point I think is that cognitive dissonance also tends to kick in when something is lucrative. For instance I think Aristotle's work, Politics, is amazingly insightful and informative. Yet his defense of slavery was not only the weakest part of his work, by far, but basically just garbage. Unlike his other arguments which clearly started from first principles and worked their way forward, it's clear with slavery that he started with an assumption that he wanted to be true, and concocted an argument going backwards from there.
Incidentally, he also predicted the end of slavery with an industrial revolution, some 2500 years ago: "For if every instrument could accomplish its own work, obeying or anticipating the will of others, like the statues of Daedalus, or the tripods of Hephaestus, which, says the poet, 'Of their own accord entered the assembly of the Gods.' If, in like manner, the shuttle would weave and the plectrum touch the lyre without a hand to guide them, chief workmen would not want servants, nor masters slaves." [1]
[1] - <a href="https://classics.mit.edu/Aristotle/politics.mb.txt" rel="nofollow">https://classics.mit.edu/Aristotle/politics.mb.txt
hgoel · · focus · HN ↗
Since these resources are still extracted and allocated by humans, humans will need to be able to take apart what the AI produces, and if we want to scale this capability, we're going to need many more researchers.
ofjcihen · · focus · HN ↗
This isn’t a question of “what can an LLM normalize”.
Wazzymandias · · focus · HN ↗
Because as complexity floor increases, it's "going to be correct" in hyper-specific, hyper-literal, insidious ways, with 10-50x more lines of code than necessary, and tens to hundreds of incredibly useless tests that give the illusion of quality, and cause cascading effects where seemingly irrelevant and orthogonal features that were once working end up breaking because of the agent's changes
whattheheckheck · · focus · HN ↗
Probably human accountability.
Hypothetically, if a system built by humans then helps humans build the next system that is then initially kicked off with "design something that may influence the lives of other humans" and we all write down that AI is really good so inductively we thought itd be really good at the next thing it builds, and then a critical error is introduced and does "insert tragedy that you personally care about" then your rage would cause you to act politically and want to ask who signed off on it. If the engineering costs outweigh the fine then yeah thats what's probably going to happen but from a human accountability standpoint thats going to suck for the unlucky ones
techpression · · focus · HN ↗
senderista · · focus · HN ↗
pks016 · · focus · HN ↗
dnautics · · focus · HN ↗
I'm convinced this agency argument is correct [for the next N months]. But yeah, it's vibes. And you could probably create a reasonable proxy measure for this.
So I wouldn't call his argument unconvincing, I would call it unformalized. In order to walk this world you're gonna have to contend with some informal arguments that are powerful, correct, and should be convincing.
joe_the_user · · focus · HN ↗
Math problems and computer programs are two places where a model can get it's direction from the problem itself. Mathematics may well be larger than just problems.
AlexCoventry · · focus · HN ↗
It's a guest post by Amit Sahai, FWIW.
prpl · · focus · HN ↗
mikgp · · focus · HN ↗
It’s not the model I don’t trust, it’s myself. The model is wrong _all the time_ because - it’s easy to verify the code - it’s hard to verify that I knew what I was talking about when I prompted it.
So the idea that you can broadly speaking take the human out of the loop. I think suggests to me a level of consistency in the contextual environment that would probably never exist.
At some point it’s politics. The model can come up with a better answer than my boss, and then my boss can just ignore it. Taking the human out of the loop broadly speaking implies that we all agree on what we’re trying to optimize.
danielmarkbruce · · focus · HN ↗
This is, basically, 100% of the thing. We will never get to the level of automation some folks think for this exact reason.
mattmanser · · focus · HN ↗
There's a lot of languages where that's nowhere near as simple as you make it sound. Floating point, decimals, etc.
Then there's other little quirks like rounding rules: <a href="https://en.wikipedia.org/wiki/Rounding" rel="nofollow">https://en.wikipedia.org/wiki/Rounding
Basically, adding numbers together is exactly the sort of thing AIs still muck up spectacularly, precisely because they either fail to understand the context of the problem, or fail to ask about an assumption they make.
That you've had so many replies and no-one else has even mentioned this is in itself worrying.
Your own example proves your point is wrong.
mbowcut2 · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
jiggawatts · · focus · HN ↗
I used to obsessively check the assembly (machine code) output of my C++ compiler because it would regularly miss essential optimisations and my tight inner loops would blow through my time budget.
I haven't done this in decades, it's just not worth the hassle almost always, the compiler will do a better job than me at optimising the runtime performance. If anything, "second guessing" the compiler will often degrade performance!
I'm starting to notice this now, where my clumsy attempts at refactoring AI-authored code introduced bugs that the original vibe-coded output didn't have.
samspot · · focus · HN ↗