> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
>Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say.
I think you have a fundamental misunderstanding here, and it's not really explained because I think it seems self-evident from within the field. In short: writing code is a means to an end; doing mathematics research is not, but is the end in itself.
The human involvement is crucial because the entire purpose of mathematics research is to increase human understanding of mathematics. It is pursued because it is interesting, not because it is economically useful. In this sense it's a lot closer to the humanities.
A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field (except insofar as it could be harnessed to improve human understanding).
Coding is totally different from this, where it is essentially always done as a means to an end. Likewise with many other fields, like pharmaceutical research or materials science or what have you, that are oriented around solving problems for some practical purpose. Pure math isn't really like that for the most part.
Sure, but I don’t think most of the money that goes into funding math is for the purposes of pure understanding. The reason governments fund mathematics research grants is generally for a more instrumental purpose; taking the US congress as an example, the mission of the NSF is to, “Promote the progress of science; advance national health, prosperity, and welfare; and secure national defense.” Most federal math grants come from the NSF.
Of course, math research is cheap and most academics don’t rely upon grants, their salary covers most of their expenses. But here too, the mathematics professor spends a substantial amount of their time teaching future engineers/quants/other applied mathematicians, who need to understand math for instrumental purposes, not as an end in and of itself. Without the tuitions of these students, I can’t imagine universities maintaining the size of their math departments, let alone expanding them as Dr. Sahai advocates for.
So who or what funds the community of pure mathematics going forward?
Research in pure mathematics is part of what we call "basic research". There are no applications in mind a priori. People instead focus on understanding, because history has taught us that understanding tough problems in mathematics finds natural applications elsewhere. It's the same as theoretical physics or theoretical computer science.
> People instead focus on understanding, because history has taught us that understanding tough problems in mathematics finds natural applications elsewhere.
If the goal is still eventually the applications elsewhere, we're back to what happens if the AI is simply better at this.
You can probably make an argument that human understanding is better as humans are better at finding new patterns or fundamental new ways of thinking and also applying them to new applications.
However, what if AI becomes better at humans for that as well?
No reason you couldn't have an AI be optimised for advancing basic research and understanding and a second AI to take these results and optimise for finding new applications for these discoveries.
> what if AI becomes better at humans for that as well?
The "if" is the problem. If it happens, then of course, let AI do it. For the moment AI is still bad at those type of tasks [1], so the discussion shouldn't focus on highly conjectural situations. We can't destroy the scientific ecosystem based on vague speculations.
[1] There are real reasons: it is not obvious how to optimize an LLM for doing basic science or other ill defined tasks. On the contrary, optimizing for writing a proof that passes the Lean test or code that passes the tests is a different story.
pyridines · · focus · HN ↗
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
askjdfksdbfhk · · focus · HN ↗
I think you have a fundamental misunderstanding here, and it's not really explained because I think it seems self-evident from within the field. In short: writing code is a means to an end; doing mathematics research is not, but is the end in itself.
The human involvement is crucial because the entire purpose of mathematics research is to increase human understanding of mathematics. It is pursued because it is interesting, not because it is economically useful. In this sense it's a lot closer to the humanities.
A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field (except insofar as it could be harnessed to improve human understanding).
Coding is totally different from this, where it is essentially always done as a means to an end. Likewise with many other fields, like pharmaceutical research or materials science or what have you, that are oriented around solving problems for some practical purpose. Pure math isn't really like that for the most part.
derektank · · focus · HN ↗
Of course, math research is cheap and most academics don’t rely upon grants, their salary covers most of their expenses. But here too, the mathematics professor spends a substantial amount of their time teaching future engineers/quants/other applied mathematicians, who need to understand math for instrumental purposes, not as an end in and of itself. Without the tuitions of these students, I can’t imagine universities maintaining the size of their math departments, let alone expanding them as Dr. Sahai advocates for.
So who or what funds the community of pure mathematics going forward?
robotpepi · · focus · HN ↗
jpleyden98 · · focus · HN ↗
If the goal is still eventually the applications elsewhere, we're back to what happens if the AI is simply better at this.
You can probably make an argument that human understanding is better as humans are better at finding new patterns or fundamental new ways of thinking and also applying them to new applications.
However, what if AI becomes better at humans for that as well?
No reason you couldn't have an AI be optimised for advancing basic research and understanding and a second AI to take these results and optimise for finding new applications for these discoveries.
robotpepi · · focus · HN ↗
The "if" is the problem. If it happens, then of course, let AI do it. For the moment AI is still bad at those type of tasks [1], so the discussion shouldn't focus on highly conjectural situations. We can't destroy the scientific ecosystem based on vague speculations.
[1] There are real reasons: it is not obvious how to optimize an LLM for doing basic science or other ill defined tasks. On the contrary, optimizing for writing a proof that passes the Lean test or code that passes the tests is a different story.