> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?
I'm especially confused about a prior statement as a scientist:
> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).
How does a set of beliefs help with reasoning?
> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I think an analogy would be helpful. As an LLM, reasoning through text, you wouldn't 'know' the idea of a man separately from, posterior to, the words man in say, French and English. As a human, you WOULD know the idea of man, and you would mean that idea when you say the French or English words for man.
For an LLM, although its approximation of knowledge would lead it to claim it knows they're the same thing, there would be differences in its weights that influence its usage of both the French and English words for man, and which may lead it to conclusions in one language it wouldn't reach in another. Because its knowledge/reasoning is interwoven to its knowledge, not prior to it.
> How does a set of beliefs help with reasoning?
It’s not, because it’s not evidence for the argument in the article.
> As a human, you WOULD know the idea of man
You’re making a claim about the fundamental mechanics of human biology. I don’t take these at face value. For one, language (in a very broad sense) put a major selective pressure on our species. So, to test your hypothesis, we’d need to find people who do not use any language and somehow test they have an abstract idea of a man.
To put it otherwise, I don’t know whether our reasoning can be decoupled from speech or language. There’re some bits and pieces here and there, like deaf people being at higher risk of neurodegeneration.
And let’s not forget there’re claims that AI models develop internal representations of various concepts. Anthropic made a very interesting claim, but I’m not an ML expert so can’t judge whether it’s sound.
f6v · · focus · HN ↗
I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?
I'm especially confused about a prior statement as a scientist:
> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).
How does a set of beliefs help with reasoning?
> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
We also often do the same as humans.
nateb2022 · · focus · HN ↗
I think an analogy would be helpful. As an LLM, reasoning through text, you wouldn't 'know' the idea of a man separately from, posterior to, the words man in say, French and English. As a human, you WOULD know the idea of man, and you would mean that idea when you say the French or English words for man.
For an LLM, although its approximation of knowledge would lead it to claim it knows they're the same thing, there would be differences in its weights that influence its usage of both the French and English words for man, and which may lead it to conclusions in one language it wouldn't reach in another. Because its knowledge/reasoning is interwoven to its knowledge, not prior to it.
> How does a set of beliefs help with reasoning?
You cannot have a syllogism without propositions.
f6v · · focus · HN ↗
It’s not, because it’s not evidence for the argument in the article.
> As a human, you WOULD know the idea of man
You’re making a claim about the fundamental mechanics of human biology. I don’t take these at face value. For one, language (in a very broad sense) put a major selective pressure on our species. So, to test your hypothesis, we’d need to find people who do not use any language and somehow test they have an abstract idea of a man.
To put it otherwise, I don’t know whether our reasoning can be decoupled from speech or language. There’re some bits and pieces here and there, like deaf people being at higher risk of neurodegeneration.
And let’s not forget there’re claims that AI models develop internal representations of various concepts. Anthropic made a very interesting claim, but I’m not an ML expert so can’t judge whether it’s sound.