Worth reading Iain McGlichrist's The Master and His Emissary. It accounts for quite a bit of the confusion here. Some folks are truly locked in the "left hemisphere," as it were. They can't tell the difference between human intelligence and AI, likely because most of their own thinking is pretty close to AI. Those with a more rounded intelligence spot it right away and struggle to get the message across to the others. You'll notice that the defenders of AI tend to fly into rages. That's a marker of what I call leftbrainitis, as is "confabulation," ie making up stories and facts to secure being right (hallucinations?). Those with a healthy and active "right brain" can hold two contradictory notions in their minds without feeling the need to settle on a conclusion. I'm just skimming the surface here. Best to read McGilchrist.
Thank you for the name I know, without remembering if I read anything of it or intended to.
I will read a transcript of one of his lectures that I just found to find out more.
I can however say that following my previous experiences and research, it would seem that we should take into account, that the studies and conclusions, are always based on observation and an interpretation part (the latter is regrettable), because not everything is visible by instruments, sensors... however powerful they are.
This is a very interesting subject, the reading will bring, I suppose, some reflections that I may not have thought of.
"actions that will destroy the integrity of civilization’s information repositories." A very powerful and provocative statement. Something new to add to the list of the unravelling of the Homocene. An valuable idea that has not gotten any/enough? scrutiny, heretofore. Many thanks.
First, it seems you might be overlooking AIs that have developed novel semantic systems, or languages, with rules we don't fully comprehend. While I wouldn’t equate this to human language creation, it appears to challenge the kind of the intelligence with which you're concerned. Therefore, it seems necessary to distinguish between AI's creation of a language (or "semantic exploration," as you term it) and our own processes in achieving the same.
Second, you seem to be overlooking Wittgenstein's account of rule following as rendered by Saul Kripke, particularly the quus dilemma. This dilemma illustrates a scenario where a rule follower, while learning addition, begins making inexplicable mistakes, consistently rendering the same incorrect result. This highlights that rule following isn't derived from singular observations but requires multiple instances to probabilistically determine the appropriate application of the rule per others' expectations for said application. For further exploration of this concept, I recommend Lorraine Daston's *Rules*, which examines how Benedictine monks utilized "prudentia" (prudence) to (perhaps probabilistically) discern rule application through repeat observation and application. Her discussion of the craft tradition in medieval Europe is good on this point also.
All of this seems to invoke Bayesianism, which I'm surprised receives no mention. So far as most psychology literature is concerned, we are purely Bayesian calculators. I do not agree with this—far from it. To refute such literature, though, requires at least a mention of the arguments that psychologists currently deploy to reduce us to calculators.
> AIs that have developed novel semantic systems, or languages, with rules we don't fully comprehend.
Can you provide a reference of such account? I'm aware of the Facebook case where it was stated two chat bots created their own language. However, that was a case of essentially using shorthand patterns. Not new semantic information.
> you seem to be overlooking Wittgenstein's account of rule following
From the premise, as I understand it, it seems to imply we can't verify that another person comprehends rules from simply data sampling. Although it may be the case we can't verify the understanding in such a testable manner. The concept is logically contradictory. The very fact that we can perceive the potential for an error outside of the stated rule conditions would imply that we comprehend the rule.
If we weren't able to do so, then we would see highly indeterminate behavior among technology created on the top of formal systems. It just simply wouldn't be possible. We wouldn't be able to build reliable interfaces between systems etc. as all such are constructed on the basis of rules or specifications.
> To refute such literature, though, requires at least a mention of the arguments that psychologists currently deploy to reduce us to calculators
Relevant, but I don't attempt to take on every argument necessarily at once. I might follow up with something on that at some point. I already struggle with getting most people to read my articles, being longer usually doesn't help :-)
You are wrong about humans able to explain their reasoning. Humans are full of heuristics and constantly hallucinate about reasons for their decisions, just like the LLM in your summation example, this is a known fact. Like 99% of your decisions are made before your inner voice explain the reasoning for them. The inner voice is actually there to explain, not to make the decisions. And it is often wrong. Unlike lower-level reasoning, humans indeed have to understand high-level reasoning in order to use it and in order to pass it, but right now you cannot prove LLMs hallucinate high-level reasoning as well.
Next, I don't understand what is the source of your statement that LLMs can not follow rules. I didn't try this, so I may be wrong, but I am pretty sure you can make up your own game, describe the rules, and LLM would play it just fine. There maybe problems with far context, for example it definitely can make errors in a chess game just because there are too many moves done and the LLM doesn't see the board in real time, it only sees the moves. Text-based chess is crazy difficult for LLM because of how context works. But if you somehow hack this, for example write the state of the board on every move, it would be much easier and LLM would do no mistakes.
BTW just tried making deepseek sum two 25-digits numbers, it did it correctly step-by-step. Did it do it with statistical pattern matching? Probably. But does it mean it is any different then human?
> You are wrong about humans able to explain their reasoning.
This is provably incorrect. As demonstrated by reasoning method transference and replicability. Are there components of reasoning that can't be articulated, sure but they are irrelevant to the point. What matters is that we have enough facets of reasoning that can be extracted into definable processes that we can build the modern civilization.
> you cannot prove LLMs hallucinate high-level reasoning as well
It is quite an extraordinary leap to consider high-level reasoning is accurate while low-level is not. If LLMs could understand the foundational principles of concepts it should be able to construct the low-level steps. Nonetheless, there is no argument to be made here. When you have a known architecture, all behavior should be explained through that architecture. Reaching for explanations outside of that known design has no basis. There would be no need to keep feeding the machines the entire world of information if reasoning were occurring.
> what is the source of your statement that LLMs can not follow rules.
LLMs are trained on data. In order to play chess, they must have millions of examples in their training set. They don't understand rules. You can't remove the training set and then have them play a game of chess. It is why they aren't good at code for new technologies.
> It is quite an extraordinary leap to consider high-level reasoning is accurate while low-level is not.
This is the case for humans, why it shouldn't be for LLMs? I am not talking about "components that can't be articulated". Humans are bad in explaining their logical reasoning, and hallucinate that a lot. Most of even the pretty difficult decisions are made by heuristics in your brain, and then you can explain that with logic, but that verbal logic actually have no connection to the heuristics that actually did the decision. Unfortunately I don't remember the source for exactly that, but as a pretty close example there are cognitive biases. Most of them are persistent even when you understand them, you can think you completely understand you reasoning and still make errors that prove you didn't actually think like that. The mechanism of such behavior is explained by Kahneman, Tversky.
I am trying to say that you can divide the reasoning into low and high levels, and in humans the border is much higher then you think. This border can be even higher in LLMs because of overfitting into a lot of data. Summing 2-digit numbers is definitely made by heuristics in your example. But this still doesn't prove they can't have higher level that is actual reasoning.
> You can't remove the training set and then have them play a game of chess.
I am not so sure about that. You can't just say that without any proofs.
> Humans are bad in explaining their logical reasoning, and hallucinate that a lot.
Agreed. But the argument that errors exist is not an argument that a true result is not possible. The human ability for self reflection is essentially the error correction protocol.
As stated in the article "While there are many human failings in the decision-making process, these don’t preclude our ability to generally articulate and understand our thoughts when we apply deliberate attention to logical tasks."
We have demonstrated this by the progression of all the things that humans can build and do so through the transfer of reasoning steps.
>> You can't remove the training set and then have them play a game of chess.
> I am not so sure about that. You can't just say that without any proofs.
It is how LLMs operate. It is the fundamental architecture. They must consume massive amounts of data to form the probabilities to construct results. If this were not the case, all of the AI labs would not have needed to consume every piece of information humans have ever written. After running out of data, now they are using synthetic data in an attempt to continue more training.
I can surely say that LLMs in fact can play a game that you describe using rules. Even if such game was not in the training data. But the more rules you have, the worse the LLM performance is. They tend to fail to follow too many instructions, especially when you try to make it into doing something that goes against usual direction of intentions. Though, the same issue applies with humans, sometimes it's very hard to make them understand and follow instructions on an unusual topic.
As for the self reflection ability, I agree with you. When I sum numbers, I use heuristics the same as LLM, but if asked, I can actually explain the exact heuristic steps I took like "35+19 should be something like 35+20. I also know that 30+20 is 50, then I add 5 and subtract 1". LLMs don't do that. Though you can make them produce something similar by asking to "calculate in your mind only without writing down things" and "describe each step as you think". I wonder how close the results will be to the real steps that they use.
> I can surely say that LLMs in fact can play a game that you describe using rules.
They can only pretend to do so. Unless the rules are incredibly simplistic without any strategy, the LLM will hallucinate and start breaking the rules.
> I wonder how close the results will be to the real steps that they use.
Anthropic revealed that they aren't close at all. How they solve a problem and the steps they claim they used do not agree. This was covered in the previous article linked at the top.
> Unless the rules are incredibly simplistic without any strategy, the LLM will hallucinate and start breaking the rules.
That's true. I believe that it might be related to the fact that LLMs are static models, meaning that they are not changing their internal state and can't develop new heuristics at the time of chatting with someone. The human ability to develop a strategy upon new rules is a consequence of their ability to learn and create new heuristics on the fly. Since LLMs are only trained by lots of examples, we are not going to see them doing "on-the-fly learning" in near future.
They are not close when you just ask how they think and they answer with the column addition bs. But if you ask like I cited above, GPT answers with something similar to my own step-by-step example which is also similar to what Anthropic described (I know that I'm talking about GPT but I assume that all LLMs do it more or less the same).
Worth reading Iain McGlichrist's The Master and His Emissary. It accounts for quite a bit of the confusion here. Some folks are truly locked in the "left hemisphere," as it were. They can't tell the difference between human intelligence and AI, likely because most of their own thinking is pretty close to AI. Those with a more rounded intelligence spot it right away and struggle to get the message across to the others. You'll notice that the defenders of AI tend to fly into rages. That's a marker of what I call leftbrainitis, as is "confabulation," ie making up stories and facts to secure being right (hallucinations?). Those with a healthy and active "right brain" can hold two contradictory notions in their minds without feeling the need to settle on a conclusion. I'm just skimming the surface here. Best to read McGilchrist.
Thank you for the name I know, without remembering if I read anything of it or intended to.
I will read a transcript of one of his lectures that I just found to find out more.
I can however say that following my previous experiences and research, it would seem that we should take into account, that the studies and conclusions, are always based on observation and an interpretation part (the latter is regrettable), because not everything is visible by instruments, sensors... however powerful they are.
This is a very interesting subject, the reading will bring, I suppose, some reflections that I may not have thought of.
Thank you for this clear and relevant development.
I hope that this will allow further reflection on the distinctions which are difficult for some to detect.
"actions that will destroy the integrity of civilization’s information repositories." A very powerful and provocative statement. Something new to add to the list of the unravelling of the Homocene. An valuable idea that has not gotten any/enough? scrutiny, heretofore. Many thanks.
Two comments:
First, it seems you might be overlooking AIs that have developed novel semantic systems, or languages, with rules we don't fully comprehend. While I wouldn’t equate this to human language creation, it appears to challenge the kind of the intelligence with which you're concerned. Therefore, it seems necessary to distinguish between AI's creation of a language (or "semantic exploration," as you term it) and our own processes in achieving the same.
Second, you seem to be overlooking Wittgenstein's account of rule following as rendered by Saul Kripke, particularly the quus dilemma. This dilemma illustrates a scenario where a rule follower, while learning addition, begins making inexplicable mistakes, consistently rendering the same incorrect result. This highlights that rule following isn't derived from singular observations but requires multiple instances to probabilistically determine the appropriate application of the rule per others' expectations for said application. For further exploration of this concept, I recommend Lorraine Daston's *Rules*, which examines how Benedictine monks utilized "prudentia" (prudence) to (perhaps probabilistically) discern rule application through repeat observation and application. Her discussion of the craft tradition in medieval Europe is good on this point also.
All of this seems to invoke Bayesianism, which I'm surprised receives no mention. So far as most psychology literature is concerned, we are purely Bayesian calculators. I do not agree with this—far from it. To refute such literature, though, requires at least a mention of the arguments that psychologists currently deploy to reduce us to calculators.
> AIs that have developed novel semantic systems, or languages, with rules we don't fully comprehend.
Can you provide a reference of such account? I'm aware of the Facebook case where it was stated two chat bots created their own language. However, that was a case of essentially using shorthand patterns. Not new semantic information.
> you seem to be overlooking Wittgenstein's account of rule following
From the premise, as I understand it, it seems to imply we can't verify that another person comprehends rules from simply data sampling. Although it may be the case we can't verify the understanding in such a testable manner. The concept is logically contradictory. The very fact that we can perceive the potential for an error outside of the stated rule conditions would imply that we comprehend the rule.
If we weren't able to do so, then we would see highly indeterminate behavior among technology created on the top of formal systems. It just simply wouldn't be possible. We wouldn't be able to build reliable interfaces between systems etc. as all such are constructed on the basis of rules or specifications.
> To refute such literature, though, requires at least a mention of the arguments that psychologists currently deploy to reduce us to calculators
Relevant, but I don't attempt to take on every argument necessarily at once. I might follow up with something on that at some point. I already struggle with getting most people to read my articles, being longer usually doesn't help :-)
You are wrong about humans able to explain their reasoning. Humans are full of heuristics and constantly hallucinate about reasons for their decisions, just like the LLM in your summation example, this is a known fact. Like 99% of your decisions are made before your inner voice explain the reasoning for them. The inner voice is actually there to explain, not to make the decisions. And it is often wrong. Unlike lower-level reasoning, humans indeed have to understand high-level reasoning in order to use it and in order to pass it, but right now you cannot prove LLMs hallucinate high-level reasoning as well.
Next, I don't understand what is the source of your statement that LLMs can not follow rules. I didn't try this, so I may be wrong, but I am pretty sure you can make up your own game, describe the rules, and LLM would play it just fine. There maybe problems with far context, for example it definitely can make errors in a chess game just because there are too many moves done and the LLM doesn't see the board in real time, it only sees the moves. Text-based chess is crazy difficult for LLM because of how context works. But if you somehow hack this, for example write the state of the board on every move, it would be much easier and LLM would do no mistakes.
BTW just tried making deepseek sum two 25-digits numbers, it did it correctly step-by-step. Did it do it with statistical pattern matching? Probably. But does it mean it is any different then human?
> You are wrong about humans able to explain their reasoning.
This is provably incorrect. As demonstrated by reasoning method transference and replicability. Are there components of reasoning that can't be articulated, sure but they are irrelevant to the point. What matters is that we have enough facets of reasoning that can be extracted into definable processes that we can build the modern civilization.
> you cannot prove LLMs hallucinate high-level reasoning as well
It is quite an extraordinary leap to consider high-level reasoning is accurate while low-level is not. If LLMs could understand the foundational principles of concepts it should be able to construct the low-level steps. Nonetheless, there is no argument to be made here. When you have a known architecture, all behavior should be explained through that architecture. Reaching for explanations outside of that known design has no basis. There would be no need to keep feeding the machines the entire world of information if reasoning were occurring.
Anthropic has shown that to the degree these models can be internally inspected, there is no true reasoning. Covered previously here - https://www.mindprison.cc/p/no-progress-toward-agi-llm-braindead-unreliable
> what is the source of your statement that LLMs can not follow rules.
LLMs are trained on data. In order to play chess, they must have millions of examples in their training set. They don't understand rules. You can't remove the training set and then have them play a game of chess. It is why they aren't good at code for new technologies.
> tried making deepseek sum two 25-digits numbers
You might want to take a look at https://www.mindprison.cc/p/why-llms-dont-ask-for-calculators
> But does it mean it is any different then human?
Yes. Covered in the first linked reference.
> It is quite an extraordinary leap to consider high-level reasoning is accurate while low-level is not.
This is the case for humans, why it shouldn't be for LLMs? I am not talking about "components that can't be articulated". Humans are bad in explaining their logical reasoning, and hallucinate that a lot. Most of even the pretty difficult decisions are made by heuristics in your brain, and then you can explain that with logic, but that verbal logic actually have no connection to the heuristics that actually did the decision. Unfortunately I don't remember the source for exactly that, but as a pretty close example there are cognitive biases. Most of them are persistent even when you understand them, you can think you completely understand you reasoning and still make errors that prove you didn't actually think like that. The mechanism of such behavior is explained by Kahneman, Tversky.
I am trying to say that you can divide the reasoning into low and high levels, and in humans the border is much higher then you think. This border can be even higher in LLMs because of overfitting into a lot of data. Summing 2-digit numbers is definitely made by heuristics in your example. But this still doesn't prove they can't have higher level that is actual reasoning.
> You can't remove the training set and then have them play a game of chess.
I am not so sure about that. You can't just say that without any proofs.
> Humans are bad in explaining their logical reasoning, and hallucinate that a lot.
Agreed. But the argument that errors exist is not an argument that a true result is not possible. The human ability for self reflection is essentially the error correction protocol.
As stated in the article "While there are many human failings in the decision-making process, these don’t preclude our ability to generally articulate and understand our thoughts when we apply deliberate attention to logical tasks."
We have demonstrated this by the progression of all the things that humans can build and do so through the transfer of reasoning steps.
>> You can't remove the training set and then have them play a game of chess.
> I am not so sure about that. You can't just say that without any proofs.
It is how LLMs operate. It is the fundamental architecture. They must consume massive amounts of data to form the probabilities to construct results. If this were not the case, all of the AI labs would not have needed to consume every piece of information humans have ever written. After running out of data, now they are using synthetic data in an attempt to continue more training.
I can surely say that LLMs in fact can play a game that you describe using rules. Even if such game was not in the training data. But the more rules you have, the worse the LLM performance is. They tend to fail to follow too many instructions, especially when you try to make it into doing something that goes against usual direction of intentions. Though, the same issue applies with humans, sometimes it's very hard to make them understand and follow instructions on an unusual topic.
As for the self reflection ability, I agree with you. When I sum numbers, I use heuristics the same as LLM, but if asked, I can actually explain the exact heuristic steps I took like "35+19 should be something like 35+20. I also know that 30+20 is 50, then I add 5 and subtract 1". LLMs don't do that. Though you can make them produce something similar by asking to "calculate in your mind only without writing down things" and "describe each step as you think". I wonder how close the results will be to the real steps that they use.
> I can surely say that LLMs in fact can play a game that you describe using rules.
They can only pretend to do so. Unless the rules are incredibly simplistic without any strategy, the LLM will hallucinate and start breaking the rules.
> I wonder how close the results will be to the real steps that they use.
Anthropic revealed that they aren't close at all. How they solve a problem and the steps they claim they used do not agree. This was covered in the previous article linked at the top.
> Unless the rules are incredibly simplistic without any strategy, the LLM will hallucinate and start breaking the rules.
That's true. I believe that it might be related to the fact that LLMs are static models, meaning that they are not changing their internal state and can't develop new heuristics at the time of chatting with someone. The human ability to develop a strategy upon new rules is a consequence of their ability to learn and create new heuristics on the fly. Since LLMs are only trained by lots of examples, we are not going to see them doing "on-the-fly learning" in near future.
They are not close when you just ask how they think and they answer with the column addition bs. But if you ask like I cited above, GPT answers with something similar to my own step-by-step example which is also similar to what Anthropic described (I know that I'm talking about GPT but I assume that all LLMs do it more or less the same).