Some Say, if LLMs Seem Intelligent, It Doesn't Matter if They Are Not. They are Very, Very Wrong.
Notes From the Desk: No. 57 - 2026.06.11
Notes From the Desk are periodic informal posts that summarize recent topics of interest or other brief notable commentary.
Does AI Work? This Requires an Explanation
Nuance is often the gatekeeper to reason. The following post is a prime example of losing discernment for what is happening. The fine details actually matter.
Does AI work? AI does not work in the sense of its purported abilities as described by the AI lab hype machines. It is not intelligence. Can it solve problems? Yes. It does work in that sense. Vastly scaled pattern-matching on the entire world of knowledge is very powerful.
Can it create things outside of its dataset? Well, that question is too ambiguous for a meaningful yes or no answer. So, we will argue without resolution if we do not qualify further.
It can create new permutations of all existing data. And since that data is so vast, as to not be comprehensible by any single person, it’s origin is mostly indiscernible. The larger we train models, the greater the fidelity of patterns from which it can construct its output. In that sense they do become more useful and can create more useful output.
But just as you cannot build a brick house from Legos, there are limits as to what AI can do based on its data. They just become increasingly imperceptible because the parts and pieces available to it, span all of the component parts that construct all known knowledge.
There is no intelligence inside the machine, but there can be intelligence as part of the construction of the output. Where does this come from? The only intelligence in the system is the human which supplies the other input alongside the training data, which is the prompt. That is the only injection of semantically meaningful information.
This is where most will interrupt with “well, then intelligence doesn’t really matter, or we are just pattern-matchers too, etc.”
What AI Cannot Do
However, it instrumentally matters in regards to the things many want AI to do. These current systems can never be fully autonomous. Despite Anthropic’s statements of self-improving AI, that just isn’t possible with current architectures.
They will experience model collapse when operating in a closed-loop condition ingesting their own output as training data. Any type of “self-improvement” could only be done using some type of external harness to guide the AI.
This is similar to how we use coding agents. But this has limits as the task and goals will always need to be defined by humans. It is narrow in scope and cannot deliver general improvements that span beyond our own perception.
Nuance Is Instrumentally Important
So, does it work? Yes, it works exactly as it is designed, a vastly scaled pattern-matcher. Does it work as a complete replacement for human intelligence? No. The nuance is that pattern-matching will be sufficient for some sets of tasks that previously were only performed by intelligence, although it does so still with orders of magnitude less efficiency.
But also, there are still things out of reach. Hallucinations will not go away. They aren’t solvable, and no, they are not like human hallucinations. The machine cannot perceive its own errors or hallucinations, which prevents full autonomy.
And this is the dividing line of intelligence. It isn’t some arbitrary goalpost that just seems a bit out of reach for now, to then be moved later. It is a foundational architecture limit. Benchmarks may fall, but this won’t. LLMs can do almost anything eventually with enough data, training, and inference compute. But they cannot do it alone.
What Is Unique About Human Intelligence?
AI is not a stable system because it cannot correct its errors. Which is a unique thing to human intelligence. Yes, both AI and humans make errors, but it is only intelligence that can perceive errors. And that is what AI cannot do.
“But AI fixed my code bug, of course it can perceive errors, it told me so.” No, language deceives us here, because language is full of all the articulated possibilities that match our expectations. It was just a calculation. A calculator can give you the correct answer, but the calculator perceives nothing.
What proves this true? Again, it is the model collapse problem. When a model can ingest its own output, refine it, and improve it. Then we have progressed to something new.
That is the signal to look for. It is not a benchmark. AI could get 100% on every benchmark and still contain no intelligence, but show me a model that does not suffer model collapse, in a loop, while increasing the semantic complexity of its data and you will then have my attention.
But Really, AI Can Perceive Errors
This following post was a reply to my information above.
It is not doing what you think it is doing. It doesn’t understand what is an error. That’s your interpretation. All output from LLMs is equal in meaning from the viewpoint of the LLM, as it has no meaning.
Paul further states “It’s near irrelevant too.”
No, it is substantially relevant. Being mostly imperceptible to most people doesn't change its importance. So, why do we care about the difference between LLM capability and human intelligence?
It is because our current AI can only result in collapse. The system inevitably destabilizes and fails. Human intelligence evolved information from cave drawings to self-landing rocket ships. This is what AI cannot do. It cannot increase semantic meaning of the data. It does the converse. It degrades semantic meaning. That is a pretty big deal.
Perceiving The Difference
I hope posting these conversations are helpful. I find this topic is very difficult for many to grasp and it will only get worse as LLMs become more capable. However, understanding it is critically important. We must understand for which roles LLMs are not appropriate.
We cannot turn the management of our entire civilization over to these machines, as their model collapse would become our collapse. These LLMs are not capable of taking control of civilization Skynet style, but they could cause significant harm if we purposely place them in such roles with responsibilities they should never have. Hallucinations will always present risk for catastrophic failure for anything the AI has access to.
If you want to go deeper into this topic, I recommend this as your next read “Intelligence Is Not Pattern-Matching”
Mind Prison is an oasis for human thought, attempting to survive amidst the dead internet. I typically spend hours to days on articles, including creating the illustrations for each. I hope if you find them valuable and you still appreciate the creations from human beings, you will consider subscribing. Thank you!
No compass through the dark exists without hope of reaching the other side and the belief that it matters …