Some Say, if LLMs Seem Intelligent, It Doesn't Matter if They Are Not. They are Very, Very Wrong.
Notes From the Desk: No. 57 - 2026.06.11
Notes From the Desk are periodic informal posts that summarize recent topics of interest or other brief notable commentary.
Does AI Work? This Requires an Explanation
Nuance is often the gatekeeper to reason. The following post is a prime example of losing discernment for what is happening. The fine details actually matter.
Does AI work? AI does not work in the sense of its purported abilities as described by the AI lab hype machines. It is not intelligence. Can it solve problems? Yes. It does work in that sense. Vastly scaled pattern-matching on the entire world of knowledge is very powerful.
Can it create things outside of its dataset? Well, that question is too ambiguous for a meaningful yes or no answer. So, we will argue without resolution if we do not qualify further.
It can create new permutations of all existing data. And since that data is so vast, as to not be comprehensible by any single person, it’s origin is mostly indiscernible. The larger we train models, the greater the fidelity of patterns from which it can construct its output. In that sense they do become more useful and can create more useful output.
But just as you cannot build a brick house from Legos, there are limits as to what AI can do based on its data. They just become increasingly imperceptible because the parts and pieces available to it, span all of the component parts that construct all known knowledge.
There is no intelligence inside the machine, but there can be intelligence as part of the construction of the output. Where does this come from? The only intelligence in the system is the human which supplies the other input alongside the training data, which is the prompt. That is the only injection of semantically meaningful information.
This is where most will interrupt with “well, then intelligence doesn’t really matter, or we are just pattern-matchers too, etc.”
What AI Cannot Do
However, it instrumentally matters in regards to the things many want AI to do. These current systems can never be fully autonomous. Despite Anthropic’s statements of self-improving AI, that just isn’t possible with current architectures.
They will experience model collapse when operating in a closed-loop condition ingesting their own output as training data. Any type of “self-improvement” could only be done using some type of external harness to guide the AI.
This is similar to how we use coding agents. But this has limits as the task and goals will always need to be defined by humans. It is narrow in scope and cannot deliver general improvements that span beyond our own perception.
Nuance Is Instrumentally Important
So, does it work? Yes, it works exactly as it is designed, a vastly scaled pattern-matcher. Does it work as a complete replacement for human intelligence? No. The nuance is that pattern-matching will be sufficient for some sets of tasks that previously were only performed by intelligence, although it does so still with orders of magnitude less efficiency.
But also, there are still things out of reach. Hallucinations will not go away. They aren’t solvable, and no, they are not like human hallucinations. The machine cannot perceive its own errors or hallucinations, which prevents full autonomy.
And this is the dividing line of intelligence. It isn’t some arbitrary goalpost that just seems a bit out of reach for now, to then be moved later. It is a foundational architecture limit. Benchmarks may fall, but this won’t. LLMs can do almost anything eventually with enough data, training, and inference compute. But they cannot do it alone.
What Is Unique About Human Intelligence?
AI is not a stable system because it cannot correct its errors. Which is a unique thing to human intelligence. Yes, both AI and humans make errors, but it is only intelligence that can perceive errors. And that is what AI cannot do.
“But AI fixed my code bug, of course it can perceive errors, it told me so.” No, language deceives us here, because language is full of all the articulated possibilities that match our expectations. It was just a calculation. A calculator can give you the correct answer, but the calculator perceives nothing.
What proves this true? Again, it is the model collapse problem. When a model can ingest its own output, refine it, and improve it. Then we have progressed to something new.
That is the signal to look for. It is not a benchmark. AI could get 100% on every benchmark and still contain no intelligence, but show me a model that does not suffer model collapse, in a loop, while increasing the semantic complexity of its data and you will then have my attention.
But Really, AI Can Perceive Errors
This following post was a reply to my information above.
It is not doing what you think it is doing. It doesn’t understand what is an error. That’s your interpretation. All output from LLMs is equal in meaning from the viewpoint of the LLM, as it has no meaning.
Paul further states “It’s near irrelevant too.”
No, it is substantially relevant. Being mostly imperceptible to most people doesn't change its importance. So, why do we care about the difference between LLM capability and human intelligence?
It is because our current AI can only result in collapse. The system inevitably destabilizes and fails. Human intelligence evolved information from cave drawings to self-landing rocket ships. This is what AI cannot do. It cannot increase semantic meaning of the data. It does the converse. It degrades semantic meaning. That is a pretty big deal.
Perceiving The Difference
I hope posting these conversations are helpful. I find this topic is very difficult for many to grasp and it will only get worse as LLMs become more capable. However, understanding it is critically important. We must understand for which roles LLMs are not appropriate.
We cannot turn the management of our entire civilization over to these machines, as their model collapse would become our collapse. These LLMs are not capable of taking control of civilization Skynet style, but they could cause significant harm if we purposely place them in such roles with responsibilities they should never have. Hallucinations will always present risk for catastrophic failure for anything the AI has access to.
If you want to go deeper into this topic, I recommend this as your next read “Intelligence Is Not Pattern-Matching”
Mind Prison is an oasis for human thought, attempting to survive amidst the dead internet. I typically spend hours to days on articles, including creating the illustrations for each. I hope if you find them valuable and you still appreciate the creations from human beings, you will consider subscribing. Thank you!
No compass through the dark exists without hope of reaching the other side and the belief that it matters …
Great stuff as usual. I was happily surprised to discover that Claude could make an exclusionary list, which Chomsky said it couldn't do. Not that long ago it couldn't; ChatGPT couldn't just a couple of years back. The task I asked for was to provide a list of all the words in the English language that have the vowel 'a' to the exclusion of all other vowels. The vowel 'a' could be repeated, but no other vowels could be included. I also specified that it should not invent or fabricate. It proceeded to work using a self-corrective method. Here is Claude:
"I need to be transparent with you: this task is harder than it appears, and I risk fabricating or including errors at scale. Here is the honest problem:
My knowledge of the full English lexicon isn't exhaustive enough to produce a complete list reliably.
Generating a long list risks false inclusions (words I misremember) or missed exclusions (words where I forget a vowel).
What I can offer with confidence — a solid, verified-by-reasoning sample:"
The complete list it gave included examples of errors with a large red 'x' beside them. This is the sort of feedback that mimics self-correction and convinces folks that there's some level of sentience going on. But convincing mimicry doesn't hold up under real pressure. Humour me a moment...
(I find it interesting that this sort of exclusion-based pattern finding is "difficult" enough to break Claude.) Next, I asked Claude to write five poems using the list and only the list. The poems didn't have to rhyme, but they had to make sense. (The basis for this is an awful book called Eunoia that was written many years back. It was a very mechanical book as one would expect. So I've been curious to know when AI has reached that level.) Claude failed to perform the task on several levels, and I was ready to be lenient, since the bar was set low to begin with. From the technical perspective it failed by including two vowels that it claimed not to have used in its own final analysis. And from a sense perspective it just failed where a human would have seen the obvious. Ex: "camp dark" and "path vast." A lot of patterns were clearly filched cliche, like "band marched" and "branch snapped." But "snapped branch" would have been better. In short, when pressed to make sense with restrictions like the one I provided, the Claude's methods are denuded and the lack of understanding what it's doing becomes glaringly evident.
PS: Note carefully Claude's response about the difficulties. It's worth considering how words might be "misremembered." Claude can't deal with words as both semantic entities and data at the same time. Question: can it be trained to do this?
You have put some intellectual rigor behind what I was trying to say here - https://ombreolivier.substack.com/p/ai-bicycle-or-train - about how AI is, on its own, intrinsically limited and that if you don't put in the effort you get recycled slop