AI Defies Description - What Will Be The Limit of Its Capability?
Notes From the Desk: No. 58 - 2026.06.20
Notes From the Desk are periodic informal posts that summarize recent topics of interest or other brief notable commentary.
Can Anyone Make Sense of It All?
Every prediction continues to be wrong in both directions. We are not approaching superintelligence, but it often looks like we are. The machines are not useless, but often they appear to be. Exactly where is all of this taking us? Where will we ultimately find ourselves between despair, hope, suffering, or utopia? Exactly what do these machines do and what will they be able to do tomorrow?
What Is the Theoretical Limit of What This Technology Can Do?
Our current AI works somewhat like an instructible copy machine. It can take all existing information and combine it based on the prompt or instructions supplied by the user.
As the models grow larger with more compute, the fidelity of the copies increases. The level of detail and the depth with which patterns are created and reassembled improves. With enough data and compute, AI can recreate variations of nearly anything.
And since the entire world of data is so vast, beyond any single individual’s ability to perceive it all, many things AI “creates” will appear as novel to many people. None of us know all the things that exist in the world, and therefore we cannot accurately perceive if something is new or merely a derivative of what already exists.
It Is by Default Very Good and Very Average
As we build ever larger models, the default output will continue to be better. So, what is the limit? The limit is functional permutations of larger and larger things that already exist. What does that mean?
It means if someone has built it before, AI can probably make some variation of that thing that is functionally equivalent, which would include large, complex applications, entire websites, or other complete bodies of knowledge.
However, there is a tradeoff. With a short, ambiguous prompt, the result is going to be the average of everything. For example, “make a professional image editor” would likely result in a very Photoshop-like result. Everyone else that attempts to create an image editor would get a very similar result. It can be a very good result, but still constrained by what already exists: the training data.
Average Is Not Desirable Even if Good
Creating things that are mostly like what everyone else is creating or have already created will have little value. Many AI enthusiasts dream of that next model that can one-shot prompt the next great wealth-generating application.
However, it is a paradox, because once that capability arrives, that capability is then worthless. We cannot escape a basic tenet of reality that dictates effort is a component of value, no matter how much utopian dreamers imagine it should not be.
Obtaining a Result That Is Not Average
We can push the model to move away from the average bias to the degree we specifically instruct the model to do something that is not average. The further we wish to move away from average and toward novel output, the greater the level of detail of instruction and information that must be provided to the model by the prompt.
However, there are tradeoffs with increasing prompt complexity and detail. Clearly, the most obvious is that this requires more effort. Most importantly, whatever ingenuity that will comprise the output must come from the prompt, as all of the other data which is part of the computed output is a statistical average of the training data.
Furthermore, any detail not specified within the prompt will be filled in with whatever is the average of data that comprises that portion of the output. In some cases this may be acceptable, but nonetheless, the more unique we wish for the details to be, the more verbose the prompt will need to be in order to describe that level of detail.
As we move further away from the distribution of data, the more information and instruction we must provide within the prompt. Not only do we need a greater level of detail in the prompt to avoid average output, but also to avoid undesirable hallucinations that may create erroneous outputs.
The Value of Things Becomes Obscured
All of this leads us to a place where discernment for the value of things and how they are constructed becomes increasingly difficult. AI will continue to make seemingly more impressive outputs, as the average of everything, for which we ourselves are not experts, is impressive to our eyes.
But there will be little distinction, for most of us, between the average and the long journey of building something with great effort. Typically, things that had the initial impression of being impressive were more likely so. There were signals we often utilized to make an initial assessment for the value of something.
For example, intelligent-sounding, articulate language was often a clue that text had been written by someone who put a lot of thought and effort into its construction. But, this has been trained to be the average default output of LLMs. Now even the most meaningless drivel is shrouded in a pseudo-intellectual, philosophical masquerade.
As it becomes overused, the patterns do become more obvious. Nonetheless, it still takes more of your time to discern. You need to examine it more closely. The level of scrutiny required to know for certain if an image is AI-generated is now very high for many image generations. All of this consumes more of your precious time.
Once You Build It, They Can Too
The incentives for long term investments are becoming less clear. Although AI cannot easily construct anything truly novel, it is rather efficient at duplicating things that already exist. Either the thing you create ends up in the training data, or AI agents can simply iterate over your work until they make an “original” duplicate, as such capability was demonstrated for the Bun conversion from Zig to Rust.
What does competition look like going forward? Is everyone standing on the line just waiting for the next person to take a step forward, and then the rest race to be the first to copy it and mass produce it?
And the AI labs are not immune to the facet either. At heart, AI is predominantly about reusing whatever exists. There is no escape from this for the builders of AI. Once a new model is released, the copying quickly begins. How long can everyone sustain this activity before the economics completely break?
Mimicry and Attention
If novelty and meritocracy are dead in the age of magic, where we can just recreate anything that already existed prior, and the things of true value remain obscured behind the avalanche of recycled ideas with shiny new skins, then the rewards no longer go to the things of true value, but instead to mediocrity highly optimized to capture attention.
AI capability is so difficult to define because it exists within a duality where it seemingly can do anything, but that anything also has no value. In order for it to have value, significant effort must exist to push beyond the model’s average bias, but then we immediately lose its value if it can be absorbed into the next training data or can easily be examined and deconstructed by AI agents for reassembly.
But we are speaking about the limits of AI capability. The obstacle between what exists now and AI that can copy and reproduce anything is still massive. That obstacle remains economically sustainable training and inference that has to continue to exponentially scale, and at present, still is not profitable.
But the trajectory is clear for as long as we can maintain it. That is, value continues to shift toward attention, and yes, that was already occurring prior to AI, but AI is the machine that massively accelerates it, as it is optimized for this capability.
Mind Prison is an oasis for human thought, attempting to survive amidst the dead internet. I typically spend hours to days on articles, including creating the illustrations for each. I hope if you find them valuable and you still appreciate the creations from human beings, you will consider subscribing. Thank you!
No compass through the dark exists without hope of reaching the other side and the belief that it matters …