Problem arguments from Yudkowsky and John Carmack
Notes From the Desk: No. 10 - 2023.10.03
Notes From the Desk are periodic posts that summarize recent topics of interest or other brief notable commentary that might otherwise be a tweet or note.
Yudkowsky and the missing alignment goal
Eliezer Yudkowsky, prominent AI researcher focused on AI alignment safety, posted this on X.
“To the realms of science fiction overrun by science fact we must now consign all stories where AIs stay at the same intelligence level throughout the whole book, instead of AIs getting visibly smarter every 4 months like they do in real life.”
This view of science fiction by Yudkowsky is wrong because we have not included stable intelligence as part of alignment. If AI is alignable, then we have to assume within our sci-fi worlds that the AIs are aligned.
The reason Yudkowsky concluded the above is that, as far as I’m aware, nobody as of yet has stated alignment must include limitations on intelligence. It is often stated in the very premise of the need for alignment. That being that we need alignment before the AI begins exponential self-improvement so that it will already be aligned before it is far too intelligent for us to comprehend and control.
However, runaway intelligence is just another version of the AI paperclip scenario. Exponential infinite scaling intelligence must also consume resources. Such as power and materials etc. If there are no hard limits, then the AI would consume everything in its growth. Therefore, if alignment is possible, then the intelligence ramp would be controlled.
John Carmack says AI catastrophic disasters are impossible
To answer the question directly: there is no way, even with an infinitely smart (but not supernatural) oracle, that a teenager can destroy all of humanity in six months.
Even the most devastating disasters people imagine will not kill every human being even over the space of years. This is a common failure of imaginative scope, not understanding all the diverse situations that people are already in, and how they will respond to disasters.
You could walk it back to “kill a million people” and get into the realm of an actual discussion, but it isn’t one I am very interested in having.
This is nonsensical. This evaluation implies from our vantage point of “infinite stupidity”, as would be the case attempting to reason about “infinite intelligence”, somehow we have a certainty about the outcome.
I do agree that it is impossible to conceive of how humanity adapts to a threat and mitigation measures that might arise. Nonetheless, primitive humans wiped out more primitive species a very long time ago. It doesn’t seem inconceivable it could not repeat when there is an order of magnitude intelligence and capability gap.
The best conceptual argument would be that AI could be a defense against other AI. However, that premise isn’t getting off to a great start yet as web security company Immunefi has stated there hasn’t been much luck with using AI as a defense.
We wish AI to solve the impossible problems of thought for which we are not certain. Immense capability will provide answers to all destinations. Seems to be a logical contradiction to be certain of any of them.
Living in the virtual world. Caprica nears.
This is a discussion on X that was prompted by Meta’s, Google’s and Apple’s capabilities of building life like virtual 3D avatars for immersion in virtual environments.
“ … in a few years, we'll be embodied in a photorealistic digital avatar almost indistinguishable from reality. You can be anywhere, with anyone, doing anything - restrictions of time and space will shatter.
*as long as you're compliant with the terms of service and community guidelines ;)”
[ could this lead to dystopia]
“It could but it doesn't have to. It's incumbent on us in this nascent stage to develop this tech in ways that ensure it enhances our experience of the real world, not just provides an escape from it.”
From X thread
It is unavoidable, dystopian outcomes at minimum in some part. It is the duality of immense capability.
Just as social media itself led to many new opportunities it also has brought about the ability for social engineering of society and disturbing behaviors brought about by engagement addiction.
Few foresaw these outcomes beforehand and the next era will bring about significantly more surprises, but they will drift further towards extremes. Significantly good and significantly bad.
You can't get better than the real world so to speak. The real world operates at the limit of physics. Virtual will always be a simulation of that no matter how good it gets. We are caught somewhat in a cycle of circular reasoning attempting to build machines to simulate what we already experience.
All of this marvelous capability often looks enticing from our point of view, but it is likely an illusion of progress. Hedonic adaptation suggests that no matter how great the new environment we will return to the same level of happiness, especially when it is mostly externally motivated.
If this is true, then energy and effort are misplaced as this cycle will compress. New technology comes more rapidly and we seek evermore advancements, but never enough time to adapt. That leaves us in a constant state of desire without ever experiencing satisfaction. A vicious cycle.
But those problems are just the ones for which we will volunteer. Far more problems exist for the state of free societies that become ever less free as no privacy of mind exists any longer going forward.
I don't think we will escape the paradox of Utopias. All attempts to build them result in dystopia.
No compass through the dark exists without hope of reaching the other side and the belief that it matters …
