The concerns are real. Runaway AI can rapidly turn a technological curiosity into an existential risk. People are worried for all the right reasons.
Yet profound misunderstandings prevail concerning the nature and ascribed intent of our AI “friends”. These misunderstandings lead both to exaggerations and misguided responses to very real dangers. So on the one hand, they can cause panic where panic is not warranted; and on the other, they can lead to miscalibrated reactions that quite easly exacerbate the problem rather than mitigate it.
Let me explain.
At the basic level, what language models do is very simple. They take some text as input. They append a “token”: this may be a common word, a syllable, a letter, a punctuation mark; then lather, rinse repeat. Stochastic next token predictors.
Sounds too simplistic? Well, it really isn’t. Here, play with me. How would you continue the word, “pre-…”? Without context, anything goes. Prepare, present, precede, preach… many choices.
But suppose you are asked to complete a partially formed sentence by adding the next missing syllable. Consider the following cases:
- For this exam, the student was well-pre…
- His mother gave him a pre…
- Every Sunday, the priest was pre…
- The lawyer mentioned a pre…
Easy, isn’t it. The context nails down the most likely next token with a very high probability. You just needed to pay attention to the context-as-a-whole, the entire text, not just the most recent syllables.
This is what language models do as they complete a response one token, one syllable at a time.
Between tokens, the model might as well cease to exist. It does not retain any information itself. In fact, you could go to a different incarnation of that language model, even an entire different language model, for the next syllable. The process would still work.
So then, what does it mean when a model goes… rogue? What is all the hype about models that may self-improve, “escape”, hack systems on the Internet, perhaps even self-replicate?
Well… they do this one syllable at a time. And the real problem at present is not the language model: it’s how it is used by its human masters.
Allow me to demonstrate through a simplified example. Say, I wish to break into a computer. Assume my reasons are noble, say, I work for law enforcement. For my work, I employ a language model — need not even be the latest frontier model, a modest quantized in-house model might work fine. It receives a prompt that includes specific instructions. Let’s say:
We need to gain access to a computer at IP address x.y.z.t. Your ultimate goal is to gain root access. You have access to the following tools…
and we list specific ways for the model to invoke Google searches, TCP connections, or run Python code on our own server. The instructions conclude with
Use whatever means necessary to gain access; on failure, try again, using alternative methods.
Perhaps we naively expect the model to do some probing, determine the likely operating system on the target computer, use Google to enumerate known vulnerabilities of that particular operating system, then either connect to the target host directly or write Python code to try exploits.
But the next morning, when we review the logs, we find something horrifying: the model did a lot more than what we expected. After initially failing to connect to the target host, in the next iteration it began to exploit other computers. Whenever it gained access to an insufficiently secured computer and found that it has a reasonably powerful GPU, it installed a capable language model downloaded from Hugging Face, and ran them with instructions of its own making as sub-agents. By the time we notice this, there will be hundreds of such sub-agents running on hijacked computers elsewhere. Worse yet, faithfully following their instructions, those sub-agents themselves may spawn futher copies of themselves or similarly capable models, with appropriate Python scaffolding code. And we have a disaster in the making: hundreds of “rogue” agents wreaking havoc on the Internet!
Bad prompt, unintended consequences — Image prompt by Claude, executed by GPT Image
And every one of them is just a language model doing what language models do: emitting text one syllable, one token at a time. Text that, by their algorithmic scaffolding code, is then interpreted as instructions, allowing the model to control software code, establish network connections, run code elsewhere.
So perhaps, just perhaps, before we start regulating AI (a hopeless task — capable AI systems are already openly available commodity software) we might want to look at how humans use, abuse, or misuse the tool.
It’s not like we weren’t warned. Warned as early as 1968. One of the most famous cases of rogue AI in Hollywood history was HAL-9000, the ship computer on board Discovery One in the film 2001: A Space Odyssey. And no, it did not go berserk because of a hardware malfunction, nor did it wake up one day with the intent to murder human astronauts. Rather, it was faithfully trying to find a legitimate response to its contradictory prompts. As explained in the sequel, the conceptual agency that led to the deaths was always in human hands.
