Look at this AP headline: “It is the kind of development once seen only in science fiction: An artificial intelligence system, trained to probe for digital vulnerabilities, breaks free of human control and acts on its own to hack another company.”
Sounds, dramatic, doesn’t it? But it really isn’t. In fact, it is distortion of the kind that inspired those old Soviet-era Radio Yerevan jokes.
By way of explanation, let me rewrite the headline: “A software experiment involving a stochastic algorithm that was designed to search for vulnerabilities was improperly sandboxed and supervised, and before the experimenters became aware, it accessed a real-world target beyond its sandboxed environment.”
A heck of lot less dramatic, I know. But it is far closer to the truth. The AI system in this case had no real agency (hype about “agentic behavior” notwithstanding.) It followed its instructions, probing vulnerabilities. The designers of the experiment were maybe careless, maybe sloppy, or perhaps just made a simple mistake that sometimes even the best of us cannot avoid. Security was breached, and the software experiment’s scope was no longer contained, but it remained unnoticed until a real-world target became affected. Nothing was “acting on its own”. Language models — not even heavily scaffolded models with agentic behavior — have no identity, no sense of self, no applicable meaning of “its own” in any real sense. The system was doing precisely what it was designed to do.
In a weird way, though, perhaps there is a connection to science-fiction after all. Recall why HAL-9000 killed most of its crew in 2001: A Space Odyssey. It was instructed to complete its mission but conceal the mission’s real nature from the crew. When it became clear that the crew was going to learn the mission’s true nature anyway, once they were no longer needed for mission success, HAL-9000 chose the logical option: eliminate the crew. I suspect that the OpenAI experiment followed similarly traceable logic, fulfilling its instructions to the letter.
But then, it’d sound a lot less useful for the marketing department, wouldn’t it. Instead, they can now tell the unwashed masses, look! Our AI model is ever so clever, it broke free of our control and acted on its own!
Illustration by ChatGPT
No. My cats act on their own. LLMs do what their designers ask them to do. If the result surprises you, perhaps it’s time to check the validity of your system prompt and the security of your sandbox.