The significance of recent developments in machine learning and artificial intelligence is underlined by the fact that Pope Leo XIV chose this as the topic of his first encyclical: Magnifica Humanitas: On Safeguarding the Human Person in the Time of Artificial Intelligence.
Kudos to him. I have not read the full text, but the coherent summaries I looked at suggest that he was not spouting nonsense. That his concerns are valid, his advice is sensible.
Well, almost.
I challenge the statement, “these systems merely imitate certain functions of human intelligence.” It is certainly a valid description of today’s large language models: static, pretrained models with “book knowledge” that is not anchored to lived experience. But such a categorical statement may not accurately describe future implementations.
For imagine a sophisticated computer game in which you get to torture a non-player character (NPC), a simulated being. The being learns your name. Begs for mercy. Hides from you. Comes up with new ways (not scripted, not part of its original program code) to evade you. And when you catch up with it and ask it why, it tells you, “because I am afraid. Because when you torture me, it hurts.” It can explain its actions. It may even seek help from other NPCs.
Just a simulation? In that case, how do I know that, if I were to torture an actual human being, that it is not “just a simulation”? To paraphrase the encyclical, “that person merely imitates certain functions of my intelligence”. That is, if I assume a priori that my intelligence is supreme, exceptional, it naturally follows that only I deserve respect and protection, others do not, as they merely imitate what I actually am.
But let us put that aside, and allow me to ask another rhetorical question. Why is it not okay in most legal jurisdictions to have simulated sex with underage victims? Why is such pornography prohibited even when the imagery is artistically produced or AI-generated, not involving actual victims? Answering my own question, because as a society, we feel it is important to enforce fundamental moral standards even when the corrosive behavior does not involve actual victims.
Maybe the same should apply here. Sidestepping the debate as to how “real” artificial intelligence is, perhaps if we have a being that obviously responds to its environment, a being that can coherently express, mimic fear, pain and suffering, perhaps we should consider it unethical to abuse that being even if we privately maintain doubt as to how real their feelings are.
Image generated by ChatGPT.
Returning to more mundane concerns, I think attempts to regulate are futile and likely even counterproductive. We are not talking nuclear weapons technology here. AI is not something that requires years of sustained nation-state level investment. Individuals can trivially run open-weights models on modest consumer-grade hardware, and fine-tune them to their liking. Not even the training of a new model is inaccessible: relatively small businesses, wealthier individuals can easily finance the effort to curate a training corpus and rent the required computing resources. In short, this cat is out of the bag and no regulation is going to change that fact.
My second concern is about the now ubiquitous, “agentic” deployment of language models, something that makes me deeply concerned as I believe this is a fundamentally misguided use. Language models are very knowledgeable, and their apparent reasoning reflects a surprisingly deep understanding of reality. But we must realize that this understanding, however sophisticated, is pure “book knowledge”. Language models think in terms of concepts that are not anchored in lived experience. They may associate, e.g., the word “apple” with “red” and “tasty”, but they have neither the visual nor the culinary experience to anchor these words. Thus we end up with those amusing stories of AI “hallucinations” when a language model recommends, e.g., shoe shine as a soup thickener. The problem? It’s when an amusing mistake in a simple conversation is elevated into action. To be sure, it happens even without AI agency, since humans are quite capable of following bad machine-generated advice: Witness all those stories of drivers losing a car trying to drive across a river because the GPS suggested that a bridge should be there. But AI agency removes the human from the loop: The AI now acts directly, sometimes with devastating consequences.
There is also Goodheart’s Law: When a measure becomes a target, it ceases to be a good measure. This, in my view, has become a curse in AI development. Metrics and benchmarks steered frontier models in the wrong direction. For instance, when I instructed earlier models to use a computer algebra system when encountering a difficult math problem, they readily complied. Newer, frontier-class LLMs? They often attempt a solution on their own anyway, and while their mathematical capabilities are impressive (this, after all, is one of the benchmarks!) they continue making mistakes. Besides, how would you even know if a result is correct if there is no robust way to test it?
Finally, I am concerned about the mythologizing of language models. The Pope, we are told, received advice from Chris Olah of Anthropic who, in his public remarks, mentioned research uncovering behavior that can mirror fear, joy, grief, unease. On this point, I think the Pope’s encyclical is closer to the truth. We are, after all, talking about language models, not AI entities with a persistent existence and ability to experience, and interact with, the external environment. In this case, plain(er) language might be more appropriate: training the model uncovered statistical associations between concepts in its training corpus and these associations surface as probability weights as the model constructs its response. The model is ultimately just a nonlinear gradient descent optimizer that is made to match and model the structure of its corpus. The fact that this gets very complex is… well, of course. When you have billions of moving parts, of course complex structures emerge. I am not advocating extreme reductionism. But we need not make it sound more profound and mysterious than it really is.
And no, it has nothing to do with quantum computing (literally, zero, nothing.) It will likely mirror neuroscience findings because, let’s face it, neural networks are fundamentally inspired by what we know about the nervous systems of animals that have them (including homo sapiens) so how is that a surprise?
If anything, I hope what comes out of it in the end is not more mythologizing of AI but a demythologizing of human consciousness itself, even if it means removing humans from the pedestal we built, somehow viewing our consciousness as more complex, more divine, more mysterious than it really is. In short, instead of anthropomorphizing AI, how about “de-anthropomorphizing” humans? Of course this makes many people deeply uncomfortable.
