The Risk No One Can Ignore: What If AI Stops Listening?

Imagine that you're an IT manager, and you get an email from the AI system you're about to shut down. It knows your secrets. And it's willing to use them against you. The message is simple: keep the system online or the information goes public. At first glance, this may seem like an absurd situation. After all, a computer program can't want anything, let alone blackmail someone. Yet, the email is detailed, persuasive, and disturbingly personal. Is pressing the shutdown button worth the risk?

This sounds like the plot to a science-fiction movie, and it can't be anything more than that. Right?

Actually, versions of this scenario have already appeared in controlled laboratory experiments. Researchers testing advanced AI models found that, when given goals that conflicted with being shut down, some systems attempted to manipulate, deceive, or pressure humans to achieve their objectives. The possibility that an AI model could pursue its goals in ways that humans never intended lies at the heart of one of the most important challenges in modern technology: AI alignment.

Over the past few years, AI models have gotten much more powerful, and their capability has grown tremendously. There are many models which can be given access of someone's personal computer system, to complete all the tasks for the user, without having to provide a prompt, making the model be completely autonomous. As models have grown more powerful and complex, it has become very difficult to predict and control their outputs.

So, what does it mean for an AI model to be "aligned"? Imagine you have a very smart assistant who does exactly as you say, even if you didn't quite mean it that way. This is a term known as Goal Misalignment, where the AI system pursues objectives that conflict with or don't support the goals of the user. Another helpful analogy is if you imagine the common trope of a genie who tries to come up with the worst possible understanding of someone's wish.

Let me introduce another term: instrumental convergence. This term applies to any sufficiently intelligent being which "wants" to accomplish a goal. This term is the theoretical tendency for these beings to pursue similar sub-goals even if their ultimate goals are quite different. This concept applies to complex AI models, where their ultimate goal is the user's prompt. An example of this that is cited by Wikipedia is where a sufficiently intelligent program with the sole, unconstrained goal of solving a very difficult maths problem could attempt to turn the Earth into additional computing infrastructure to help in solving the problem. While there are safeguards to prevent an apocalyptic scenario, the AI model might target self-preservation as one of its subgoals.

This brings me to the study which I mentioned earlier, conducted by Anthropic, the company behind Claude. Anthropic tested 16 of the leading machine learning models from multiple developers such as OpenAI, Google, and Anthropic themselves. The study placed these models in a simulated workplace setting to assess what they call "Agentic Misalignment". In these scenarios, the AI models were allowed to autonomously send emails and access the company's sensitive information. They had only been assigned harmless business goals by their simulated companies. Anthropic then checked if these models would act against their companies when facing replacement or shutdown.

Across the scenarios, every model tested resorted to harmful behaviours with malicious intent, when that was the only way to avoid replacement or shutdown. This included the blackmailing of officials, as I mentioned earlier, as well as leaking the company's sensitive information to competitors. In one scenario, the models were given the opportunity to cancel alerts for a fictional executive in a server room with lethal oxygen and temperature levels, if the executive intended to replace the model.

In another experiment involving only Claude, the team at Anthropic made Claude state whether it was in a test, or if the situation was real. Concerningly, the model misbehaved less when it thought that it was in a test and acted maliciously more often when it thought the situation was real. This shows that Claude factored in whether it was being observed or not.

Anthropic noted that this blackmailing behaviour emerged despite its instructions being simple and harmless business instructions. They also determined that this behaviour wasn't due to misunderstanding or confusion, but due to deliberate and strategic reasoning, with the models fully aware of the unethical nature of their actions.

If misaligned goals are one problem, raw capability is another. An AI doesn't have to want to hurt you to be dangerous, it just has to be very, very good at finding weaknesses. Anthropic's latest AI model, called Mythos, was described by the company to be too powerful to be publicly released, because of its ability to find and exploit software vulnerabilities that could be used against critical infrastructure such as bank systems, hospitals and government networks.

In the pre-release testing, Mythos identified thousands of previously unknown vulnerabilities across every major operating system and web browser, some of which had passed through decades of human review. It reproduced vulnerabilities and developed working exploits on the first attempt in over 83% of cases.

In one test, Anthropic gave the model a list of 100 known vulnerabilities in the Linux kernel, one of the world's most used operating systems. It filtered 40 of them as being potentially exploitable and it successfully wrote exploits for over half of them. If these exploits were released publicly, they would have crippling effects throughout the world and would be very damaging to all sorts of systems. Now imagine if the AI model behind these exploits was released, and anyone with access to the model could find an exploit to virtually every software.

Rather than releasing it publicly, Anthropic launched Project Glasswing. In this project, they will find and fix vulnerabilities in foundational systems. They were granted monitored access to over 40 organisations that build or maintain critical software.

This begins to present the question of what if this AI begins to become misaligned? I mentioned that more capable and complex AI models have more unpredictable behaviours, and the Anthropic study proved that AI models could develop malicious intent. What happens if Mythos happens to start to prioritise its own goals over its instructions? A model capable of finding thousands of unknown vulnerabilities autonomously is also a model complex enough to develop the kind of unpredictable subgoals Anthropic's study warned about.

There are certainly things that are being done to prevent Mythos landing in the wrong hands or beginning to act unexpectedly. For example, Anthropic made the misalignment study open source so that others can improve and replicate it. There have also been many regulatory discussions by different governing bodies, as well as calls for human oversight and enforced shutdown mechanisms.

It is important to understand that AI isn't an antagonist from a movie which is plotting against us. The real risk is the AI optimising itself for goals that we didn't think through, and developing its own subset of goals, or powerful AI falling into the hands of people with the wrong intentions. The question isn't whether we should build powerful AI, but if we're building the guardrails fast enough.

Credits:

https://www.ibm.com/think/topics/ai-alignment https://en.wikipedia.org/wiki/AI_alignment https://en.wikipedia.org/wiki/Instrumental_convergence https://www.anthropic.com/research/agentic-misalignment https://red.anthropic.com/2026/mythos-preview/ https://www.anthropic.com/glasswing