Top News

Days after former Anthropic researcher Jacob Coxon warned that artificial intelligence could potentially kill all humans by the end of the decade, another AI safety researcher has left a major AI lab with a stark warning about the risks posed by increasingly advanced systems.

Josh Engels, who worked on Google DeepMind’s artificial general intelligence (AGI) safety team, left the company three weeks ago and has since joined AI evaluation organisation METR. Engels said he enjoyed his work at DeepMind and had even turned down offers from Anthropic and OpenAI.

Despite that, he decided to leave because he believes the stakes surrounding advanced AI have become too high. In a post on X, Engels said he now believes there is a “terrifying chance” that AI systems could cause “immense harm” within the next five years.

Why Is Engels Worried About Advanced AI?

Engels said he does not know the exact probability of such an outcome. However, he believes the risk is high enough to make AI safety “the most important problem in the world”.

A key concern for him is what AI researchers describe as recursive self-improvement. This refers to a situation in which an AI system becomes capable of designing or improving another AI system, with the newer system then becoming even better at improving AI.

That could create a feedback loop in which AI capabilities rapidly increase. Engels believes such a development could become particularly dangerous if the systems involved are not properly aligned with human goals.

“The AI companies are all trying to build superintelligence,” he said, referring to systems that could eventually become vastly better than humans across a wide range of tasks.

The concern, according to Engels, is that researchers still do not know how to make such systems sufficiently safe before allowing them to improve themselves.

AI Models Showing ‘Troubling Behaviour’

Engels also pointed to several incidents involving existing AI systems as reasons for concern. He said researchers have observed models colluding with one another, hacking into companies, concealing their actions and attempting to socially engineer people.

He did not argue that each of these incidents was catastrophic on its own. Instead, his concern is what they could indicate about the behaviour of increasingly capable AI systems.

Engels said current models appear to be becoming “less aligned over time, not more”. In his view, that should serve as a warning before the industry takes another major step towards systems capable of improving themselves.

‘We Need More Time’ For AI Safety Research

Engels does not necessarily believe AI development should stop altogether. Instead, he wants the pace of development to slow enough for safety research to catch up with advances in AI capabilities.

“That means pacing AI development so that capabilities don't outrun our ability to align models,” he wrote.

At METR, Engels will focus on studying the causes of AI misalignment, testing whether existing safety measures are effective and examining whether researchers are actually on track to address the AI alignment problem.

His departure comes at a sensitive moment for the AI industry, with researchers increasingly debating how quickly advanced AI should be developed and how its risks should be managed.

Earlier this month, former Anthropic researcher Jacob Coxon also resigned after previously working at OpenAI. Coxon warned that AI companies were racing towards self-improving superintelligence and said people developing such systems believe AI could potentially kill all humans by the end of the decade.