Top News

• Anthropic executive Evan Hubinger estimates more than 10% odds that AI will cause human extinction within the next decade.
• Samuel Marks said current alignment methods may only steer AI toward better behavior, not reliably control superintelligence
• AI safety debate fanned after Anthropic researcher Jacob Coxon resigned over risks from the industry’s race toward self-improving AI.

AI safety concerns have come into sharper focus after an Anthropic researcher quit over fears that the industry is racing toward self-improving systems that could eventually become uncontrollable, with two senior Anthropic executives now publicly acknowledging the possibility of an AI-driven catastrophe.

Evan Hubinger, Anthropic’s Alignment Science lead, backed former colleague Jacob Coxon’s warning that AI could pose an existential threat. “Jacob is correct here—we really do earnestly believe AI could kill all humans!” Hubinger said, adding that he personally puts the odds at greater than 10% “within the next decade.” 

He acknowledged that Anthropic does not yet have a solution. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” Hubinger wrote.

He clarified that he sees the immediate risk from current models as low, but is concerned about superintelligence emerging through recursive self-improvement.  

Samuel Marks, Anthropic’s Scalable Oversight lead, offered a broader assessment, saying AI developers believe their technology “could cause human extinction” and that this could happen within the next few years. While saying that the view is his own and not Anthropic’s, Marks argued that commercial incentives and competition with less safety-conscious developers are keeping companies in the race.

He also said current alignment techniques can only “nudge AIs towards better behavior” rather than robustly align them. In his view, the industry’s tentative plan is to develop AI capable of aligning its successors better than humans can align today’s systems.

Anthropic Researcher Exits, Triggers AI Safety Debate

The comments followed Coxon’s resignation from Anthropic. The 27-year-old British researcher, who previously worked at OpenAI, said he was leaving because he did not want to participate in an industrywide rush toward self-improving AI. He told The Wall Street Journal that colleagues increasingly describe the trajectory as “crunchtime” and “endgame,” warning that “by the end of next year things could be out of control already.”  

Coxon said he joined Anthropic partly because of its reputation for AI safety, but concluded that even earnest safety efforts cannot make the development of human-level or superhuman AI responsible without government intervention or a coordinated industry slowdown.

Coxon’s departure drew a terse reaction from billionaire investor Bill Ackman, who responded with a “concerning” remark on X.  

Anthropic IPO Watch

The episode adds to growing unease inside leading AI labs, even as companies race to develop increasingly capable systems and Anthropic prepares for a potentially massive IPO.

Anthropic’s IPO, which could be launched later this month, is shaping up to be one of the biggest public-market debuts ever. Reports suggest it could target a valuation of as much as $2 trillion, potentially making it larger than SpaceX’s record-setting IPO.

It could also set a benchmark for OpenAI and other AI companies considering IPOs, making Anthropic’s debut a crucial moment for the broader AI investment boom.

Anthropic’s private market valuation was $1.25 trillion as of the last check, according to Nasdaq Private Market. On Stocktwits, the retail sentiment for the firm was ‘bearish.’

For updates and corrections, email newsroom[at]stocktwits[dot]com.<