According to Anthropic, it is time to hit the brakes on the development of AI technology. Contemporary models are becoming so proficient at writing code that humans are at risk of losing control.
In a blog post, Anthropic makes a striking appeal: it asks to temporarily slow down or even pause AI development. According to Anthropic, technology today is evolving faster than society can keep up with. Social structures and research must be given the chance to catch up with AI.
Coming from Anthropic, these words are all the more remarkable. After all, the company is in a constant neck-and-neck race with OpenAI to develop new, even better AI models. With Claude Mythos, it recently released a model to the world that is so good at detecting vulnerabilities that even the US government is getting involved in who should and should not have access to the model. “The way we secure AI systems, monitor them, and guide their behavior is becoming increasingly important,” write co-founder Jack Clark and Marina Favaro, lead researcher at The Anthropic Institute.
AI develops AI
Anthropic is partly offering a mea culpa for the current speed at which AI is developing. In just five years, it went from the first Claude model to coding agents that can write code as well as or even better than humans. Now, according to Anthropic, we are in the phase of autonomous agents that no longer need instructions from humans and can direct each other. This is a trend that was also eagerly capitalized on this week at Computex in Taiwan.
As AI agents become increasingly better at coding and operate more independently, the next phase draws closer. At that point, AI systems would be capable of developing and improving their ‘successors’ themselves. “We’re not there yet, but it could happen faster than most institutions are prepared for,” Anthropic warns.
Quantity and quality
Anthropic sees ‘evidence’ for its claim in its own backyard. Over the past month, eighty percent of internal code was written by Claude Code. Company employees testify in the blog that they haven’t written any code themselves for months and have multiplied their productivity thanks to AI. Since this year, Anthropic has reported eight times more lines of code per employee.
AI systems developing their own successors? We’re not there yet, but it will happen faster than institutions are prepared for.
Jack Clark, co-stichter & Head of Policy Anthropic
It’s not just quantity that is advancing rapidly, but quality as well. Whereas until six months ago AI code was not yet of the same quality as that of an experienced programmer, that gap is narrowing by the day. Anthropic expects coding agents to be able to deliver even better work than human developers before the end of this year.
While this currently yields major efficiency gains, Anthropic notes that human control is declining. If it ever reaches the point where AI models are capable of evaluating and improving themselves, there will be nothing left to stop them, and humans will definitively lose the reins.
The question is whether Anthropic’s call will be heard. After all, major investments are at stake. In the coming months, the company intends to bring together policymakers, researchers, and its industry peers to discuss this.