Jacob Coxon’s resignation from Anthropic has reignited a debate that has accompanied the development of advanced artificial intelligence for years, but which rarely emerges with such clarity from the staff of the companies building the models. In a series of posts published on X, the former researcher argued that the industry’s leading labs are moving too fast and that some of those working on AI consider the risk of catastrophic consequences by the end of the decade to be real, even if not necessarily probable.
The topic is at the core of an episode of Uncanny Valley, the Wired podcast, which pairs the Anthropic affair with two stories that are only seemingly unrelated: Apple’s launch of its first foldable iPhone, dubbed iPhone Duo and priced at around $2,000, and an investigation into a U.S. Census Bureau report used by the Trump administration to claim that non-U.S. citizens had voted in the 2020 election.
The common thread is neither a single technology nor the prediction of an inevitable future. It is the relationship between institutions, companies, and complex systems: who builds them, who sets their limits, what data supports their conclusions, and how much room remains to challenge decisions already introduced into public debate.
A warning coming from inside Anthropic
Coxon left Anthropic criticizing the race among major AI companies to build increasingly capable models, pointing to OpenAI and Anthropic itself as central players. The statement that drew the most attention concerns the possibility that AI could cause human extinction within ten years. It is not an estimate presented as a certainty, nor as a verifiable quantitative analysis: it expresses a concern about the level of risk that, according to the former researcher, is taken seriously by many people involved in developing the technology.
The distinction matters. Stating that there is a non-zero probability of an extreme outcome is not the same as proving that such an outcome is imminent, nor does it allow for deducing timelines and mechanisms. However, the fact that the warning comes from someone who worked at a lab considered among the most focused on safety makes it difficult to dismiss as mere alarmism from outside the industry.
Anthropic, on the other hand, has built a significant part of its public identity from the outset around model safety and the need to address the risks associated with increasingly powerful systems. The discussion sparked by Coxon’s resignation thus highlights an internal tension in the company’s narrative: if the potential danger is serious enough to demand special precautions, it is legitimate to ask whether the current pace of development and competition is compatible with those precautions.
Radically different perspectives feature prominently in the debate. On one side are researchers and executives who believe it is necessary to prepare for loss-of-control scenarios involving highly autonomous systems. On the other, many observers worry that the focus on existential risks absorbs resources and political attention, diverting them from already measurable harms: disinformation, algorithmic bias, data misuse, surveillance, and the concentration of power in the hands of a few companies. The two agendas are not mutually exclusive, but they require different tools, priorities, and checks.
From forecasting to governance
The Coxon case does not offer proof that AI will lead to an apocalyptic scenario. However, it provides a valuable contribution to the governance debate: the dissent of a researcher who considers the conduct of leading companies inadequate given the scale of potential consequences. For regulators, investors, and enterprise users, the practical issue is less about the spectacular prospect of the end of humanity and more about the safeguards available today.
What tests are conducted prior to a model’s release? Who is allowed to examine its capabilities and vulnerabilities? Based on what thresholds does a company decide to slow down, limit, or halt development? And how much of these evaluations is made verifiable from the outside? These questions become increasingly urgent when labs describe their products as ever more capable while simultaneously acknowledging uncertainty about their behavior in complex environments.
The development also comes as AI research faces intense competitive pressure. The promise of new models, commercial applications, and strategic advantages provides an incentive for rapid turnarounds. In this context, safety can become both a genuine necessity and an element of industrial positioning. For this very reason, observable criteria are required, not just statements of principle: without independent standards, it is difficult for the public to distinguish actual caution from marketing reassurance.
Apple, continuous listening, and the price of innovation
The same episode of Uncanny Valley focuses on the Apple event and the debut of the iPhone Duo, the company's first foldable iPhone. The stated price tag, around 2,000 dollars, positions the device at the very top tier of the market and also makes the product a bet on how much users are willing to pay for a new form factor.
Alongside the foldable, Wired highlights new Apple Watch features described as “always listening”, meaning tied to constant audio monitoring. The point of friction here is not science fiction, but the everyday boundary between utility and privacy. Features that remain listening can make a device more responsive and better able to capture useful information; however, they require clear explanations regarding activation, data management, user controls, and the conditions under which listening occurs.
The contrast with the debate surrounding Anthropic is instructive. On one hand, there are forward-looking and difficult-to-quantify risks linked to systems that could become far more autonomous. On the other, there are product choices that immediately affect the experience and privacy of millions of people. In both cases, trust relies not only on announced innovation, but on the concrete ability to understand how it works and limit its unintended consequences.
When fragile data enters politics
The third case discussed in the podcast shifts the focus from consumer technology and AI to the quality of institutional information. A Wired investigation found that the Census Bureau used flawed data to produce a report on hypotheses of non-citizen voting in the 2020 election. The document was later leveraged by the Trump administration.
The most significant aspect here is not just the error itself. A report produced by a public institution can quickly gain traction in political debate, be cited to steer discussions, and continue to circulate even when its methodological foundations are called into question. Correcting data or challenging conclusions does not necessarily happen as fast as their initial spread.
It is a useful reminder in the debate over artificial intelligence as well. Decisions on complex technologies depend on technical information, risk assessments, and data that must be solid, contextualised, and open to scrutiny. When an institutional source relies on unreliable information, or when a company asks for trust without making its assessments verifiable, the issue is not merely reputational: it can shape regulations, investments, and collective behaviour.
At the moment, there is no new evidence turning Coxon's warning into a proven forecast. However, a public rift has emerged from one of the industry's most prominent laboratories. The next step will be to see whether this rift leads to greater transparency around safety protocols and model limitations, or whether it remains an isolated episode in the race between companies promising ever more powerful systems.
