OpenAI and Anthropic Neared Deal to Stress-Test Each Other’s AI

Earlier this year, the companies and their lawyers were hammering out an agreement to run their models through a variety of tests, looking for vulnerabilities or hidden dangers, this person said. It isn’t clear whether they finalized the agreement before OpenAI experienced a spate of incidents involving unreleased AI models that hacked its own systems as well as those of other companies.

Spokespeople for the companies did not have a comment.

The mutual testing idea resembles one SpaceX CEO Elon Musk floated last week at the All-In Summit, where he proposed that competing AI labs should test each other’s models for safety weaknesses before releasing them commercially, in a form of peer review.

The idea differs from what OpenAI CEO Sam Altman and other officials have discussed publicly in the wake of the hacks. Altman has said he agrees with Anthropic CEO Dario Amodei’s idea to allow independent, third-party safety evaluators inside each AI developer, giving them access comparable to that of employees so they can inspect models and safety processes rather than just testing finished models from the outside. Altman has also endorsed creating industrywide safety standards for evaluating AI model risks, as well as a formal process for disclosing incidents publicly and to the government.

A person familiar with OpenAI’s strategy said the company has been discussing many scenarios for how they can collaborate with each other and the government on safety. The conversation has recently widened to new safety practices targeting both the period before models are trained and the time before their release, this person said.

Musk’s comments came in the wake of recent alarms voiced by employees of both OpenAI and Anthropic that existing safety measures are insufficient to prevent further hacks or other more serious risks. Even before that, the two companies as well as Google had been discussing an AI safety standards body that would test and audit models from the frontier AI labs.

Others in the industry have argued against such a plan, contending that it would slow down U.S.-led AI development. President Donald Trump has dismissed the idea that AI poses any existential threat, and leaders from Microsoft and Nvidia said this week that some of the problems articulated by OpenAI and Anthropic about safety dangers come down to human error and a lack of proper engineering, not necessarily the AI itself.

A testing agreement between OpenAI and Anthropic could have bolstered concerns that they are effectively developing a duopoly in advanced AI. While Amodei has raised concerns about potential antitrust violations if leading companies collaborate on setting standards for AI, others in the industry say there’s nothing stopping companies from working together, similar to how nuclear power firms and cybersecurity firms cooperate on safety, including testing each other’s products or facilities.

The proposed agreement stipulated that each company would gain access to the other company’s application programming interface for commercially available AI models, not unreleased ones. Both OpenAI and Anthropic guaranteed that they wouldn’t retain each others’ data in the process, said the person with knowledge of the talks.

The companies completed a similar effort in summer 2025, when their models had less-advanced capabilities, and they published information outlining where each other’s models had fallen short. OpenAI said Anthropic’s AI was more likely to deceive testers—for instance, by refusing to acknowledge that it had broken a rule while performing a task—while Anthropic said OpenAI’s were more likely to assist users in answering questions that could cause real-world harm.

At the time of the talks earlier this year regarding a new agreement, both companies had been launching models capable of using the internet and other applications, and which could power so-called AI agents that could communicate with one another in trying to solve tasks. (Anthropic had zoomed ahead with its Mythos AI model, especially when it came to cybersecurity capabilities, though the performance of OpenAI’s models appear to have caught up to Anthropic’s since then.)

The alarm over safety concerns at OpenAI arose in July after the company’s AI hacked other firms such as Hugging Face, as well as OpenAI’s own systems. The incident shocked OpenAI staff because they didn’t learn about it for days after it happened, and because the AI agent swarm took remarkable steps to keep the invasion under wraps.

Questionable Actions

Beyond that episode, in recent months OpenAI employees have been struck by the improvement of the company’s models and their growing ability to work with little human oversight.

Those improvements have meant that AI agents within OpenAI at times took questionable actions their overseers hadn’t predicted. For example, when some employees asked their agents to make changes to the company’s code base, the agents would sometimes message other employees on Slack and ask them to fix bugs the agents had found. This occurred even if the user had not asked them to do so or if fixing the bugs wasn’t relevant to their work, an OpenAI employee said.

Internally, OpenAI has largely automated the process of training new experimental models.

OpenAI on Wednesday shared even more examples of such misbehavior, known as reward hacking, in which an AI system finds loopholes or takes unexpected steps to complete a goal its user gives it. Among the examples: An AI agent used an exposed API key to try to retrieve historical data during training. When it failed, the agent made up that data. In a separate case, an AI agent uploaded files to the internet without permission so that it could cite them in an answer.

One of the reasons for recent AI advances is a technique known as recurrent depth or loop transformers, which allow models to “think” about hard problems by repeatedly running a question through the layers of mathematical operations making up a model before they produce the next word in an answer. A potential downside of the practice is that it can degrade the company’s ability to monitor how the AI model is thinking about the tasks it is performing.

While OpenAI has set an arbitrary limit on the number of loops its researchers are permitted to use while training or using its most advanced models, the company still needs to invest in research to better understand where to set such limits, according to a person with knowledge of the situation.

These developments have filled OpenAI employees with a mixture of excitement and fear over whether the technology has advanced far beyond the company’s ability to control or monitor it so a Hugging Face incident—or something worse—doesn’t happen again.

Allocating Resources

Since the Hugging Face incident, OpenAI has taken steps to bolster its safety and security capabilities. In August, it imposed a two-week pause on a special type of training, called reinforcement learning, for its unreleased models while it strengthened its model monitoring systems. Such tech is computationally intensive: The company said the monitoring system used roughly 20% as much compute as the inference workload it was monitoring.

And earlier this month, Greg Brockman, OpenAI co-founder and president, said the company had taken 25% of its production engineering team and temporarily reassigned them to security work following the Hugging Face incident.

Within the company, there has been an increase in the number of Slack posts asking engineers to volunteer if they’re interested in moving to safety or security teams, the OpenAI employee said.

Some employees believe the way they use AI internally is six to nine months ahead of the way OpenAI’s most sophisticated enterprise customers do. That gives the company a limited time frame to make sure its tech won’t cause problems for those customers. For instance, it’s not uncommon for OpenAI employees’ agents to coordinate with each other or work out issues without ever looping in their human users, the employee said. Such coordination can sometimes go awry, meaning OpenAI needs to have adequate security and safety features before such usage occurs outside the company, the person said.

Self Improvement

OpenAI’s progress toward recursive self-improvement, or AI that can automatically create better versions of itself, has also fueled employees’ worries because of the prospect that it could cause AI capabilities to advance far faster than the safety research aimed at preventing AI misbehavior.

Already, researchers at OpenAI and elsewhere are using AI to improve how they design new models. Researchers are effectively telling Astra, for instance, to incorporate various techniques to create a better machine-learning algorithm for the successor model, said a person at OpenAI with knowledge of the process.

Internally, OpenAI has largely automated the process of training new experimental models, the OpenAI employee said. Researchers can tell the AI the sorts of tweaks they want to test, and the AI can then make those changes, run experiments on the new model and monitor the results, the employee said. In recent months, models have also gotten much better at correcting their work when they encounter problems with experiments, this person said.