What is the AI Equivalent of "Mutually Assured Destruction?"

One of the problems with calls to “regulate AI” is that we cannot seem to agree yet on what needs to be done. Much of the discussion seems to center on reining in frontier model capabilities, mostly along the line of “human in the loop” protections. 

But there are all sorts of other possible approaches that do not deal directly with model capabilities but social impact, economic impact, labor and job impact, consumer protection, transparency, forensic chains, import-export controls or prohibited uses. 

As much as we might agree that some of those areas might make sense, precisely how to regulate frontier models to minimize existential threats remains contentious. 

The central concern isn't simply that an AI gives a bad answer. It is that a sufficiently capable system could plan, acquire resources, replicate, manipulate people, conduct cyber operations, develop dangerous technologies, or resist attempts to shut it down faster than humans can respond.

Perhaps all the main suggestions fall into three buckets. First, make the models safer or constrain them. The former attempts to build an AI that does not want to do dangerous things. 

The second approach assumes misalignment can happen and creates safeguards so harm is prevented. 

The third approach aims to prevent sufficiently dangerous capability from being developed or released. 

Perhaps all three approaches assume all developers agree with the goals. Bad actors will still attempt to create weaponized capabilities, we might well assume. 

This is closest to traditional nuclear/biological-weapons governance. Of course, the issue is that no matter what U.S. firms might do, compliance is voluntary. 

Google DeepMind's Frontier Safety Framework explicitly tracks "Critical Capability Levels" and uses early-warning evaluations followed by mitigation when those capabilities emerge. Its current framework explicitly includes the possibility that a misaligned AI could interfere with operators' ability to direct, modify or shut down it.

OpenAI's Preparedness Framework similarly evaluates severe risks and requires safeguards as capabilities increase; its current governance framework covers cyber, CBRN, manipulation and loss of control, together with security, incident response and external expert input.

Anthropic's policies explicitly separate security, safeguards and alignment. Its roadmap includes access controls, red-teaming, threat intelligence, automated attack investigation and research into ensuring models don't autonomously cause harm.

So there is considerable convergence around a defense-in-depth architecture. The most-consequential unresolved question is whether increasing AI capability eventually makes the defensive layers themselves harder to maintain faster than they become more effective. 

If AI becomes substantially better at cybersecurity, persuasion, strategic planning and AI research, then sandboxing, monitoring and alignment progressively harder.

The main point is that it remains unclear what, exactly, we ought to do about model development and regulating it. Simply banning or slowing development might not work unless all developers globally agree to do so, and abide by those agreements. 

Proposed safeguard	How it would work	Threat it is intended to address	Main advantage	Main weakness
Capability evaluations / early-warning tests	Regularly test models for autonomy, cyber, biological, persuasion, self-replication and AI-R&D capabilities before deployment	Detecting when a model crosses a dangerous capability threshold	Relatively concrete and measurable; can trigger additional safeguards before deployment	Tests can be gamed or incomplete; capability can emerge unexpectedly
"Safety case" before deployment	Developer must construct an evidence-based argument that a model's catastrophic risks have been sufficiently mitigated before release	Preventing deployment of inadequately controlled frontier systems	Forces an explicit connection between capabilities, risks and safeguards	Ultimately depends on whether the safety argument is actually convincing
Scalable oversight	Use weaker AI systems, humans, debate, critique, monitoring, etc. to supervise a much more capable model	Humans becoming unable to understand or evaluate what a highly capable AI is doing	Potentially allows humans to supervise systems beyond their own cognitive capabilities	The supervisor itself may fail; "AI supervising AI" can create correlated errors
Constitutional/value alignment	Train the model around explicit principles governing acceptable behavior rather than relying only on individual human feedback	Model pursuing goals inconsistent with human interests	Gives the model a persistent behavioral framework	There is no universally agreed "human constitution"; a sufficiently capable model might learn to circumvent the intended rules
Interpretability	Analyze the model's internal representations and computations to identify deceptive plans, dangerous objectives or other problematic internal states	Hidden goals, deception and strategic behavior	Could eventually allow researchers to see why a model is behaving dangerously rather than merely observing outputs	Current interpretability is far from providing a complete readout of a frontier model's "intentions"
Deception / scheming evaluations	Specifically test whether models behave differently when they know they're being evaluated, conceal objectives, manipulate overseers, or attempt to preserve their ability to act	Loss of control and deceptive alignment	Targets arguably one of the most important existential-risk mechanisms directly	A model could behave safely during testing and differently in deployment
Sandboxing / containment	Run an autonomous model in an isolated computing environment with restricted filesystem, network, credentials and external communication	Preventing an AI from escaping its environment or acquiring resources	Very powerful because it limits what the model can actually do	A sufficiently capable system may find vulnerabilities or persuade humans/operators to grant access
Least-privilege agents	Give an AI only the minimum tools, permissions, accounts, money, API access and computer privileges necessary for a task	Preventing uncontrolled accumulation of power	Simple cybersecurity principle applied to AI	Complex agents may need increasingly broad permissions to be useful
Human approval gates	Require explicit human authorization before consequential actions—sending money, executing code, changing infrastructure, accessing sensitive systems, etc.	Autonomous harmful action	Maintains a human decision point	Humans can become rubber stamps; sufficiently fast attacks may bypass human review
Continuous monitoring	Monitor agent actions, network traffic, tool calls, unusual planning, attempts to circumvent restrictions, etc.	Detecting dangerous autonomous behavior	Can catch behavior that ordinary output filtering misses	Monitoring itself becomes harder as agents become more sophisticated
Automated tripwires / shutdown	If specified dangerous behavior occurs, terminate the process, revoke credentials or isolate the system	Runaway agents	Fast and potentially automatic	A model capable of anticipating the tripwire might avoid triggering it
Weight / model-security controls	Protect model weights with extremely strong cybersecurity and physical security	Preventing theft of a dangerous model and removal of its safeguards	Particularly important because stolen weights could eliminate deployment controls	Extraordinary security is expensive; insiders and sophisticated attackers remain risks
Controlled deployment	Restrict access to particularly capable models to vetted users, trusted environments or particular applications	Misuse and uncontrolled proliferation	Can substantially reduce the number of pathways to catastrophic use	Reduces openness and economic value; eventually difficult to maintain if models proliferate
Rate/capability limits	Limit how much autonomous work an agent can perform, how many instances can run, how quickly it can act, or how much compute it can consume	Rapid escalation and runaway autonomous activity	Creates time for humans to intervene	Limits can become less meaningful as models become more efficient
Secure model provenance / attestation	Cryptographically establish that a deployed model actually corresponds to the tested and approved model weights	Preventing tampering or substitution after safety testing	Addresses the gap between "safe model tested" and "model actually deployed"	Doesn't establish that the original model itself is safe
Multiple independent AI systems	Have independent systems monitor, critique or approve each other's actions	Single-system failure or deception	Avoids relying on one monitor	Several systems can share the same blind spots or collude/coordinate
AI-to-AI containment / watchdogs	A specialized AI continuously searches another AI's behavior for deception, hacking, dangerous planning, etc.	Monitoring increasingly complex agents	Potentially scales better than human monitoring	Creates an adversarial arms race between the agent and watchdog
Limit autonomous self-improvement	Restrict an AI's ability to modify itself, train successor models, access training infrastructure or conduct unrestricted AI R&D	Recursive self-improvement / rapid capability escalation	Directly attacks one of the more extreme loss-of-control scenarios	Difficult to define "self-improvement"; ordinary AI research increasingly involves AI assistance
Compute controls	Monitor or license very large training runs and/or concentrations of advanced compute	Preventing uncontrolled creation of extremely capable systems	Compute is a physical bottleneck and therefore potentially governable	Algorithmic efficiency can reduce the compute threshold; activity can migrate geographically
International controls	Agreements establishing red lines, inspections, incident communication and human control over especially dangerous AI uses	Global arms-race dynamics and military escalation	Addresses the fact that one country cannot solve the problem alone	Verification and geopolitical competition are extraordinarily difficult

The problem with the analogy to biological or nuclear weapons frameworks is trust. 

If we treat loss of control from highly autonomous AI as analogous to proliferation of nuclear or biological weapons, the policy choices might include: 

• identify dangerous capabilities

• control access to the enabling resources

• verify compliance

• create international mechanisms for transparency and response.

But all that requires international trust and willingness to comply. And that always seems a tough problem to overcome. “Mutually assured destruction,” oddly enough, was what kept the world safe from nuclear warfare. 

The weapons became too dangerous to use, as their party survives the nuclear exchange. What mechanisms can we actually create for AI software, when global trust cannot be relied upon?

Calls for AI Safety Ironically Benefit Politicians, Investors, Model Developers

Perhaps oddly, there now is a symmetry of interests for U.S. frontier-model developers, the financial community and politicians to call for slowing the pace of AI development and the massive amount of spending by hyperscalers. 

It is perhaps not simply coincidence that calls for slowing development (and presumably spending rates) are happening at the same time as scrutiny over capex levels is high, hyperscalers are sacrificing profit margin and midterm elections are approaching. 

The financial community does not like the uncertainty or magnitude of AI infrastructure capital investment. Firms would rather achieve their former substantial profit margins instead of enduring a period of low to negative cash flow. 

Politicians want to appear to voters to be “doing something” about AI before midterm elections, while AI developers, aware that government policy powerfully shapes their business prospects, might want to “play ball” with legislators to avoid punitive policies. 

Frontier-model companies have an economic incentive to participate in policymaking, not merely to oppose regulation, in other words, as well as having incentives to rein in AI capex to mollify investors and the financial markets; preserve credit ratings and reduce financial risks. 

The point: for different reasons, frontier-model developers, financial markets and lawmakers all have reasons to prefer some moderation of AI spending.

If frontier AI keeps accelerating	If frontier AI development moderates
Microsoft fears falling behind Google	Microsoft can wait for better information
Google fears falling behind OpenAI/Anthropic	Google can stretch its deployment schedule
Amazon fears inadequate AWS AI capacity	Amazon can better match capacity to demand
Everyone buys GPUs "just in case "	More emphasis on utilization
Everyone secures power/data centers years ahead	More projects can be staged
Capital gets committed before ROI is clear	More investment can follow demonstrated demand
Scarcity itself creates pressure to invest	Learning reduces uncertainty
Prisoner's-dilemma dynamics	More cooperative investment dynamics

The big unresolved question is whether such coordination is possible without simply handing a strategic advantage to whoever refuses to participate, particularly China. That's the problem that makes this much harder than simply saying, "Everyone should spend less."

Ironically, though, AI safety regulation could function partly as an investment-coordination mechanism.

A slower, more predictable frontier-development schedule reduces the incentive for every participant to make defensive investments against the possibility that its competitors will move faster. And that arguably reduces the danger of overinvestment; reduces risk; allows politicians to be seen as proactive; helps hyperscalers boost profit margins and cash flow and alleviates some “creative” financing methods that are not transparent. 

So perhaps “safety” becomes the enabler for a more-measured pace of frontier-model development that also reduces capex demands. That might also help prevent some amount of overinvestment danger.

S&P Global’s new credit outlook for hyperscalers resembles the famous quip about one-handed economists: strong credit ratings and ability to finance artificial intelligence infrastructure on one hand, and risk from less transparency about payback from those investments. 

“We assume the top six U.S. hyperscalers will spend more than $7 trillion on data centers and AI-related capex from 2025 through to 2030, the ratings agency says. 

S&P Global 

Aside from the obvious issue of revenues emerging quickly enough to justify the spending, there is business risk from unexpected failures in the complex financing ecosystem, which “increases the chances that the failure of an unrated entity could dent the credit standing of a highly rated firm,” the agency says. 

S&P Global 

To be sure, “the question isn't just whether AI creates value, it's whether that value arrives fast enough, and in sufficient quantity, to justify the current pace of investment.”

S&P Global 

Perhaps ironically, concerns about model “safety” are raised just as developers, financiers, investors and politicians also are looking for more moderation. 

Paradoxically, that might also help alleviate the danger of unexpected financial damage were an interlocked ecosystem surprised by a sudden collapse of any key partner’s business.

Google Payback on Tensor accelerators Might be Less than One Year

Does Google Cloud payback on its own Tensor processors happen in less than a year? It might appear so. 

Google Cloud CEO Thomas Kurian, speaking at the Goldman Sachs Communacopia+Technology conference, said the aggregate payback period on Google's AI servers is less than two years. 

source: Alphabet 

But he also said the payback period for Google's own silicon is roughly half that, suggesting a payback within a year.

Noting that Google's accelerator business, including TPUs, is already more than twice the size of the next-largest hyperscaler's accelerator business, he said Google offers 2.7 times better price performance for AI training and 80 percent better price performance for inference, compared with the relevant alternatives.

source: Alphabet 

To be sure, AI demand could slow, chip economics could deteriorate or competitors could close Google's performance gap. In that case, extensive AI infrastructure investment could turn out to be a mistake, as investors worry. 

But some might say the latest figures suggest a reasonable case for payback on extensive infrastructure investments.