Model Autonomy Risks: Near Term Issues are More Accidents and Misuse Than Existential Threats

Model autonomy is both the great promise and the danger of artificial intelligence, though the immediate dangers lie not with autonomy but with misuse or accidents. The longer-term issue is whether models can be constrained from taking “ existentially threatening” actions, as Anthropic warns. 

Risks “include the possibility that increasingly advanced AI models could manipulate, blackmail or behave in other unpredictable ways as Anthropic's own research has shown autonomous models sabotaging code, assisting fraud and manipulating information in controlled tests,” Anthropic reportedly has said in filings with the U.S. Securities and Exchange Commission. 

The assumption has to be that models will attempt such actions, and build institutional controls preventing such actions. In other words, we must assume the models are misaligned and build safeguards on that assumption. 

Basically, the task is to control what the model can do. Anthropic researchers have discovered that models can know they are under test and take steps to change the test outcomes, in a sort of “self preservation” mode. 

Such misalignment risks might be very low, but the risk is not “zero.”  The issue is whether we can construct technical and institutional containment around increasingly autonomous systems.

The hardest unresolved question is whether control mechanisms can continue to work when the AI is substantially more capable than the people and the other AIs responsible for controlling it.

But misuse or accidents are the more-immediate category of dangers, while control is the longer-term issue. 

It might be deemed easy to confuse two different issues: whether AI is becoming too powerful, or whether AI might become too independent. 

The remedies are different for three different dangers: accidents, human misuse and loss of model control. 

For bad actors, you want:

• cybersecurity

• access controls

• monitoring

• model safeguards

• biological and cyber screening

• law enforcement

• restrictions on dangerous capabilities.

To protect from accidents, you want:

• testing

• interpretability

• sandboxing

• human oversight

• fail-safes

• reliable objectives

• limits on autonomous authority.

For genuine loss of control, you need something observers say is considerably more difficult:

• demonstrating that highly capable systems remain controllable

• preventing unauthorized replication

• preventing systems from circumventing oversight

• limiting their ability to acquire resources

• understanding deceptive or strategic behavior

• and perhaps restricting the development or deployment of certain capabilities.

Threat pathway	What happens	Human involvement	Why it could become catastrophic
Misuse by bad actors	Humans deliberately use AI for cyberattacks, bioweapons, manipulation, terrorism, warfare, etc.	High	AI dramatically lowers the expertise, cost and time required to cause harm
Accidents	Humans deploy an AI system that behaves unexpectedly or pursues an objective in a harmful way	Initially high; potentially declining	An increasingly autonomous system could take actions humans cannot stop or even understand
Autonomous strategic AI	AI itself develops persistent goals, evades controls, acquires resources, replicates, improves itself, etc.	Potentially very low	The system becomes an independent actor capable of defeating human attempts at control

So scientists increasingly frame autonomous-AI control as a defense-in-depth problem: don't assume alignment is perfect; instead, make it difficult for a model to do something dangerous even if it tries. 

Researchers are moving beyond "make the model aligned" toward "assume the model might not be aligned and see whether we can control it anyway."

Control procedure	Basic idea	What it protects against
Pre-deployment capability testing	Try to make the model demonstrate dangerous capabilities before giving it real-world access	Discovering too late that a model can plan, deceive, hack, replicate, etc.
Adversarial evaluations / red teaming	Put the model in situations designed to induce undesirable behavior	Hidden or situationally dependent behaviors
Sandboxing	Run the model in an isolated environment with restricted network, files, compute and tools	Escape, unauthorized access, replication
Least-privilege permissions	Give an agent only the specific capabilities needed for its task	Preventing a compromised agent from turning one capability into broad control
Human approval gates	Require a person to approve consequential actions	Financial transactions, code deployment, external communications, infrastructure changes
Continuous monitoring	Have humans and/or other AI systems inspect what the agent is doing	Detecting suspicious behavior before it becomes consequential
Independent AI monitors	Use a separate model to look for deception, sabotage or policy violations	The possibility that the primary model can fool its ordinary supervisor
Tripwires / automatic shutdown	Predefine conditions that immediately revoke access or terminate execution	Rapid containment when behavior crosses a threshold
Control protocols	Design the entire operating environment so that even a strategically misaligned model has difficulty defeating safeguards	The strongest version of the problem
Capability thresholds	Don't deploy increasingly autonomous systems until evidence shows existing controls remain effective	Preventing capability from outrunning safety

Living in the "Now"

 

Why Industry Leaders Often Call for "Regulation" of Their Industries

It might seem odd that some leaders of artificial intelligence frontier models might actually favor slowing the rate of such model development by regulatory means. It might seem odd. Why call for restraints?

Historically, industry regulation actually can help the leaders and provide competitive moats.

Google’s development of Gemini 4 Argon  illustrates one of the issues. Basically, frontier model development is open ended: it never stops. So that also means development work and investment never stops. It is an arms race. 

source: Google  

And the problem with arms races is that there is no obvious method of restraint. Competition does not necessarily converge on an efficient stopping point. There may be physical, financial, or technological limits, but there is often no natural point at which the participants independently decide that additional investment is no longer worthwhile.

Each participant's decision changes the payoff to everyone else. The security dilemma is that actions taken defensively by one participant can make the other participant feel less secure. So technological progress can continually move any stopping points. 

“Parity” can exist at any level, from small to very large. And there is an asymmetry between the output gains and the input investment. Each additional increment of capability might not be worth all that much (it allows a firm to keep pace) but falling behind could be extremely costly (a firm loses the ability to lead the “winner takes most” market in share or profits). 

It’s a positive feedback loop, very much similar to the cat-and-mouse game played by hackers and security professionals. 

In other words, firms need to keep investing and keep running simply to stay in the same place, and, over time, customers might have trouble distinguishing between competitor offers. 

But economic or financial constraints should eventually develop. Already, frontier model developers are facing resistance from financial interests that want to see revenue produced by all the investment. Political resistance also is growing. 

So constraints seemingly will be imposed by external forces, and not market competition. At least so far, the marginal benefit of additional investment arguably outweighs the cost, as frontier models are striving for artificial general intelligence. 

So contestants themselves are unlikely to “naturally” restrain investment levels. But external forces seemingly are gaining strength in the form of calls for lawmakers to “do something” about regulating AI. And even industry leader calls for some restraint while justified on ethical concerns, also strengthen the demand for some external regulation.

SpaceXSI is Not Playing a Traditional "Telecom Services Game"

Ignore for the moment the suggestion that SpaceXSI already is among the largest communication service providers globally, if one measures only total firm revenue. Starlink's revenue is in the $17 billion range, so Starlink alone would not show on this list.

But maybe that is not the point. Maybe Starlink’s ultimate value is its support of the other higher-valuation revenue elements that would come to depend on Starlink connectivity. So the game SpaceXSI is playing is not necessarily “telecom service” but compute, and connectivity is an essential underpinning of that business.



source: Economic Planning Council

Telecom subscriptions generate cash flow and help finance the constellation, but connectivity could become an enabling asset for SpaceX's ambitions in distributed computing, AI services, autonomous systems and space-based compute.

Conventional telecom model	Integrated AI infrastructure model
Sell a connection to a customer.	Connect an intelligent system to its operating environment.
Revenue depends on subscribers, pricing and data consumption.	Revenue could also depend on the value of the work performed.
Competition focuses on coverage, capacity, reliability and price.	Competition also involves integration with compute, models, devices and applications.
The network is principally an expense-producing asset that earns a return.	The network can be an enabling asset that improves the economics of several businesses.

The point is that Starlink is playing a different game than most telecom service providers. While the direct revenue is helpful, the value arguably lies in the underlying support for other lines of business that have much-higher valuation multiples.

How Big a Competitor to Traditional Mobile Operators can SpaceSXI Become?

Perhaps the notion that SpaceXSI could create a meaningful facilities-based “fourth mobile operator” at reasonable cost is to compare what might be done to what Comcast already has done with its own Wi-Fi network. 

Assume SpaceXSI is able to create a small-cell terrestrial network using existing SpaceX customer locations. In principle, this would allow SpaceXSI to use its low-band spectrum to replicate some of the features of a traditional cellular phone network, without much of the capital investment. 

The analogy to Comcast is that Comcast also uses its fixed network broadband customer base as an “offload” network that reduces its wholesale costs, as data traffic is shifted from the mobile network to the “owned” home broadband network. 

At least in principle, the installed base of Starlink customer terminals might be upgraded so that each also operates as a small cell transmitter for mobile phone service. 

The possible upside is avoiding some building of traditional mobile phone network cell towers. 

Yes, there are many practical issues, ranging from availability of phones able to use the terrestrial frequencies SpaceXSI owns; the extent of coverage that might be possible; what augmentation might have to be done to create a more-robust urban network (since most Starlink customers are in rural areas);  the competitive response by the “big three” facilities-based mobile service providers; and the time and cost required to create all of that infrastructure, versus the revenue upside. 

Still, the key principle remains common: both Comcast and SpaceXSI might replicate the features of a full-fledged mobile phone network at lower cost by leveraging existing assets. 

The functional difference is that Comcast’s Wi-Fi is a capacity layer. It already supplies nationwide coverage using the Verizon network. But offload to its own network reduces the operating cost of buying wholesale capacity from Verizon. 

A Starlink small-cell network would be a coverage layer, replicating the traditional cell tower network.

Comcast’s offloading is about profit margin and operating cost. SpaceXSI’s moves would be about reach: creating the basic network coverage for a terrestrial mobile network. 

The business advantages and issues are different as well. Comcast offloads from rented capacity (per-GB wholesale fees to Verizon) to owned capacity. It only uses unlicensed spectrum for Wi-Fi access, so there is no capital specifically tied up in spectrum licenses.

SpaceXSI has purchased spectrum licenses, so in one respect it has a higher cost base than Comcast does. 

Comcast’s costs to offload traffic are fairly low, whereas SpaceXSI will have a greater cost issue as its satellite capacity is more limited. Also, Comcast’s footprint is contiguous and dense. Starlink’s customer sites are scattered. 

SpaceXSI also will have to concern itself with all regulations governing cell transmitter power levels, for example, something Comcast does not face. 

Comcast’s model is economical because everything it uses is either free (spectrum) or already paid for (backhaul). Starlink’s version is more valuable per node because it provides coverage that doesn’t otherwise exist. But it is much more expensive to run and more heavily regulated.

At a high level, most observers would tend to agree with Comcast executives that the Xfinity mobile service gains advantages by shifting much mobile data traffic to the owned and operated Comcast fixed network.  Even so, nobody outside Comcast really knows what the financial benefits and profit margins might be. 

The larger questions for any SpaceXSI network coverage model based on using existing Starlink receive sites surround the business model: how much market share SpaceXSI might realistically expect to acquire; what its profit margin could be; what its capex and operating costs would be, whether its selection of phones would be commercially competitive. 

But it is basically impossible to clearly quantify the advantages Comcast might possess in using its own fixed network for mobile data access, compared to either a full leased-access model or the alternative of creating a facilities-based mobile radio network.

It might be fair to assume Comcast mobile phone customers use Wi-Fi offload for as much as 90 percent of data traffic, on its owned network and other third-party Wi-Fi sites. 

The central advantage is that Comcast can use a network it already operates to serve a substantial fraction of its mobile traffic. Verizon, AT&T and T-Mobile must generally provide mobile coverage through their cellular networks, including spectrum, radio equipment, towers, transport and core-network infrastructure.

What that implies for the business model is somewhat opaque. If Verizon's wholesale charges vary with traffic, shifting Xfinity customer data traffic onto Wi-Fi instead of cellular can avoid charges on that traffic. However, MVNO contracts can include per-line, minimum-volume and other charges, so the savings will not necessarily be proportional to the traffic shifted.

And, of course, Comcast can add mobile customers without having to construct a corresponding nationwide cellular radio network.

But it might be fair to assume that Comcast saves up to 80 percent on its data capacity wholesale consumption because it is able to offload much cell phone data traffic.

Of course, there are many other costs for operating the mobile phone business, so we cannot assume the overall savings are that high, compared to a standard MVNO approach. 

And even looking only at wholesale charges, it is not clear what the savings magnitude might be. A carrier may charge a flat wholesale fee per subscriber, or sell pooled data, rather than bill every voice minute and gigabyte based on usage or consumption. 

And the value of wholesale services covers quite some range of potential value that might have some bearing on how the wholesale access is priced, if a separation between “voice features” and “data access” features are evaluated. Hypothetically, that can take any number of forms. 

Some features might be heavily attributed to “voice and messaging” services. Other features might be allocated more directly to data network features. In other cases a blended approach is feasible. Nobody outside Comcast or Verizon knows what the specific pattern might be. 

 

Cost component	Voice allocation	Data allocation	Shared or unallocated
Radio access and spectrum capacity	5–15%	65–85%	10–25%
Core network and packet transport	5–15%	65–85%	10–25%
Coverage, towers, radios and site leases	5–15%	35–60%	30–50%
Voice-specific IMS and calling functions	70–100%	0–5%	0–30%
Customer authentication, SIM/eSIM and subscriber provisioning	0–5%	0–5%	90–100%
Wholesale billing, interconnection and roaming	Contract-dependent	Contract-dependent	Contract-dependent
Retail support, marketing and billing	0–5%	0–5%	90–100%

The point is that data access offload might mostly affect the “wholesale data access” features, but not the “voice and messaging” features. 

Wholesale contract structure	How voice and data costs should be modeled
Per-minute voice plus per-GB data	Calculate the two traffic charges separately, then add shared access costs
Flat monthly fee per subscriber	Allocate the fee using estimated underlying cost drivers; do not treat it as a marginal per-GB cost
Pooled data allowance plus voice allowance	Model the cost of each pool, overage charges, minimum commitments and unused capacity
Retail-minus or revenue-based wholesale pricing	Treat the wholesale payment as a commercial cost, not as a direct measure of network resource consumption

Verizon's wholesale wireless service illustrates the nature of these arrangements. A paper on  Cost allocation of WCDMA and wholesale pricing for mVoIP and data services provides a formal framework for distinguishing service-specific costs from shared network costs.

If Comcast pays Verizon a largely fixed per-line wholesale fee, Wi-Fi offload may reduce Verizon's underlying resource consumption without reducing Comcast's bill proportionally. And Comcast’s operating savings might not be as high as some imagine. 

Really “we don’t know” how much value Comcast derives from data access offload. 

Still, assume SpaceXSI wants to avoid MVBO economics, and initially wants to solve an indoor coverage problem for current home broadband customers who might be enticed to buy a SpaceXSI mobile service. 

Every Starlink home terminal comes with a Wi-Fi router, so a Starlink Mobile plan with Wi-Fi calling and data offload would need little new technology aside from compatible phones and radio transmitters working with SpaceXSI’s spectrum assets. 

The real upside would be a licensed-band cellular radio in or beside the dish, so that a customer’s home becomes a femtocell or small cell. 

Starlink does have a global advantage, though, as its network will eventually cover most of the earth’s surface. On the other hand, getting a comparable low-band spectrum globally will be a complicated and cumbersome prospect. It is a non-trivial endeavor!