What AI Changes in the Area of Connectivity

Year in and year out, it is always safe to predict that connectivity bandwidth demand will grow. But we might always ask how specific innovations drive growth in different parts of the connectivity fabric.

Broadly speaking, artificial intelligence computing workloads and need for connectivity occur across several parts of the network:

• Intra-Data Center: GPUs within a single cluster must constantly synchronize "weights" and "gradients" during training. This requires 800G (and soon 1.6T or 3.2T) optical links.

• Data Center to Data Center (DCI): Large models are increasingly trained across distributed clusters in different geographic regions to tap into available power grids. So data center to data center capacity must be reinforced.

• Data Center to End User: While inference uses less bandwidth than training, the content density is increasing, moving from text-based AI to real-time video. This is primarily access network augmentation and already is happening for other reasons.

But it isn’t simply the data volume that AI might be changing. 

AI traffic, being machine traffic, has different characteristics than human-generated traffic, which has fairly well-defined hourly patterns across any 24-hour period. AI traffic, on the other hand, is less predictable. 

Redundancy and load-shifting are traditional ways of dealing with temporary spikes in compute demand, but AI demand spikes might paradoxically also cascade, if additional resources in other regions are insufficient, or if the connectivity fabric cannot support spikey load shifting. 

So some might say the architecture of internet traffic is evolving from human-shaped to machine-shaped. 

Connectivity Segment	Growth Driver	Primary Demand Metric	Expected Impact/Tech Shift
Intra-DC (Back-end)	GPU-to-GPU synchronization for LLM training.	10x fiber density increase.	Shift to 1.6T/3.2T Ethernet; Co-packaged optics (CPO).
DCI (Data Center Interconnect)	Distributed training & "Check-pointing" across regions.	145% CAGR in pluggable optics.	800G ZR/ZR+ becomes the standard for long-haul.
DC to Edge/User	Real-time Video GenAI & "Agentic" workflows.	30% - 50% annual traffic surge.	Growth in Edge AI data centers to lower latency.
User to DC (Uplink)	AI Wearables (Video glasses) & sensor-rich IoT.	Shift in Asymmetry.	Uplink demand starts to rival Downlink in specific sectors.
Network Management	AI-driven traffic orchestration (NaaS).	38% CAGR in software.	Shift toward Network-as-a-Service (NaaS) for dynamic scaling.

It is reasonable to expect many other app and behavior-related changes to happen as well. 

If AI is rapidly increasing network traffic volume and unpredictability, enterprises and global Internet infrastructure providers will have to redesign their systems for resilience. 

So new enterprise bandwidth, latency, and congestion controls will be needed to handle these loads, many will argue, including:

• Treating AI workloads as having distinct traffic patterns

• Add traffic shaping, rate limiting, intelligent filtering, and workload isolation features  

• Reducing cross-region data movement, placing data closer to models

• Building redundancy across regions and providers

• Using real-time monitoring and predictive analytics to detect anomalies

• Assuming double to five times “typical” traffic spikes.

Compute platforms supporting “AI compute as a service” will likely have to consider:

• Expand bandwidth, backbone capacity, and route diversity

• Deploy distributed inference and GPU capacity at the edge

• Implement AI-aware routing and advanced congestion management

• Increase regional redundancy, maintain reserve capacity

• Enable dynamic scaling and improved tenant isolation.

The point is that the costs of creating new AI compute facilities has a number of costs in addition to shells (buildings), power and processors. 

Category	Examples	Why needed for higher AI compute	Very rough cost indications (order of magnitude)
High‑performance interconnect inside clusters	InfiniBand/Ethernet switches, NICs, NVLink bridges, optical cabling, spine‑leaf fabrics	Distributed training and large MoE models need low‑latency, high‑bandwidth links between thousands of GPUs; networking can be a large fraction of AI cluster capex.	Per large AI cluster, networking (switches, NICs, optics) can easily run into hundreds of millions of dollars; per GPU, interconnect can add roughly USD 3,000–10,000 over server cost depending on scale and topology
Storage systems	High‑performance NVMe in servers, parallel/distributed file systems, object storage, backup/archival storage	Training data lakes, checkpoints, model artifacts and logs require very high throughput and capacity; storage performance strongly affects GPU utilization.	Full AI stack hardware (servers + storage + networking) often implies base systems at roughly USD 5,000–45,000 per server before GPUs; petabyte‑scale storage systems add millions to tens of millions of dollars per region.slyd​
Advanced cooling infrastructure	Direct‑to‑chip liquid cooling, immersion tanks, rear‑door heat exchangers, upgraded chillers, pumps, heat‑rejection systems	Rack densities >30–100 kW for GPU servers make air cooling insufficient, forcing large investments in liquid cooling plants, distribution loops, and monitoring.	Liquid‑cooling deployments for AI halls can cost tens of millions of dollars per site; over life of the facility, cooling energy is a major part of 15–25% “power & cooling” share of AI TCO.
Power delivery beyond basic “power”	Substations, high‑voltage switchgear, UPS, PDUs, busways, redundant feeds (N+1/2N)	Dense AI clusters require huge, highly reliable power; providers must oversize and harden electrical systems to avoid outages and support higher rack densities .	Upgrading a site’s electrical plant for AI (substation, UPS, distribution) typically runs in the tens to hundreds of millions of dollars for hyperscale campuses, depending on MW added and redundancy.aegissoftte
Data‑center facility upgrades (non‑shell)	Containment systems, raised floors, structural reinforcement for heavy racks/tanks, fire suppression tuned for liquid cooling, white‑space re‑fit	Existing halls often must be rebuilt to handle heavier racks, new coolant loops and different airflow patterns; safety systems are upgraded for new thermal/chemical risks.	Retrofit of an existing hall to AI‑grade density can cost several thousand dollars per square meter; full hall conversions often run into the tens of millions of dollars per building.
WAN and inter‑DC networking	Metro and long‑haul fiber, DWDM equipment, edge routers, private backbone upgrades	AI workloads move large datasets and models between regions and availability zones; cross‑DC bandwidth demand grows sharply with multi‑region training and inference.	Large cloud backbones already represent multi‑billion‑dollar capex programs; incremental AI‑driven capacity (fiber pairs, optical gear) can be hundreds of millions of dollars over a few years for a major provider.
Orchestration , MLOps, and control‑plane software	Cluster schedulers, container platforms, model registries, CI/CD for ML, usage metering/billing	To sell “AI compute as a service,” providers need sophisticated software to allocate GPUs, manage jobs, track utilization, and integrate storage/networking; complexity grows with scale.	Many platforms are internally developed; external software licensing and support can be on the order of 10–15% of AI infrastructure TCO over five years in some deployments.
Observability, telemetry, and optimization tools	Monitoring for GPUs, fabric, cooling, DCIM/BMS integration, AI‑driven optimization (e.g., cooling control)	Keeping thousands of GPUs fully utilized and within thermal/power limits requires deep telemetry and automated tuning, which are non‑trivial engineering investments.aegissoftt	Enterprise‑grade observability stacks and DCIM/BMS integration typically cost millions of dollars per large site over their life (licenses plus engineering and integration work).
Security and compliance	Hardware security modules, key management, secure enclaves, data loss prevention, access controls, audits	Enterprise AI workloads often involve sensitive data and regulated industries; clouds must harden AI clusters against exfiltration and meet compliance standards.	Security tooling and compliance programs add ongoing opex in the millions per year for large environments, plus capex for dedicated hardware and secure facilities.
Personnel and specialized operations	Site reliability engineers, network engineers (InfiniBand/HPC fabrics), MLOps teams, facilities engineers for liquid cooling	AI data centers need more specialized skills than traditional IT: tuning fabrics, managing liquid cooling, optimizing training pipelines, and running large clusters efficiently.	Personnel can represent 20–30% of AI infrastructure TCO over time; for hyperscalers, this means tens to hundreds of millions of dollars annually across global AI regions.slyd​
Support, maintenance, and spares	Hardware support contracts, spare parts pools, planned refresh cycles, vendor field engineers	High‑availability AI services require rapid replacement of failed components and regular firmware/software updates, increasing support intensity per rack.	Maintenance and support are often modeled as about 10–15% of AI infrastructure TCO across a 3–7 year horizon.
Land, water, and sustainability programs	Additional sites for new AI regions, water treatment/recycling for cooling, heat‑reuse infrastructure, carbon procurement	AI data centers often face local constraints on water use and emissions; providers invest in water‑efficient cooling, heat reuse, and carbon/renewable projects.	Water‑optimized cooling, treatment and heat‑reuse can add millions to tens of millions per site; broader sustainability and renewable programs for AI loads are multi‑billion‑dollar commitments across portfolios.

Bye Bye Marginal Cost Pricing

Marginal cost is the reason enterprise software economics have been changed by artificial intelligence. 

Simply put, the additional cost of supplying traditional software seat number 5,000 is very close to zero, once the software has been written and deployed commercially. That is why profit margins for software were traditionally so high. 

The same impact happens based on usage. The cost of supporting a very-active license user was not very different from supporting a light-usage customer or user. 

AI breaks that model as each inquiry involves additional real cost. So marginal-cost pricing doesn't perform as in the past. Instead, costs have to account for usage, since AI inference imposes real new costs.

source: Adeayo

Basically, infrastructure cost has become variable (at scale) for the first time in software-as-a-service history, since the cost of the next operation is not insignificant.

So AI inference costs have to be recovered in some way other than the licensed, per-seat mechanism. One way or the other, variable usage has to be accounted for and priced into the retail models.

In AI Era, Intent Replaces Software Infrastructure?

The process of booking a hotel, today and using an artificial intelligence agent, illustrates the reasons investors are so worried about AI disruption of enterprise software. At least in principle, lots of processes and functions that once were mandatory simply are abstracted. 

Activities are not properties of particular pieces of software but rather are composited on the fly and then deconstructed. Everything is custom, which might have been totally impractical before everyday AI and application program interfaces.  

source: Gabriyel Wong 

To some extent, we might say intent replaces infrastructure. You still need access to the capabilities. But those capabilities are invoked by the agent on your behalf. You don’t necessarily “own” the hardware, software, apps or operating systems that get invoked to complete your desired task. 

It’s analogous to the way cloud computing changed the necessity of “ownership” to reliance on “services.”

Two Thumbs Up for Project Hail Mary

Project Hail Mary really is this good. I read the book first, and except for time constraints, the screenplay is faithful. It's a just great fun and very entertaining. 

"Project Hail Mary" is Worth Seeing

If you are a fan of Andy Weir or science fiction, you'll love Project Hail Mary. It retains the book's hard science, problem-solving under pressure and humor.

The story is essentially a chain of scientific puzzles. Each solution creates a new constraint.

The tension comes from figuring things out.

It’s just a great story.

But the movie also is full of familiar themes about sacrifice, vocation, and what it means to “save” others.

The protagonist, Ryland Grace, is not a willing hero at first—he’s drafted into a mission to save humanity.

For those of you with some familiarly with the biblical stories of Jonah or even Moses, Grace resists the calling.

If his vocation is to save earth, he must do so despite fear or reluctance. His enthusiasm is not the point, but rather his obedience to a higher good, including the sacrifice he has to make, for others. Again, those familiar with the Christian or Jewish themes of sacrifice and substitutionary atonement will see the parallels.

Grace’s transformation from self-centeredness to love and service also is a story about redemption.

The relationship with the alien Rocky is key to the story and so you might also hear echoes of “love your neighbor” and questions about “who is my neighbor (the parable of the Good Samaritan), on a cosmic scale.

Some might find the idea odd, at first, but there also is a scientific parallel to the way Catholics are taught to approach faith itself. Some might argue faith is not central to scientific inquiry. 

It is a common misconception that science is a purely "objective" pursuit devoid of belief, while religion is a "blind" leap into the dark. In reality, both systems rely on a specific architecture. 

Science cannot prove itself using its own methods. To even begin an experiment, a scientist must operate on several unprovable assumptions that necessary are a form of intellectual faith:

• Uniformity of Nature: The belief that the laws of physics are the same in Tokyo as they are on Mars, and the same today as they will be tomorrow

• The Intelligibility of the Universe: The assumption that the human mind is actually capable of perceiving and understanding the "logic" of the cosmos

• Trust in Testimony: No scientist can replicate every experiment ever done. They must have "faith" in the peer-review process and the honesty of the global scientific community

• Predictability: If we don't assume the universe is predictable, the scientific method stalls before it starts.

Basically, science requires faith that the universe is orderly. You might think “reason” is an odd approach to religion (I sure did). But the Catechism has faith and reason in Part one, section one, chapter one. It is a foundational approach.  

Unlike "fideism" (faith without reason) or "rationalism" (reason without faith), the Catholic approach is based on “checks and balances:”:

• Without reason, human thought tends to slide into extremes either of Fideism (Faith without Reason) which leads to superstition, fanaticism, and "blind" obedience, or “rationalism” (what matters is only what can be measured). If faith doesn't have to be reasonable, then anything can be justified in the name of religion.

• God is rational (so the universe is ordered), not capricious

• Faith is informed by reason: Faith provides a broader context for the "why" behind the "how" that science discovers

• Non-Contradiction: Because God is the author of both the physical world (science) and revelation (faith), the two cannot truly contradict one another (science uncovers the footprints of God)

• Reason informs faith because the universe is a coherent whole.

If a scientific discovery seems to contradict a religious doctrine, the Catholic approach assumes either:

• The science is incomplete (the data is wrong), or

• Our theological interpretation is flawed (we are reading the scripture too literally or narrowly).

So Project Hail Mary is about the scientific method: Hypothesis;  testing; trust in unseen causes. But that is not so different from reasoned belief.

Obviously, Grace and Rocky collaborate to save two planets and life forms. So the notion of stewardship and salvation are involved. 

Even the name “Grace” is hard to ignore. In Christian theology, grace is a free, unearned gift. “Flawed” Ryland Grace “does the right thing” in spite of his initial instincts and preferences.

Project Hail Mary is a really-entertaining movie with notable religious themes:

• Salvation requires sacrifice

• Moral growth requires choice

• Love extends beyond boundaries

• Hope operates under uncertainty

• Why should we sacrifice for others?

• What makes a life meaningful?

• Who counts as “neighbor”?