"Five Nines" Availability Might Not be Possible Anymore

In the internet era, something profound has happened to traditional connectivity provider thinking and spending related to network service availability. Something equally important has happened to consumer expectations about service or app availability, as well. 

Service and app providers using the internet as the access platform generally cannot expect to reach 99.999 percent availability. Neither can mobile services of any type. Virtually all fixed network services except perhaps voice also now will fail to reach 99.999 percent availability. 

The reason is simply the math. Availability of any single device, in principle, could reach 99.999 percent availability, though that is not true of routers or servers. End user devices also never reach 99.999 percent availability. 

But system availability is the sum of every single device in the delivery chain. And even if every device and network element had 99.999 percent availability (“five nines”), reliability shrinks with the number of devices. 

It is demonstrably not possible for most end user internet experiences or applications, almost certainly. In the transport business, 99.999 percent availability is probably possible only on the portions of the network with the fewest total network elements, end to end. 

As a rule, I think we safely can assume that any internet-delivered will not ever reach 99.999 percent availability at the user level. It also is safe to assume that mobile voice, text messaging and internet access virtually never reach 99.999 percent. Possibly only legacy fixed network voice can hope to do so. 

It goes without saying that the Internet is a complex system, with lots of servers, transmission paths, networks, devices and software all working together to create a complete value chain.

And since the availability of any complex system is the combined performance of all cumulative potential element failures, it should not come as a surprise that a complete end-to-end user experience is not “five nines.”

Any end-to-end network, and the applications delivered over that network, have many single points of failure. Even if most network element has five nines levels of availability, and there are just eight total elements in series, with one device having 85 percent availability, and one device has 95 percent availability, total end-to-end system availability is just 59.87 percent. 

That is calculated as 85%*90%*99.9%*98%*85%*99%*99.99%*95%, or 59.87 percent. 

Redundancy is the way performance typically is enhanced at a data center, by a transmission network or application. But the total number of network elements for any single app also is going to be more than eight. 

Component	Availability
Web	85%
Application	90%
Database	99.9%
DNS	98%
Firewall	85%
Switch	99%
Data Center	99.99%
ISP	95%

So one way of thinking about reliability or availability is that modern application delivery systems cannot actually meet the old “five nines” standard, end to end. There are simply too many network elements involved in any end-to-end system. 

The business implication is that “five nines” increasingly is not possible. 

But consumer expectations also are involved. Consumers know mobile phones are not as reliable as fixed network phone service. They are willing to make the trade off. They know computers, operating systems and web apps are not “five nines” available. They know they will have to reboot on occasion.

The business implication is that consumers can live with “less than 99.999 percent availability if value is high enough. The corollary is that they also might not be willing to pay what it theoretically costs to move closer to five nines availability. 

The bottom line is that five nines cannot be provided, nor will consumers pay for it. Some level of availability less than 99.999 percent is acceptable and even expected. The question is how much availability is required to keep a user satisfied, as they might not be willing to pay much more--if anything more--to get that level of availability. 

Much More AI Use in Telecom in 5 Years

If a recent forecast by Omdia is correct, telecommunications service providers will be spending quite heavily on artificial intelligence software over the next five years. Though AI will be a bigger capability for virtualized networks in the future, the current use cases often focus on customer service, especially call centers. 

source: Omdia 

AT&T, for example, uses AI-assisted systems to optimize technician schedules, minimizing commute time, for example. AT&T reportedly recorded a seven percent reduction in miles traveled per dispatch and a five percent increase in productivity.

The AI incident management process also uses AI to detect network issues in real-time, even before the customers are aware there is a potential problem. It is said that AT&T has the ability to manage about 15 million alarms per day.

Users Will Migrate to 5G, Maybe Because of New Value, But More Likely Because of Supplier Push

End user demand for various connectivity services changes over time, but typically when some new need arises. But sometimes supplier push drives the change.

That might be the case for 5G, early on. Since 5G requires new devices, 5G device availability, especially for the most-popular brands, is a key requirement for higher 5G adoption. It is less clear that any immediate new use cases will be obvious.

Changes in demand have many drivers other than new needs, of course. At other times, revolutionary changes in supplier policy can shift demand quite radically. 

There was a time when a deliberate pricing strategy by AT&T mobility drove a huge amount of product substitution, literally leading to huge use of mobile calling as a direct substitute for long distance, and also leading to mobile substitution for local telephone service as well. 

 source: Telecom Policy Institute

Few now recall it, but AT&T Wireless once drove rapid mobile adoption by consumers by revolutionizing the way consumers paid for--and how much they paid for--long distance calling.

Back in 1998, AT&T unveiled a major new pricing plan for mobile users that priced all domestic calls for 10 cents a minute. That Digital One Rate plan effectively erased the distinction between local and long distance calling and provided a major incentive for consumers to buy mobile service.

Even many astute industry watchers did not appreciate the revolutionary nature of the plan. Technology pundit Walt Mossberg once called Digital One Rate “marketing hype.”

“It promises to make your cell phone as simple and affordable to use as a landline phone, so that you'll use it even for casual calls without a second thought,” said Mossberg. “The actual service behind the marketing isn't good enough to really allow that. I've been frustrated with it again and again.”

Digital One Rate was anything but hype. It really did revolutionize the way consumers and businesses made calls. It really did begin the mass adoption of mobile phones and the demise of landline service.

Beyond that, Digital One Rate changed the key profit drivers in telecom, from long distance revenue to mobile service. Within a decade, the industry profit driver changed from long distance to mobility. The shift from voice to data would follow.

In the ensuing years after Digital One Rate was introduced, long distance revenue plummeted, as did use of landline phone services. Digital One Rate was the driver, as the whole industry shifted pricing and packaging.

It is unclear how much 5G can shift end user demand. It is clear that at least some mobile service providers in the U.S. market are going to try and drive a substitution of mobile broadband for fixed network access. It is not clear whether end user demand shifts will mostly be driven by end users or suppliers. 

Some of us can remember the new needs that caused us to upgrade to specific devices, and then to web-centric devices. I actually do not recall which 2G device I was using when the “need” for a Blackberry happened. Odd enough though it will seem now, “mobile email” was a huge draw, providing enough value to make it an easy choice. 

I also remember when the Blackberry became a barrier. There was a time when smartphones did not routinely offer access to turn-by-turn directions, and for no subscription fee. That alone was the killer app that made the Blackberry suddenly less capable than I needed, and made the value of a 4G device much more attractive. It meant I did not need a discrete GPS device, with a required subscription. 

I had used 3G dongles for wireless internet access when on business travel, as painful as it was, experience wise. The shift to a 4G dongle was a “no brainer” decision because the user experience was so much better. 

The visual web made routine upgrades of my fixed network, to get higher speeds, a logical decision. It has been years since download speed actually has been an issue, though, as speeds and pricing have been more than adequate for all my use cases. 

Work from home has not been a particular issue, as I have worked remotely, and out of my home, for most of the last 25 years. But what is new is the amount of time I spend on videoconferencing services. And since my connection is asymmetrical, with what now insufficient bandwidth for consistent user experience. 

The new need is “more upstream bandwidth,” for the first time. At least so far, my 4G experience is not a pain point, as I tend not to do work videoconferencing on the mobile device. 

In all those instances, a shift to different or new services, though anecdotal, was driven by a change in demand for features or services. 5G will be the same sort of thing for most people, and might initially be driven simply by the need for a new device that just happens to use the 5G network. 

Are Happy Workers More Productive? Maybe Not

Are happy workers productive workers? That always seems to be the assumption, just it always seems assumed that “better broadband produces higher economic growth.” But it that a reasonable assumption? Yes, most studies suggest. But some question the theory that happiness leads to productivity.  

Industrial psychologist Fredrick Herzberg is among those who might argue the link is not as causal as most of us believe.

One issue is what we actually are measuring, though. Is it morale, engagement, employee satisfaction scores, fulfillment, or something else that is the proxy for “happiness.” Sometimes such measurements look at job satisfaction. 

Other times “well being” is what researchers try to measure. By definition, well being involves not only the job, but also everything else in a person’s life. In other words, it might be possible that people are happy for other reasons, or engaged for other reasons, also are able to produce more effectively. 

At least in principle, organizations might have a mix of happy, yet low-performing employees, as well as unhappy low-performing employees. Organizations might also have unhappy high performers or happy high performers. 

source: Professor Charles D. Kerns 

Though studies suggest all are correlated with higher productivity, the empirical results vary based on which proxy is used.  

But some argue--perhaps counterintuitively--that happiness doesn’t necessarily lead to increased productivity. 

“A stream of research shows some contradictory results about the relationship between happiness — which is often defined as “job satisfaction” — and productivity,” according to researchers André Spicer, professor of Organizational Behavior at Cass Business School in London and Carl Cederström professor of Organization Theory at Stockholm University.

“One study on British supermarkets even suggests there might be a negative correlation between job satisfaction and corporate productivity,” they note. The more miserable the employees were, the better the profits, they note.

But it also is fair to point out that the direct relationship between performances and positive attitudes has not been proved, at least not yet. 

The belief that happy workers “must be” more productive workers is widespread. Most of us instinctively wish to believe it. The correlation might generally exist (though the causal relationship remains arguably unproved). 

Correlations might be enough, though, in the sense that people act as if the statement were true, regardless of fact.  

Most of us act as though “broadband causes economic growth.” Correlations obviously exist. But nobody yet has been able to prove causation, as there always are other explanations. 

Places with high economic growth tend to produce wealthier and better-paid employees, who are in position to pay for better broadband. Places with high employment of knowledge or information workers likewise might also be places where demand for faster broadband is highest, both for reasons of “we can afford it” as well as “we prefer and want it.” Higher income is always correlated with higher demand for quality broadband. 

Such places also tend to be populated by people with advanced college degrees, and that also correlates with demand for more-costly or faster broadband. Places with more younger people have higher broadband demand than places populated with older people. 

Demand for better or more-costly broadband also correlates with workforce status. Retired people tend not to have as much demand for the “fastest” services, which also are the most costly. People still at work have higher levels of demand. 

We might generally prefer that employees be happy, for all sorts of reasons. Whether they also are more productive for that fact is what is harder to prove, especially since there is no unquestioned way to measure “happiness” directly. What we actually can measure are proxies for “happiness.” 

And we might even have that wrong. The point is that we actually cannot “prove” that happier workers are more productive. In extreme cases, the “happiest” employees might be those who are paid generously, yet produce questionable results, and are not expected to do so. 

Few of us would question the likely negative impact of unhappy workers at any workplace. But it is also possible to argue that removing the sources of unhappiness has value. 

That arguably is a different process from fostering higher productivity. Low pay might make you unhappy. But high pay might not produce long-term happiness or productivity. Unsafe work conditions might make you leave a job. That does not mean making the workplace safe necessarily increases one’s productivity. 

Think of “unhappiness” and “happiness” as being on two separate scales, not one. The task is remove sources of unhappiness and low productivity on one scale, and hopefully increase happiness or productivity on the other scale. 

Industrial psychologist Frederick Herzberg is known among organization theorists for his motivation-hygiene theory of satisfaction and dissatisfaction. Put simply, the opposite of satisfaction is no satisfaction.

The opposite of dissatisfaction is no dissatisfaction. Satisfaction and dissatisfaction are in separate dimensions.

Is More Facilities-Based Competition Coming for Australia NBN?

National telecom infrastructure policy in big countries is an example of turning a huge ship in a new direction: it takes time. Consider Australia’s NBN. Over the last decade the national broadband network--designed as a single wholesale network--has faced acrimony, slower than expected build rates and potential policy reversals. 

Only now there are moves toward facilities-based competition that tend to raise the question of whether the plan to build a wholesale network was the right choice. 

On the face of it, the NBN’s new plan to lower prices for business customer connections in suburban areas is a simple reflection of growing competition from facilities-based firms. 

The move likely is a response to growing facilities-based competition, from municipal broadband to 5G offers from Telstra. After a seemingly endless struggle to get the NBN designed, funded and built, at least some officials believe facilities-based competition should be reintroduced.

Traditionally, wholesale facilities have made sense where there is no easy way to enable rival physical networks, for whatever reason. Under such circumstances, regulators often have opted for one physical network that sells wholesale service to any telecom services retailer. 

The stated advantages include lower overall capital investment and possibly faster time to deployment. The downside, some of us would note, is a loss of innovation and differentiation possible when different facilities-based networks compete with each other. 

Perhaps the best examples so far are the role of cable operators in the broadband business. In the U.S. market, for example, cable operators have 70 percent of the installed base and get virtually all the new net additions in the fixed network broadband business, and have done so for years.

 Possibly the next examples are entry by mobile operators, in the 5G era, into the home broadband business, using fixed wireless for higher-bandwidth use cases but even standard 5G for many users. 

In 2019, about 19 percent of customers were mobile-only for internet access, for example, according to the Australian Communications and Media Authority. That percentage is expected, in some quarters, to increase over time as 5G and following mobile networks increasingly are able to supply bandwidth enough to substitute for a fixed connection in a growing number of use cases. 

source: ACMA 

  

As part of the new plan, NBN says it will invest up to $700 million to create up to 240 NBN Business Fiber Zones across Australia, in 85 regional areas. The effort includes financial incentives for its retail partners, in the form of waiving the typical charges to build out an access lateral.

NBN says that all businesses within these zones will have access to Enterprise Ethernet, at significantly reduced wholesale prices. In total, these zones are expected to cover more than 700,000 businesses, NBN says. 

Businesses in Business Fiber Zones also will see Enterprise Ethernet pricing reduced, some by up to 67 percent. 

Enterprise Ethernet is NBN’s fastest symmetrical wholesale product and premium-grade business offering. It has options for prioritised traffic, high capacity and symmetrical upload and download wholesale speeds from 10Mbps to close to 1Gbps.

NBN also says that when an internet retailer places an order for Enterprise Ethernet, for an estimated 90 per cent of business premises in the national NBN network footprint, NBN will not charge the retailer for building the connection to the customer, it seems.

If an internet retailer signs up for a three-year Enterprise Ethernet plan, NBN will not charge the retailer an up-front connection cost.

Change or Die? The Advice You Always Hear

At a recent session of the PTC Academy, APTelecom President Sean Bergin starkly contrasted the choices connectivity providers (telcos and others) now face. As demand for traditional connectivity services continues to decline, service providers face two basic choices: stick to the core business or create a new business. 

The former strategy includes options such as reducing operating costs, perhaps making acquisitions to gain scale or taking other steps that lower costs to match declining revenues, thereby sustaining profit margins. 

The latter strategy, admittedly riskier, requires creation of new products beyond the current offerings, and almost always involves some degree of movement into different parts of the ecosystem or value chain.  Think of the former as “role” (connectivity, device, application). Think of the latter as “function” (semiconductors, devices, connectivity, platform, app, business model) within any single role. 

As a practical matter, not every firm in the connectivity business can actually “change roles or functions.”  Doing so requires new competencies, capital, different supply chains and often different sales channels as well. 

source: IP Carrier

Of course, telcos are not the first firms ever to confront operating in a declining or changing business. There are some strategies that make sense, if they are not pleasing or even easy. 

Sometimes firms must withdraw from parts of the market where they are not the present market share leaders and are  unlikely to be leaders. They can reduce costs and find other ways to be more efficient about core business processes. They can try to sell greater volumes or complementary products to customers, or acquire rivals to increase scale, which offers a chance to reduce costs. Sometimes, expansion beyond the current service boundaries is possible. 

source: Julius Bailey

Sometimes a firm can focus on a still-growing part of the portfolio, but only when growing segments are available. Beyond that, specialization in a more-defined portion of the present business might be possible. 

source: Julius Bailey

Looking at some of the business or consumer opportunities one often hears suggested, you can see immediately that nearly all the proposed opportunities require moving outside the core connectivity competence. Even some of the “core competency” opportunities, such as over the top messaging or voice, have proven quite problematic.

source: A. Saghaeian

To be sure, telcos have shifted voice operations from analog to digital, using VoIP to support their remaining voice operations. But has that boosted revenues? In virtually all cases, the answer is no. Text messaging, at the same time, has continued to generate less revenue. every year, as customers choose messaging alternatives.

Two decades ago, one frequently heard advice to “partner with OTT providers.” What the means always requires some explanation, but the advice was to partner with the likes of Skype and others as a way of either entering the VoIP market or at least trying to protect some voice revenues. 

You can judge for yourself whether that has worked. On a global basis, service revenues from fixed voice services are expected to decline at a compound annual growth rate of -5.64 percent between 2018 and 2023, with service provider revenue dropping from $142 billion in 2018 to $106 billion in 2023, according to Omdia.

Globally, fixed voice subscriptions will fall from 931 million in 2018 to 821 million in 2023, a CAGR of -2.49 percent, driven by fixed-to-mobile substitution.

The big problem is not only that OTT providers take so much of the growth, or that they take so much revenue from mobile or fixed service providers. Rather, the big problem is that OTT messaging and voice essentially destroys the service provider markets for voice and messaging. 

Text messaging revenue could decline 40 percent over the next three or four years in the European and Middle East markets, many executives predict. About 84 percent of respondents to a Telco 2.0 survey thought the main reason for such declines was the expected price reductions mobile service providers would adopt to compete with OTT services and apps.

The point is that survival, for many firms, might well hinge on essentially “getting into a new line of business or getting out of the business.” That is one reason why connectivity provider consolidation is expected to be massive. 

Either major choice--sticking with the core competency and current services or moving elsewhere--carries great risk. In the former case, your business continues to shrink and you eventually exit the market, through sale or death. In the latter case you run a great risk of failing and likewise exiting the market, through bankruptcy or sale. 

But only one of those paths normally leads to sustainability, and that is the “up the stack” or “across the ecosystem” strategy.

Latest FTTH Stats from A.D. Little

As a rule, the countries with the highest fiber to home levels tend to be small city-states or  smaller countries. But among the top-10 globally are several medium-sized countries as well, A.D. Little reports. 

source: A.D. Little 

As always, there is a difference between making a product available and propensity to buy. Take rates may hinge on competition in each market, government support and regulatory framework. Faster deployment arguably is possible with wholesale models where all retail internet service providers use one physical network.

source: A.D. Little

The choice of physical media, measured here, is important, but an incomplete indicator of bandwidth. There was a time when FTTH meant 10 Mbps speeds. These days, speeds can be almost arbitrarily high.