AI KPIs Will Evolve, as did Those of Internet Firms

Since artificial intelligence is so new as a driver of firm revenues, growth and valuation, professionals have few ways to model AI equity reward and risk, as was true for professionals in the early days of the internet. 

Traditional valuation methods, such as price-to-earnings (P/E) ratios and discounted cash flow (DCF) analysis for firms with negative earnings or a clear path to profitability.

Instead, assessments will have to turn on other more-subjective angles, such as business model “potential," traffic or user engagement, much as early internet-watchers had to rely on the number of unique visitors, the average time spent on site, and the click-through rate for advertisements. 

Click-through rates were important proxies for apps in the early days of the internet. And customer acquisition cost might be less important for AI firms than repeat usage (lifetime value). 

Model owners and suppliers of computing as-a-service will measure volume of transactions and data processed. 

But key metrics may evolve. In the early days of the internet, user engagement and traffic seemed more important. In 1995, for example, page views, unique visitors, time spent on site and click-through rate were arguably the most important.

With greater maturity and actual profits, standard metrics such as revenue growth, profitability, customer acquisition cost or customer lifetime value became relevant.

In its mature phase, active users and user engagement seem more important. So is loyalty, as often measured using the net promoter score. 

Internet, 1995	AI (2023)
Website Traffic: Measured the number of unique visitors to a website. Indicated the popularity and reach of the website and its potential to attract users and advertisers.	User Engagement: Measures the level of interaction and involvement of users with an AI-powered product or service. Indicated the effectiveness of the AI in providing value to users and its potential for long-term adoption.
Page Views: Measured the number of times a page on a website was viewed. Indicated the depth of user engagement and the potential for advertising revenue.	Data Volume and Processing: Measures the amount of data processed by an AI system. Indicated the system's ability to handle large amounts of data and its potential for generating insights and value.
Click-Through Rate (CTR): Measured the percentage of users who clicked on an advertisement on a website. Indicated the effectiveness of the advertisement in attracting user attention and driving clicks.	Accuracy and Precision: Measures the ability of an AI system to produce correct and consistent results. Indicated the system's reliability and its potential for generating valuable insights.
Unique Visitors: Measured the number of individual users who visited a website. Indicated the reach of the website and its potential to attract a diverse user base.	Revenue Growth: Measures the increase in revenue generated by an AI-powered product or service. Indicated the commercial viability of the AI and its ability to generate financial returns.
Time Spent on Site: Measured the average amount of time that users spent on a website. Indicated the level of user engagement and the potential for monetization through advertising or subscriptions.	Return on Investment (ROI): Measures the financial return generated by an AI investment. Indicated the value of the AI in driving business outcomes and profitability.
Customer Acquisition Costs (CAC): Measured the cost of acquiring new customers for an internet-based business. Indicated the efficiency of marketing and sales efforts and the potential for profitability.	Customer Lifetime Value (CLV): Measures the long-term financial value of a customer to an AI-powered business. Indicated the ability of the AI to retain and grow a customer base and generate recurring revenue.
Brand Awareness: Measured the recognition and perception of a company's brand among potential customers. Indicated the company's ability to attract and retain users in a competitive market.	Industry Recognition: Measures the recognition of an AI company's achievements and innovations within the AI industry. Indicated the company's reputation and potential for leadership in the field.

Presumably, AI firms will move through similar stages as they mature. Early on, analysts will try to quantify the value of intellectual property, data assets or the danger competitors represent. 

In some cases, volume of data processed, accuracy of AI models, or adoption of AI solutions by customers could be proxies for growth. Customer acquisition costs could be another metric when traditional metrics do not yet apply. 

Revenue mix or reliance on few or broadly-diversified customers might also matter. But key metrics will change as the AI industry matures. Then the more-traditional financial metrics will apply. 

Usage matters now, but engagement will matter later.

Cable TV and ISPs Still Get No Love from Customers

For as long as I can remember, and that is 40 years, cable TV services have been unpopular and unloved in customer satisfaction surveys. That same unhappiness now applies for internet service providers as well. One way of illustrating that is net promoter scores. 

The foundation of the net promoter score is the answer to the question “how likely is it that you would recommend this event to your colleagues and friends?” 

source: Customer Experience Update 

The score is calculated by comparing numbers of “promoters” to the numbers of “detractors.”

For example, if 10 percent of respondents are Detractors, 20 percent are Passives and 70 percent are Promoters, an NPS score would be 70-10 = 60.

source: Attendeaze 

NPS expected scores for events tend to be high, compared to many business-to-consumer products. An event tends to get 45 percent “excellent” scores, for example. 

source: Retently 

Are Satsified Customers More Loyal? Maybe Not.

Are “happy” customers “more loyal?” It might be hard to say. Satisfied customers--it often is believed--lead to loyal customers, which in turn leads to profits. 

Customer satisfaction typically is thought of as a predictor of customer buying intentions and loyalty, propensity to desert one provider in favor of another, account longevity, revenue per relationship and financial performance. That is perhaps one reason so many executives take stock in the net promoter score, a measure of customer satisfaction, in telecom and other industries. 

“Customer satisfaction is a leading indicator of company financial performance,” says the American Customer Satisfaction Index. “Stocks of companies with high ACSI scores tend to do better than those of companies with low scores.” 

But the relationships are not always so clear. 

“What we’ve found is that the relationship between loyalty and profitability is much weaker—and subtler—than the proponents of loyalty programs claim,” say Werner Reinartz, Professor of Marketing at the University of Cologne, and V. Kumar, executive director of the Center for Excellence in Brand and Customer Management at Georgia State University’s J. Mack Robinson College of Business.

source: Harvard Business Review

“Specifically, we discovered little or no evidence to suggest that customers who purchase steadily from a company over time are necessarily cheaper to serve, less price sensitive, or particularly effective at bringing in new business,” they argue. The researchers find “no evidence” to support such claims.

In fact, some would argue, some potential buyers should be actively discouraged. In the colloquial, “there are some customers you do not want.” 

source: Harvard Business Review

I’ve never been completely convinced that satisfaction and loyalty are related in a linear way, though. For starters, satisfaction and loyalty have different reference points. 

“Customer satisfaction is a self-reported measure of how much customers ‘likes' a company and how happy they are with goods purchased or services obtained from the company,” says Mark Klein, Loyalty Builders CEO. “Customer loyalty, on the other hand, is a company-calculated metric of likelihood to purchase again or not defect to a competitor.”

Also, customers can “like” a product and yet buy a competitor’s offering, without having any change in a “satisfaction” score. “Just because they’re happy with their current brand doesn’t mean they won’t switch if a lower price is offered elsewhere,” notes Actionable Research. 

Loyalty is what firms want, and satisfaction is seen as a proxy for loyalty. That might not generally be the case. 

But some question net promoter score relevance and predictive power, as popular as NPS is in many firms. “Two 2007 studies analyzing thousands of customer interviews said NPS doesn’t correlate with revenue or predict customer behavior any better than other survey-based metric,” two reporters for the Wall Street Journal report. “A 2015 study examining data on 80,000 customers from hundreds of brands said the score doesn’t explain the way people allocate their money.”

Of all the criticisms, lack of predictive capability might be the most significant, since that is what the NPS purports to do: predict repeat buying behavior. 

“The science behind NPS is bad,” says Timothy Keiningham, a marketing professor at St. John’s University in New York, and one of the co-authors of the three studies. “When people change their net promoter score, that has almost no relationship to how they divide their spending,” he said. 

Others might argue that social media has changed the way consumers “refer” others to companies and products. Some question the methodology. 

As valuable as the “loyalty drives profits” argument might be, it is reasonable to question how well the NPS, or any other metric purporting to demonstrate the causal effect of loyalty or satisfaction on repeat buying, actually can predict such behavior. 

Some argue that “satisfaction” might not predict very much. What might have predictive value is “totally satisfied” customers. Mere “satisfaction” is not predictive of loyalty, in other words. 

Xerox, for example, discovered that Its totally satisfied customers were six times more likely to repurchase Xerox products over the next 18 months than its satisfied customers. Merely satisfying customers is not enough to keep them loyal, in competitive markets. 

In other words, “satisfied customers” can, and will, defect. Totally satisfied customers tend not to churn, and tend to buy more from any supplier. 

source: Harvard Business Review

It might be hard to find anyone who believes there is no relationship between customer satisfaction and business outcomes. But the relationship might well be more complicated than we suppose. 

Effectiveness or Efficiency; Insight or Automation? What Do We Measure, and How Do We Apply What we Think We Know?

It is fair enough to argue that some organizations make better use of resources than others. It follows that some connectivity providers (mobile or fixed) will perform better than others, using any chosen set of key performance indicators, on business or network performance measures.

Firms also will tend to perform differently on different KPI measures, which can be set on business outcomes or network outcomes, for example. Financial metrics might be one thing, customer satisfaction or lifetime value another matter. 

Business KPIs can include revenue per account, revenue per cell site, subscriber counts, market share, market share growth, margin per customer or account or subscriber acquisition cost, churn rates, lifetime account value, inquiry conversion rates, capital expense per account or operating cost per account, for example. 

source: Sai Venkatesha

Consider the net promoter score, a measure of customer willingness to refer a company to others. Sometimes the scores are described on a scale of zero to 10, other times on a scale of -100 to +100. Either way, the NPS score tries to determine brand effectiveness at customer experience, boiled down to a willingness to recommend the product to others. 

Aside from the potential word of mouth impact, NPS studies tend to find that detractors spend less, promoters spend more. That stands to reason. Any customer that is unhappy will tend to look for other substitutes. Any customer that is quite satisfied will tend to keep buying, might tend to buy more and also look more favorably upon a brand’s other products. 

Having taken many such surveys, I would say the results are somewhat misleading.  Normally one is asked how willing one would be to recommend a product to others, or sometimes whether one would be proud to work for a certain company, or recommend that someone else work there.

It is a brute force methodology, in many respects. I might recommend a particular product, from a particular company to some people, and not to others. Some buyers are price conscious, some are value conscious and some are brand conscious, irrespective of price. 

Some products have a wide zone of tolerance--in terms of value or effectiveness--while others might require more discernment. Flour, sugar, gasoline or salt might not be products where brand matters. For a gourmet cook, spices and meats might be products where high discernment is required. 

Most people likely have high preferences in fragrances, clothing, personal electronics, cosmetics, pets,  transportation or housing. Most other products, though, might have high zones of indifference and high product substitutability. 

I would make different recommendations to each type of buyer, so I often question the relevance of NPS rankings. Some products have very high value for segments of the population; others have relatively low value, or no value. Many products arguably have high substitution potential, simply because the products are commodities, or are items about which a particular customer has no great preferences. 

source: Statista

“What to measure” and “what we can do with what we measure” are key issues for efforts to apply artificial intelligence to business processes and network management. Consider the discipline of AIOps, the effort to apply machine learning to network management. 

Whether AIOps is about insight or automation often is unclear, especially because the stated goal of AIOps to gain insights that lead to automated action, supervised or unsupervised by humans. Ericsson, for example, talking about applied artificial intelligence in communications networks, prefers the “automation” label. 

In large part, that might reflect the particular challenges of the connectivity service provider industry, where many revenue and business problems grow from unhappy customers and poor customer experience (dropped calls, no coverage, slow internet speeds, poor connection quality). 

“Previous Ericsson research shows that almost 50 percent of consumer network perception is based on personal experiences of the network, indicating the huge importance of network quality as the key to customer satisfaction,” Ericsson says. 

source: Ericsson

Customer satisfaction, in turn, is believed to be causally related to word of mouth referrals, prospect perceptions of quality and value, customer churn, customer acquisition and therefore revenue, profits and market share. 

“Whether customers give you US$2 or $20 ARPU, if you do not provide the quality they expect, they will churn,” says Vicente Cotino Director of Network Operation Maintenance, Orange Spain. 

To be sure, network performance is not the only driver of satisfaction. “At least one-third of the total NPS (net promoter score, a measure of customer willingness to recommend a product or service to others) score is derived from network performance.” 

Of the service providers surveyed by Ericsson, 80 percent use NPS as a key metric in operations. Also, several service providers indicate that 40 percent to 60 percent of operations key performance indicators are business-related.

Service providers, in other words, must do many things right, beyond ensuring that the network works as it is supposed to work. What is hard to untangle is the percentage of benefit that AIOps might provide, and whether it is automating changes or detecting issues that is “more important.” 

Clearly the two are related. Action is not possible without insight; nor is insight useful unless changes can be made fast. 

In surveys undertaken by Ericsson, 80 percent of respondents say automation is key for the cost and customer experience. About 90 percent of operations personnel say AI is important in protecting customer experience. 

As always, methodology matters. I do not know how the questions were framed. It is not clear how the responses might have been different if poll takers were asked about whether better network and consumer behavior insight and prediction would protect customer experience. 

But Ericsson frames the issue as one of “automation.” That might simply be a reflection of our general confusion about effectiveness and efficiency. People often speak about the importance of efficiency, which might be expressed as getting work done faster, with fewer people, at less cost or with fewer mistakes and rework. 

source: Insight Squared

Less common are evaluations of effectiveness, which might be expressed as “doing the right things” to produce value or desired outcomes. A traditional way of illustrating the difference is to note that an organization gains little to nothing by automating things that should not be done. 

In principle, AIOps as applied to a communications network requires automated insight to produce proactive network responses. So far, we seem to have gotten better at insight, while IT managers still debate the value of unsupervised action by AI-driven systems to correct and modify network operations. 

Still, there already are instances where radio resource management, for example, is made both more effective and efficient because AI allows the network to apportion capacity where it is needed most, “right now.”

It is inevitable that human managers are going to want to start small and apply AIOps in narrow areas, gaining confidence to apply more generally. Put simply, “nobody trusts the system to behave autonomously” at the moment.

Telecom Customer Satisfaction Might Not be as Bad as Some Think

Customer satisfaction with telecom services often measures lower than in other industries. But that does not necessarily mean customer satisfaction with telecom services is “poor.” To be sure, one net promoter score survey scored video subscriptions at “zero,” on average. That sounds terrible, but net promoter scores, a measure of customer satisfaction measured by willingness to recommend a firm, are measured on a scale of minus 100 to positive 100. 

So a score of zero is actually right at the edge of a  “good” score. Many consider any score up to 30 to be in the “good” category. That is not to say many connectivity services generally are considered “very good” by most consumers. Connectivity services typically do not have the highest industry scores. But a “good” score is not a bad thing. 

Also, there is some evidence that connectivity suppliers naturally spend more effort on their larger--and higher revenue generating--customers. One survey of NPS by Analysys Mason shows precisely that trend. Larger firms--assumed to be larger accounts--are more satisfied than small businesses, for example. 

Though mobile service provider satisfaction tends to be higher than that of fixed services, the same pattern holds: bigger organizations report higher NPS. 

source: Analysys Mason

Such low scores do not appear unusual. A 2018 analysys actually produced negative average net promoter scores for video subscription services, while mobile service providers averaged a 22 score.

source: Qualtrics

source: Qualtrics

How Important is Net Promoter Score in Telecom?

The net promoter score is considered useful as a predictor of potential revenue growth, the theory being that customers willing to recommend a firm are loyal, and therefore, repeat buyers. So the higher the net promoter score, the better positioned a firm is supposed to be, in terms of ability to generate a profit.

Bain and Company fellow Fred Reichheld, inventor of the net promoter score, an index of customer willingness to refer a product to others, once famously argued that loyal customers were more profitable. 

The argument is that loyal customers generate increasing profits each year they stay with a company, in part because they buy more, and because they impose fewer operating costs. They know how to use a company’s products, have figured out why they use a product and therefore are less likely to have questions about billing and other elements of the product experience. 

They also arguably make more referrals to others, which is what the NPS attempts to measure. In many cases, loyal customers might also be willing to pay a premium rather than switch. 

“In financial services, for example, a five percent increase in customer retention produces more than a 25 percent increase in profit,” Reichheld argued. 

But some question its relevance and predictive power, as popular as NPS is in many firms. “Two 2007 studies analyzing thousands of customer interviews said NPS doesn’t correlate with revenue or predict customer behavior any better than other survey-based metric,” two reporters for the Wall Street Journal report. “A 2015 study examining data on 80,000 customers from hundreds of brands said the score doesn’t explain the way people allocate their money.”

Of all the criticisms, lack of predictive capability might be the most significant, since that is what the NPS purports to do: predict repeat buying behavior. 

“The science behind NPS is bad,” says Timothy Keiningham, a marketing professor at St. John’s University in New York, and one of the co-authors of the three studies. “When people change their net promoter score, that has almost no relationship to how they divide their spending,” he said. 

Others might argue that social media has changed the way consumers “refer” others to companies and products. Some question the methodology. 

As valuable as the “loyalty drives profits” argument might be, it is reasonable to question how well the NPS, or any other metric purporting to demonstrate the causal effect of loyalty or satisfaction on repeat buying, actually can predict such behavior. 

For similar reasons, it might be fair to question relevance in some industries that habitually score at the very bottom of U.S. industries on NPS, such as internet service provider business or the cable TV business. Where NPS scores range from zero to 100, cable TV and ISP service ranks in negative numbers, 2019 U.S. NPS scores show. 

source: Satmetrix

One issue with the NPS is that some argue customer satisfaction is what is measured, not loyalty. The difference is subtle, but possibly important. 

Surveys have shown that even satisfied customers will switch brands. The point of loyalty is that customers show resistance to switching behavior. And some point out that only “complete satisfaction” is highly correlated with loyalty (repeat buying behavior). Merely “satisfied” customers arguably are as fickle as unhappy customers. 

source: Satmetrix

Those rankings are congruent with satisfaction surveys published by the ACSI, which show internet access and cable TV at the bottom of all industries ranked, virtually year after year. 

 source: ACSI

source: ACSI

Some might argue that the NPS or other measures of satisfaction are more important in highly-competitive industries, while of little use in monopolized businesses. This 1995 chart shows how little customer satisfaction mattered in the telephone business, then a monopoly. Whether very satisfied or completely dissatisfied, buying behavior was not affected. There were no choices. 

These days, as the telecom business is significantly competitive, we can argue about the importance of customer satisfaction, to a degree. Possibly nobody would claim customer satisfaction does not matter, as a contributor to customer loyalty (repeat buying). But neither is it completely clear how important satisfaction actually happens to be. 

Nor is it possible to divorce the importance of customer targets from the broader satisfaction measures. Any firm has to match its offers with the right audience, not just the right features and value proposition.

Targeting the wrong customers will generally fail, with high rates of churn and customer dissatisfaction. The oft-cited example is chasing price-sensitive customers who will quickly churn off once the discounts end. 

source: Harvard Business Review 

Customer satisfaction is not the same as customer loyalty, in other words. But it might still be argued that net promoter scores do matter within an industry, as a way of measuring performance against a firm’s competitors. In other words, it might well matter if Verizon’s service gets higher NPS than Comcast. 

Still, little research seems to have been done on circumstances when NPS actually is misleading or irrelevant. Industries that are declining might be an instance where even lower NPS or higher NPS scores do not matter much, as revenues are shrinking inexorably. At the margin, slower rates of decline are better than faster rates of decline, so higher NPS might have some value. 

Stil, if demand is declining, ultimately even high NPS does not matter. The market is shrinking, so high recommendations will not fundamentally change revenue prospects.