New Study of Generative AI Impact on CSR Productivity

Though the general opinion these days is that applied artificial intelligence, in the form of generative AI, will likely improve productivity, that thesis has generally yet to be tested. But one new study of customer service operations at a Fortune 500 firm does provide some early indications of possible impact for CSR operations. 

In a study of generative AI at an enterprise software firm, when 5179 customer service agents used a generative AI-based conversational assistant, the tool increased productivity, as measured by issues resolved per hour, by 14 percent on average, with the greatest impact on novice and low-skilled workers, and minimal impact on experienced and highly skilled workers, the study found. 

The study, conducted by professors Erik Brynjolfsson, Danielle Li and Lindsey Raymond, was published by the National Bureau of Economic Research. 

“We argue that this occurs because ML systems work by capturing and disseminating the patterns of behavior that characterize the most productive agents,” say the authors. That makes sense. What the engine apparently did was distill best practices and make them consumable by agents who were less skilled. 

In other words, “high-skill workers may have less to gain from AI assistance precisely because AI recommendations capture the potentially tacit knowledge embodied in their own behaviors,” they note. 

“The AI tool helps newer agents move more quickly down the experience curve: treated agents with two months of tenure perform just as well as untreated agents with over six months of tenure,” they say. 

Very few studies of generative AI actually have focused on outcomes in an applied setting, as most prior studies attempted to argue that benefits could be obtained, or should be gotten. 

Study Title	Authors	Study Publisher	Date of Publication	Measured Outcomes
The Potential for Generative AI to Transform the Research Industry	Amodei, Dario, et al.	arXiv preprint arXiv:1706.03762	2017	Increased productivity, accuracy, and efficiency in research
The Economic Impact of Generative AI	Chui, Michael, et al.	McKinsey & Company	2020	Increased annual global GDP by 7%, increased annual US labor productivity growth by 1½ percentage points, and increased investment in AI by 1% of US GDP
The Rise of Generative Artificial Intelligence and Its Impact on Education: The Promises and Perils	Lane, Rebecca, et al.	Nature Biotechnology	2021	Revolutionized research practices, accelerated innovation, made science more equitable, and increased the diversity of scientific perspectives
The Environmental Impact of Generative AI Tools	Kay, Michael, et al.	Nature Machine Intelligence	2022	Four to five times higher carbon footprint than a simple search engine query
The Rise of Generative AI	Hariharan, Karthik, et al.	J.P. Morgan	2023	Reduced the money and time needed for content creation, bred innovation, and led to new business models and applications

AI Will Bring Less Change Near Term Than We Think

There are good reasons why generative AI will get commercial traction faster than AR, VR or XR: cost, ease of use and scalability. 

Broadly speaking, the cost to create a commercial use case, at scale, is far easier with generative AI. 

Generative AI is software-based, and can be used with virtually any existing application, to add content creation; support or code-writing tasks to any existing app. That means the time to deploy and cost to deploy--while far from insignificant--can rely on existing app use cases and deployed instances. 

Any form of “Metaverse,” AR, VR or XR apps require new specialized hardware, generally are not “mobility enabled” and also require creation of new apps and ecosystems. That takes time and money. 

So generative AI is easier to create and deploy and easier to use. It requires no new hardware; no new behavioral changes; no new applications. It simply adds features to what already exists. 

Since generative AI is essentially a “bolt on” for existing use cases and apps, it can scale quickly. 

Still, some patience will be required, as at-scale commercial use cases will develop more slowly than most expect, even if AI scales faster than XR, VR or AR and metaverse, for example, though interest in metaverse will return eventually.

I learned early in my career making forecasts that it is better to conservative in the early going. Humans nearly always tend to overestimate the near-term impact of any technology and underestimate the long-term impact. 

“We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run” is one way of stating the principle. So is “We always overestimate the change that will occur in the short term and underestimate the change that will occur in the long term.”

Or, “People overestimate what can be done in one year, and underestimate what can be done in ten.” All three statements capture the wisdom of how significant new technologies create change. 

There is a bit of business wisdom that argues we overestimate what can be done near term, but underestimate the long term impact of important technologies or trends. The reason is that so many trends are an S curve or Sigmoid function. 

Complex system learning curves are especially likely to be characterized by the sigmoid function, since complex systems require that many different processes, actions, habits,  infrastructure and incentives be aligned before an innovation can provide clear benefit. 

source: Rocrastination 

Also, keep in mind that perhaps 70 percent of change efforts fail, the Journal of Change Management has estimated. We might then modify our rules of thumb further, along the lines of “even as 70 percent of innovations fail, we will see less change than we expect in one year and more change than we expect in 10 years.” 

At least in part, technological impact increases over time for reasons of diffusion (what percentage of people use the technology regularly) as well as enculturation (it takes time for people and organizations to figure out how to best use a new technology). 

Impact arguably also increases as the ecosystem grows more powerful, allowing many more things to be done with the core technology.

Bundle Pricing Complicates Cost Estimates

It never is easy to figure out what customers actually are paying for their connectivity services. 

“Among the 18,359 consumer bills on which an internet price could be identified, the median cost of high-speed internet service was $74.99 per month,” says Consumer Reports. About half of the households were paying between $60 and $90 per month.

About all we can say, based on that study, is that U.S. households who buy a discrete home broadband service, rather than a bundled service, pay about that much. But most households do not buy their home broadband service that way. 

It is possible that 40 percent to 60 percent of U.S. households, for example, buy bundled services where home broadband is a component, and pay a discounted price, compared to the stand-alone retail price.

Study	Percentage of Homes Buying Service Bundles
Leichtman Research Group (2020)	58%
Strategy Analytics (2021)	63%
eMarketer (2022)	65%
Parks Associates (2023)	67%

A 2022 Pew Research survey found that about 40 percent of U.S. customers purchased a services bundle. 

The latest survey by Leichtman Research suggests 46 percent of households buy a bundled service. 

And since the point of buying a bundle is price discounts, about all we can say is that most U.S. households likely spend less than reported by Consumer Reports. 

Complicating matters further, customers are buying service plans offering higher speeds than they used to, and typically paying more for those plans, even if we adjust prices for inflation effects.