Generative AI Productivity is an Issue, But So is Most IT

Quantifying or documenting generative artificial intelligence value is a top issue, respondents said in a Gartner survey of information technology executives. That really should not come as a surprise, as documenting the value of most technologies in knowledge or office work is challenging. 

And since generative AI is used for customer service interactions, producing summaries, developing code, drafting documents or messages, the issue is how well we can document the productivity lift from virtually any IT tool, in those instances. 

source: Gartner 

Quantifying the productivity gains from new IT solutions in customer service can be surprisingly challenging, experts often say. As applied to customer service agent operations, IT tools are said to improve customer satisfaction, handle volume fluctuations, and reduce training times. But isolating the impact on individual agent output can be difficult.

Generative AI and other IT might increase the number of customer contacts per hour, for example. Chatbots are a substitute for human agents as well, so might contain customer service costs. But that all hinges on the quality of the chatbot to answer the questions customers actually have.  

In addition, customer service involves interactions with various channels (phone, email, chat), making it hard to isolate the impact of IT on a single metric. Improved customer satisfaction might not directly translate to a quantifiable reduction in call times.

But that might not always correlate with improved ability to actually solve a customer problem. In other words, quantity is not the same as quality. 

Also, changes in productivity may not be immediate.  Learning curves, process adjustments, and cultural shifts within the team can take time to settle before the full impact is realized.

Accurately measuring before-and-after states requires clean data and proper attribution. Factors like seasonal variations, changes in customer behavior, or external promotions can skew the results. 

Demonstrating a clear return on investment (ROI) for new IT implemented in customer service can be challenging. Here's a breakdown of the difficulties:

Generative AI also might not eliminate tasks, but rather shift them. Increased efficiency in handling routine inquiries might free up agents for more complex issues, making it difficult to show a direct reduction in overall workload or quality of outcomes. 

Improved agent morale, reduced stress, and better customer experiences are all positive outcomes, but they're not easily captured in traditional productivity metrics like call resolution times.

Source	Author(s)	Publication	Key Findings
The Impact of Information Technology on Customer Service Productivity	Brynjolfsson, Erik, et al.	Management Science (1993)	Found that the impact of IT on productivity depends on the specific technology and how it's implemented. Identifying productivity gains requires careful analysis.
Does IT Really Pay Off? Measuring the Effects of Information Technology Investment on Customer Service	Lee, Sang-Pil, and Byung-Il Park	Journal of Service Research (2001)	Highlights the difficulty of isolating the impact of IT on productivity due to the presence of confounding variables. Emphasizes the need for a multi-faceted approach to evaluation.
The Challenges of Measuring the Business Value of Customer Relationship Management (CRM) Initiatives	Rust, Roland T., et al.	Journal of Marketing (2004)	Argues that traditional ROI metrics might not capture the full value of CRM systems, which often include customer satisfaction and loyalty benefits alongside productivity improvements.

We might observe similar issues with other tasks GenAI might help with, such as creating documents and text. Past applications of word processing arguably provide speed and quality advantages that are hard to quantify. 

Study Name	Venue	Date	Key Conclusions
"The Effects of Word Processing Software on Writing Performance" by Keith S. McNeil	Educational Technology Research and Development	1988	Found that word processing had a minimal impact on writing speed but improved editing efficiency.
"The Impact of Technology on Writing: A Review of the Literature" by Charles A. MacArthur, et al.	Review of Educational Research	2001	Concluded that the impact of technology on writing quality is mixed and depends on factors like task and user skill level.
"The Myth of Increased Productivity: How New Technologies Slow Us Down" by Sherry Turkle	Basic Books	2015	Argues that constant connectivity and information overload can hinder focused work and deep thinking, potentially impacting writing productivity.
"Beyond the Efficiency Paradigm: Rethinking the Role of IT in Knowledge Work" by Wanda J. Orlikowski	Organization Science	2007	Shifts the focus from measuring just efficiency gains to considering how IT can enhance creativity, innovation, and collaboration in knowledge work like writing.
"The Myth of Increased Productivity: How New Technologies Can Slow Us Down" by Daniel H. Pink	Harvard Business Review	2016	Argues that constant connectivity and information overload can actually decrease focus and productivity, even with advanced tools.
"The Paradox of E-Mail: A Sociotechnical Perspective on Communication Overload" by Stefan Klein, Christian Bartsch, and Jan Marco Leimeister	Journal of Communication	2004	Highlights how email, a seemingly efficient communication tool, can lead to information overload and ultimately hinder productivity.
"The Impact of New Information Technologies on Task Performance: A Meta-Analysis" by Steven E. Fiore, James R. Salas, Michelle H. Cuevas, and Cheryl A. Bowers	Human Factors	2003	Reviews multiple studies on the impact of technology on task performance and concludes that the effect can be positive, negative, or neutral depending on the specific task and user characteristics.