A new survey by the Wharton School (University of Pennsylvania) Human-AI Research suggests that enterprise leaders now use generative artificial intelligence routinely, for tasks including data analysis, document summarization, and document editing and writing.
Language models also are reported used routinely for information technology professionals writing code, human relations personnel for employee recruitment and onboarding and legal contract generation by legal personnel.
The survey found:
82 percent of respondents use Gen AI at least weekly
46 percent of respondents use it daily
89 percent agree that Gen AI enhances employees’ skills
As usage climbs, 43 percent see risk of declines in skill proficiency
72 percent are formally measuring return on investment (productivity gains,incremental profit).
The key caveat is that the ROI is based on hard-to-measure outcomes including employee efficiency, productivity, quality, creativity and security.
It might be fairer to characterize the findings as leaders subjectively believing there is ROI based on efficiency metrics, quality or creativity, but being fundamentally unable to measure such outcomes in a discrete way.
In other words, when choosing quantitative metrics, we have to assume the chosen metrics bear a direct relationship to outcomes, even when as nebulous as "quality" or "creativity." Such metrics as hours, clicks, speed, performance, impact, benchmarking or retention rates arguably can be measured quantitatively, but with often-subjective assessments required.
source: Wharton School, University of Pennsylvania
Also, in real life, it often is the case that multiple business functions (customer service, marketing, product development) are changed, simultaneously. It then becomes difficult to isolate the precise extent to which the chatbot, versus other factors (marketing campaigns or seasonal demand), is responsible for a positive shift in a proxy metric.
If a chatbot helps a firm draft a "better-quality" marketing email, and sales subsequently increase, it's hard to attribute the revenue gain solely to the chatbot's contribution to the email's quality. And how do we quantify “better?”
How do you assign a dollar value to an employee who is "more creative?" One might estimate the value of time saved, but not the value of the new ideas generated with the freed-up time requires a proxy metric. What is that metric?
If a new marketing campaign is launched, even with no increase in spending overall, but the campaign also uses new and different channels, how do we assess the contributions of the change in channels versus the “quality” of the chatbot-assisted campaign?
All such metrics require creation of “baseline” performance, often just as subjective as the possible improved outcomes.
Also, there is the time element. If the outcome is “better brand awareness” or “perception,” there is a lag time between initiative and outcome, even if most other elements of the marketing mix are held “constant.”
Most of us would agree that LLMs do save time. Using them does save time or increase speed. What is harder to estimate are elements such as the quality of output; the “creativity” or other outcomes; the value of rival work effort that is enabled.
The fundamental problem is that the “cost” can be quantified rather easily: (license fees, training costs, IT support). The outcomes tend to be softer and harder to measure, even when tracking employee time to complete specific tasks.
But that is true for all sorts of innovations, not just AI or language models.
No comments:
Post a Comment