Wednesday, November 20, 2024

It Will be Hard to Measure AI Impact on Knowledge Worker "Productivity"

There are over 100 million knowledge workers in the United States, and more than 1.25 billion knowledge workers globally, according to one Anatomy of Work estimate. And “work about work,” including unnecessary meetings, status checks and information searchers occupy as much as 60 percent of knowledge worker time. 


Hence the interest in AI agents that can conduct activities autonomously, presumably eliminating much of that “work about work.” Customer support, regulatory compliance, security and marketing are areas where agents are expected to contribute. 


To the extent we can measure knowledge worker productivity--and there is an argument to be made that we actually cannot measure it effectively--efforts to boost knowledge worker productivity since 2019 have been quite minimal. 


Title

Date

Publisher

Key Conclusions

How Do You Measure Knowledge Worker Productivity?

N/A

Serraview

- Outputs are intangible and difficult to define

- Results often based on team output rather than individual

- Companies not necessarily tracking hours for salaried employees

- Time spent working increasingly blurred in mobile workforce

The Best Way to Measure Knowledge Worker Productivity

2022

Maura Thomas

- Quantitative metrics don't help in short term for knowledge workers

- Qualitative metrics matter most in short term

- Best measure is questioning how employees feel about their work

How to Measure Employee Productivity in the Workplace

2024

Robin

- Knowledge work is intangible and difficult to categorize

- Existing productivity measures rooted in 'machine age' organizations

- Impossible to come up with single universal measure for knowledge worker productivity

Knowledge Worker Productivity

N/A

mediaX at Stanford University

- Serious productivity gap exists between available knowledge and how it is used

- Many enterprises fail to fully engage energy and intellect of employees

Research: Knowledge Workers Are More Productive from Home

2020

Harvard Business Review

- Knowledge workers' inputs and outputs can't be tracked like other workers

- They apply subjective judgment to tasks and decide what to do when

- Can withhold effort often without anyone noticing

"Boosting the Productivity of Knowledge Workers"

2006

McKinsey

Measuring productivity is complex due to the amorphous nature of tasks. Performance is constrained by physical, social, and contextual barriers. Interactions account for a significant portion of work, making targeted improvements essential.

"Measuring Knowledge Worker Productivity: A Taxonomy"

2004

Emerald Group Publishing

Identifies a lack of universally accepted metrics for productivity. Proposes a taxonomy categorizing productivity dimensions and highlights critical areas for future research.

"Rethinking Knowledge Work: A Strategic Approach"

2021

McKinsey

Structured approaches (e.g., workflow systems) improve some metrics but can undermine autonomy and collaboration. Freeform and creative aspects of work are harder to quantify.

Knowledge worker productivity: is it really impossible to measure it?

2021

ResearchGate

Argues that traditional productivity metrics are inadequate for knowledge workers and proposes a new methodology based on human capital efficiency.

Broken Speedometers: Quantifying Knowledge Worker Effectiveness?

July 1905

VeraSage Institute

Highlights the challenges of measuring knowledge worker productivity due to the intangible nature of their work and the limitations of traditional metrics.

What Really Matters for Knowledge Worker Performance

July 1905

Allsteel

Reviews existing research and concludes that a single, universal metric for measuring knowledge worker productivity does not exist.

Measuring Knowledge Worker Productivity

July 1905

Global Workspace Association

Discusses the complexity of measuring knowledge worker productivity and the limitations of traditional methods.


Knowledge workers are those who “think for a living,” making productivity challenging to measure. According to the U.S. Bureau of Labor Statistics, measuring employee productivity means calculating “output per hour” of work. 


But how does one quantify outputs, which often are intangible and difficult to define. Also, when “teams” produce the outcomes, how can individual contributions be assessed?


And there are other complications, such as quantifying “hours worked,” either in-office or remotely. 


None of that stops government agencies from doing their best to measure and quantify knowledge worker productivity. 


For example, total factor productivity in the United States is said to have grown 0.8 percent from 1987 to 2023, but only 0.5 percent from 2019 to 2023, according to the Bureau of Labor Statistics


All of which will raise questions when firms and entities start to report “productivity” gains from using AI.  If all we can be sure of is that we can measure or quantify some outcomes we believe to be measures of output.  


Whether that output actually represents knowledge worker productivity is less certain. Most of us would be circumspect about metrics such as “hours worked” or “email volume” or “meeting attendance.”


We’d probably have some greater confidence about tasks completed, revenue generated by a team, assuming it can be identified. 


But lots of common metrics are only quantitative, and cannot measure the quality of work performed or outcomes. People can produce lots of documents, lots of code or “ideas,” but it is hard to measure the quality of those outputs. 


Proxy Measure

Description

Time spent working

Hours logged or time tracked on tasks14

Email volume

Number of emails sent/received5

Meeting attendance

Number of meetings attended or hours in meetings

Task completion

Number of tasks or projects completed4

Revenue generated

Financial output attributed to individual or team2

Information searches

Time spent looking for information15

Internal collaboration

Time spent working with colleagues5

Documents produced

Number of reports, presentations, or other deliverables created

Client interactions

Number of client meetings or calls conducted

Ideas generated

Number of new ideas or innovations proposed


Content Licensing Deals to Train AI Proliferate

As has been the case for earlier generations of conflicts between content owners (media firms, for example) and new types of firms (search, social media), conflicts over the training of large language models is being resolved in similar fashion: licensing deals. 


Microsoft, for example, recently signed a deal with News Corp.’s Harper Collins allowing “select non-fiction back titles”  to be used for training of artificial intelligence models, if individual authors agree. 


The content is said to be for a new model Microsoft is creating, but not intended to “write books.”  


Such deals have become more common as model owners work to defuse content owner objections to AI training using their copyrighted works. 


Content Owner

AI Company

Deal Details

Payments

News Corp

OpenAI

5-year deal for access to current and archived content from publications like The Wall Street Journal, The New York Post, The Times, etc. Includes display of content in response to user queries and sharing of journalistic expertise56

Over $250 million over 5 years56

Various Publishers

OpenAI

Annual licensing deals for training AI models, including companies like The Associated Press, Axel Springer, Prisa Media, Le Monde, and Financial Times56

$1 million to $5 million per year136

The Atlantic

OpenAI

Access to archives for AI model training and collaboration on product development, including an experimental microsite6

Not specified

Vox Media

OpenAI

Access to archives for AI model training and assistance in creating products for consumers and advertising partners6

Not specified

Hearst

OpenAI

Licensing deal for content use in training AI models2

Not specified

Mumsnet, The Center for Investigative Reporting

OpenAI

No deal; instead, these entities have initiated legal complaints against OpenAI2

-

Conde Nast, NBC News, IAC (People and Daily Beast owner)

Apple

Discussions for licensing content archives for AI training, but no public deals announced yet2

At least $50 million over a multiyear period (reported offer)3

Financial Times, Axel Springer, The Atlantic, Fortune

Prorata.ai

Licensing deal with revenue-sharing model; 50% of subscription revenue shared with content creators2

Revenue-sharing basis

Time, Der Spiegel, Fortune, Entrepreneur, The Texas Tribune, Automattic (WordPress.com owner)

Perplexity

Revenue-sharing deal with access to analytics and technology for creating custom answer engines2

Revenue-sharing basis

Reddit

Google

Licensing deal for user-generated content to train AI models4

Not specified


source: Seeking Alpha 



Content Owner

AI/Search/Social Media Firm

Deal Details

Payments

News Corp

OpenAI

Access to current and archived content from publications like The Wall Street Journal, The New York Post, The Times, etc. for training AI models and displaying content in response to user queries. Includes sharing of journalistic expertise155

Over $250 million over 5 years

The Associated Press

OpenAI

Licensing deal for training AI models and developing technology for news gathering45

$1 million to $5 million per year

Axel Springer

OpenAI

Licensing deal for training AI models and developing technology for news gathering45

$1 million to $5 million per year

Prisa Media

OpenAI

Licensing deal for training AI models and developing technology for news gathering5

$1 million to $5 million per year

Le Monde

OpenAI

Licensing deal for training AI models and developing technology for news gathering5

$1 million to $5 million per year

Financial Times

OpenAI

Licensing deal for training AI models and developing technology for news gathering15

$1 million to $5 million per year

Hearst

OpenAI

Licensing deal for training AI models1

$1 million to $5 million per year

Time, Der Spiegel, Fortune, Entrepreneur, The Texas Tribune, Automattic (WordPress.com owner)

Perplexity

Revenue-sharing deal with access to analytics and technology to create custom answer engines. Revenue generated from sponsored related questions will be shared with publishers1

Revenue-sharing basis

Conde Nast, NBC News, IAC (People and Daily Beast owner)

Apple

Discussions for licensing content archives, but no public deals announced yet. Apple is offering more substantial remuneration for broader rights to use the content124

At least $50 million over a multiyear period (reported offer)

Reddit

Google

Licensing deal for user-generated content to train AI models3

Not specified

Mumsnet, The Center for Investigative Reporting

OpenAI

No deal yet. Instead, these entities have initiated legal complaints against OpenAI


The New York Times

OpenAI

No deal yet.  The New York Times is suing OpenAI and Microsoft for copyright infringement



It Will be Hard to Measure AI Impact on Knowledge Worker "Productivity"

There are over 100 million knowledge workers in the United States, and more than 1.25 billion knowledge workers globally, according to one A...