How Much of What We See in Remote Work is "Hawthorne Effect?"

 We still actually know very little for certain about how productivity changed because of enforced remote work for knowledge and office workers. The problem is that we cannot measure the productivity of such workers easily, if at all.

Also, there are measurement effects, to the extent that enforced remote work is a bit of an experiment. The Hawthorne Effect is that subjects in an experiment tend to perform better. 

There also are demand characteristics. In experiments, researchers sometimes display subtle clues that let participants know what they are hoping to find. As a result, subjects will alter their behavior to help confirm the experimenter’s hypothesis.

Then there are novelty effects: The novelty can lead to an initial increase in performance and productivity that may eventually level off as an experiment continues.

Performance feedback is similar to the Hawthorne Effect. Increased attention from experimenters tends to boost performance. In the short term, that could lead to an improvement in productivity.

That assumes we can measure knowledge worker or office worker productivity, however. 

The problem with all studies of officer worker or knowledge worker productivity is measurement. What can be counted so we know whether inputs have changed. And how do we measure the output of knowledge work? 

Presumably a call center operation has quantifiable metrics, but most office or knowledge work does not have any obvious and convenient measurement criteria. We commonly measure “time working” with the assumption that additional time worked is better. Maybe. But hours worked is an input metric, not an output metric. It is the denominator, not the numerator. 

Logically, increasing input (denominator) can work to reduce productivity (output) unless output measures also increase faster than inputs increase. 

The other common issue is that we equate worker attitudes with outcomes. Happier workers might, or might not, be more productive. All we can measure is a subjective attitude. More happy or less happy does not necessarily correlate with outcomes. 

In principle, one could have happier but less productive workers; less happy but more productive workers. One would need a way to correlate output and outcomes with feelings in ways that outlive simple Hawthorne effects (people work better when they know they are part of an experiment). 

Work team collaboration might have fared better under full remote work conditions, but there is some evidence that firm-wide collaboration has decreased, though the amount of time spent collaborating (meetings, emails, messaging) has grown.  

Actual output is different from input or collaboration time and effort. It might be difficult to measure “creativity,” but there is some belief that has not done better under conditions of remote work.  

Meetings are inputs, not outputs. Having more meetings, or spending more time in meetings, does not make firms or organizations more productive. A Microsoft survey of 182 senior managers in a range of industries found support for that thesis. 

We might say the same for collaboration during the enforced remote work period. It is common to hear technology business or policy leaders argue that remote work has not harmed productivity. 

Leaving aside the issue of whether remote work productivity changes can be measured, collaboration--deemed by most to be vital for knowledge workers--might have gotten far worse because of Covid. 

People like the freedom to work from home, no question.  

That might have happened despite reports that suggest information, knowledge and office workers now are spending more time with electronic forms of communication. But “communication” is not necessarily “collaboration.”

If collaboration is defined as “people working in teams or with others,” then collaboration seemingly has suffered. 

According to Gensler, “high-performing people at top companies tend to do individual work and collaborative work in equal measures—45 percent each, according to our research--with the remaining 10 percent made up of learning and social time.” 

For better or worse, those balances were changed during the period of enforced work from home policies. “While at home during the pandemic, people reported working in individual focus mode 62 percent of the time and 27 percent in collaboration, a disparity that negatively impacts company creativity and productivity,” Gensler argues. 

Before the pandemic, U.S. workers spent an average of 43 percent of their work weeks collaborating either virtually or in person. That number fell to 27 percent for workers who worked from home in 2020, for example. 

“At the onset of the pandemic, our analysis shows that interactions with our close networks at work increased, while interactions with our distant networks diminished,” say Microsoft research. “This suggests that, as we shifted into lockdowns, we clung to our immediate teams for support and let our broader network fall to the wayside.”

There is a downside: similar companies almost certainly became more siloed than they were before the pandemic. 

“And while interactions with our close networks are still more frequent than they were before the pandemic, the trend shows even these close team interactions have started to diminish over time,” Microsoft researchers say. 

Younger workers (25 or younger) also reported more difficulty feeling engaged or excited about work, getting a word in during meetings, and bringing new ideas to the table when compared to other generations.

“Bumping into people in the office and grabbing lunch together may seem unrelated to the success of the organization, but they’re actually important moments where people get to know one another and build social capital,” says Dr. Nancy Baym, Microsoft senior principal researcher “They build trust, they discover common interests they didn’t know they had, and they spark ideas and conversations.”

Microsoft researchers noticed that “at the office: worker instant messages slowed  25 percent during lunchtime, but remote workers at home reduced IMs by 10 percent. Also, IMs grew by 52 percent between 6 p.m. and midnight, suggesting that at-home remote workers might have been working more total hours than employees in the office.

At-home workers also spent about 10 percent more time in meetings. Those results might be interpreted as either good or bad effects of collaboration. 

Microsoft research also suggests that while collaboration within work teams increased, collaboration outside of the teams, with the rest of Microsoft personnel, decreased.  

The point is that we actually know quite little about potential changes in productivity, especially longer-term impact. In the short term, there is a Hawthorne Effect at work, which would “boost productivity” in the short term. 

Are Meetings and Messages Really "Work" or "Outcomes?"

There is some evidence that workers and their managers do not agree on the impact of remote work on productivity. A Microsoft survey of 20,000 people in 11 countries found 87 percent of employees reporting they are more productive remotely or with a mix of in-office and remote work. 

But 85 percent of leaders say hybrid work makes it difficult to determine if their workers are being productive.

We should be clear that nobody has yet developed an accepted and trustworthy way of measuring knowledge worker or office worker productivity. 

So the debate about the productivity of remote work will likely never be fully settled, in large part because it is so difficult, perhaps impossible, to measure knowledge worker or office worker productivity. 

Whether knowledge worker productivity is up, down or flat is almost impossible to say, despite claims one way or the other. Much of the debate rests on subjective reports by remote workers themselves, not “more-objective” measures, assuming one could devise such measures. 

Pearson's Law and the Hawthorne Effect  illustrate the concept that people who know they are being measured will perform better than they would if they are not being measured. 

Pearson's Law states that “when performance is measured, performance improves. When performance is measured and reported back, the rate of improvement accelerates.” In other words, productivity metrics improve when people know they are being measured, and even more when people know the results are reported to managers. 

Performance feedback is similar to the Hawthorne Effect. Increased attention from experimenters tends to boost performance. In the short term, that could lead to an improvement in productivity.

In other words, “what you measure will improve,” at least in the short term. It is impossible to know whether productivity--assuming you can measure it--actually will remain better over time, once the near term tests subside. 

So we are only dealing with opinions: whether those of workers or their managers. People might think they are productive, when they are not. Managers might believe some set of tools allows them to measure such productivity, when they actually cannot. 

Let us be truthful: people like working remotely for all sorts of reasons that have nothing to do with “productivity” in a direct sense. Managers distrust such work modes in part because they lose what they believe to be impressionistic measures of input. 

But input does not matter. Output matters. And “output” is difficult to measure in any knowledge or office work, in ways that correlate well with organizational outputs. 

And by some quantitative measures, remote work might be reducing productivity, using either time spent on communication--whether by messaging or in meetings. Some people act as though sending messages constitutes work, when it might be, at best, a way to coordinate work. 

Some seem to believe that meetings are work, when most meetings are efforts to coordinate work. “Work” happens outside chats, messages or meetings. So if time in meetings or spent communicating increases, that increases the “time spent” denominator for purposes of determining outcomes numerators. 

In other words, if remote work creates a need for vastly-more communication about work, then the effort to produce outcomes from work almost certainly increases, compared to output. Hence, lower productivity.

“Since February 2020, the average Teams user saw a 252 percent increase in their weekly meeting time and the number of weekly meetings has increased 153 percent,” says Microsoft.

The average Teams user sent 32 percent more chats each week in February 2022 compared to March 2020 and that figure continues to climb. Workday span for the average Teams user has increased more than 13 percent (46 minutes) since March 2020, and after-hours and weekend work has grown even more quickly, at 28 percent and 14 percent, respectively.

source: Microsoft 

It might only be one indicator, but vastly-increased time spent on coordination might be a sign that remote work is not as “productive” as remote workers believe, unless the amount of work time increases to compensate. And even there, increasing the denominator, compared to the numerator, always leads to a smaller result: arguably lower productivity.

What Changes for Enterprise Spending After Covid?

As someone who believed Covid would end sooner than it has, and would not have as much long-lasting impact on enterprise work practices, there is some new evidence suggestive of long-term changes. The issue is that the changes pull in different directions. 

To be sure, as the pandemic is not yet “over,” it might not be possible to assess the magnitude of potential shifts for some time. Still, there is some evidence that external collaboration impact might be the opposite of internal collaboration impact. 

Also, face-to-face meetings might be more important for some roles--such as sales--than for customer support, training and marketing. Meeting clients is seen as the top reason to resume travel, according to a Deloitte survey, while internal meetings and training are more likely to stay online.

U.S. spending on business travel is expected to only reach 25 percent to 35 percent of 2019 levels by the fourth quarter of 2021, and 65 percent to 80 percent a year later, according to a Deloitte survey of 150 travel managers.

To the extent that more remote work is going to be a permanent change, it is worth noting that collaboration and productivity trends might not be as rosy as many self-reports suggest. 

A study by researchers of Microsoft employees shows firm-wide remote work caused the collaboration network of workers to become more static and siloed, with fewer bridges between disparate parts. Furthermore, there was a decrease in synchronous communication and an increase in asynchronous communication.

Company-wide remote work caused business groups within Microsoft to become less interconnected, the researchers say. 

Remote work also reduced informal collaboration. Furthermore, the shift to firm-wide remote work caused employees to spend a greater share of their collaboration time with their stronger ties, which are better suited to information transfer, and a smaller share of their time with weak ties, which are more likely to provide access to new information, the study suggests. 

The researchers  suggest that hybrid and mixed-mode work arrangements may not work as firms expect. 

To be sure, long-term effects might not be the same as the short-term effects. the period of time over which we measured the causal effects of remote work are quite short (three months), and it is possible that the long-term effects of firm-wide remote work are different. 

For example, at the beginning of the pandemic, workers were able to leverage existing network connections, many of which were built in person. This may not be possible if firm-wide remote work were implemented long-term.

In other words, social capital decays over time. 

The point is that we might see several trends that run counter to each other, with new “distributed and remote” work modes producing less face-to-face interaction, while sales activities might need to be reinstated for sales operations. 

On the other hand, many enterprises also will find less need for face-to-face support for some internal operations and some customer-facing tasks. 

On a broader level, we might look at an analogy to changes wrought by widespread use of the internet and mobility.  Every content business you can think of was changed, first by the internet and then by mobility. 

To the extent that the internet reduced information gaps or friction, demand for trade shows, magazines and newspapers actually fell. Much the same has happened with mobile-based communication and media. Newspaper revenues fell while online revenue grew. 

Home video shifted to online. Linear subscriptions shifted to internet-based subscriptions. Music likewise shifted from physical media to internet delivery.

In the business world, software distribution also shifted from physical media to online fulfillment; products to services. With quality broadband nearly ubiquitous, computing shifted from local to remote, as well.   

To the extent that trade shows, trade journals and specialized business publications were needed to reduce information friction, there simply was less need once the internet made information more transparent and easy to get. 

There may be parallels with the ways business information and commerce changes after Covid. Demand for some activities will decline while others replace them. Business travel, trade shows and collaboration itself might be altered on a permanent basis.

Nobody Can Say For Sure Whether Productivity is Up, Down or Unchanged

Some worry that productivity has fallen recently, since the fourth quarter of 2021, according to U.S. Federal Reserve data. But studies have produced contradictory findings, showing

 higher, lower or unchanged worker productivity since about 2020. 

Arguably much of the data most susceptible to change were generated by knowledge or office workers forced to work from home because of the Covid pandemic. 

With the caveat that some of us are skeptical about the ability to measure knowledge worker or office worker productivity, studies can be cited showing higher; lower and “no change” in productivity because of remote work, for example. 

And intangible products accounted for as much as 80 percent of total U.S. output in 2022. The goods-producing sector includes agriculture, forestry, fishing, and hunting; mining; manufacturing; and utilities. 

The service-providing sector includes trade, transportation, and utilities; professional and business services; education and health services; leisure and hospitality; and other services.

Still, productivity changes in both goods and services has been in a 1.5-percent to two-percent annual rate of change since shout 1980, according to the Bureau of Labor Statistics. That estimate is based on hours worked, compensation levels and therefore unit costs. 

Just anecdotally, one might conclude that knowledge worker productivity has neither increased or decreased since the enforced “work from home” policies caused by the Covid pandemic. The line of reasoning is that so many people report being able to get their work done in less time while working at home. 

But studies exist that argue productivity is higher, lower or unchanged because of remote work. 

Study	Date	Publisher	Findings
The Impact of Remote Work on Productivity	2020	Stanford University	Increased productivity by 13%
Remote Work: A Review of the Evidence	2021	McKinsey & Company	No significant change in productivity
The Future of Work: How Remote Work is Changing the Way We Work	2022	Gallup	Productivity increased by 5% for remote workers
Remote Work and Productivity: A Meta-Analysis	2023	Harvard Business Review	Productivity increased by 10% for remote workers
Knowledge Worker Productivity in the Age of COVID-19	2022	University of Oxford	Productivity decreased by 2% for knowledge workers
The Impact of Zoom Fatigue on Knowledge Worker Productivity	2022	University of California, San Francisco	Productivity decreased by 10% for knowledge workers who use Zoom for more than 2 hours per day

So people use the time to do other things. That might well be interpreted as higher productivity by some; lower productivity by others.

But, strictly speaking, the ability to get one’s expected workload completed in less time should represent higher productivity, assuming one believes the output is measurable. The issue, perhaps, is that employees do not always use the time saved to do additional work. 

So firm productivity is likely unchanged, but personal productivity arguably could be higher. Firm unit costs might not change if the same output happens in a remote rather than workplace setting. 

But remote workers might create a time surplus that is not used to boost output further. 

That would essentially mean a  hypothesized “loss” of firm productivity, even if individuals were “more productive,” in terms of getting their accustomed workloads finished faster. There is extra time, but that  extra time is not used in the form of “productive” effort for their employers. 

Here we must also acknowledge that “extra time” exists even when workers are face to face at work, and such time is not mobilized “productively” in that context, either. 

There are all sorts of other possible productivity-related variables. To the extent some firms have been “hoarding” workers “in case” they needed them would be expected to result in lower “output” from those workers. They actually were not hired for “real” unmet needs, but as a hedge against needing their work later. 

That might be considered unproductive using any measurement method. A recent study by Microsoft of the amount of time people are using tools such as Microsoft 365 might, or might not, shed much light on “productivity.”

The study looked at the amount of time users spent with various Microsoft 365 apps. Some might say this is a measure of input, out output, but presumably has some indirect effect on “output.” 

The study shows people spend 57 percent of their time communicating in meetings, email, and chat, with 43 percent in activities that Microsoft calls “creation” and include use of spreadsheets, word processing, presentation software or notes. 

The 2023 Work Trend Index, which surveyed 31,000 people in 31 countries, is not a direct productivity measurement, but tracks time spent in applications, all of which are assumed to have some enabling role in producing actual knowledge worker output. 

source: Microsoft 2023 Work Trend Index

The key issue for some of us is that we do not actually believe knowledge worker productivity can be measured accurately. And most of the proposed measures--high level or detailed--seem to confirm the coarseness of metrics. 

At a high level, some might argue that knowledge worker or office worker productivity, at a high level, could possibly be measured using metrics such as the “number of units” produced. 

That might be lines of code, calls answered, courses taught or stories written. None of that can account for the quality of output, however. 

But much knowledge work is intangible, and the intangible is, by definition, harder to measure than the tangible. How do we measure the productivity of those who produce “experiences” for users or customers?

Also, much knowledge work is collaborative. By design, that makes individual contributions hard to measure. And much knowledge work is creative, such as problem solving or innovation. How do we measure, in quantitative terms, qualitative outcomes?

Some might argue that “revenue generated” is a productivity metric, but is nearly impossible to measure at an individual level, since sales are a product of the work of so many different teams. 

Yet others might prefer measures of “customer satisfaction.” Again, this can be hard to measure on an individual basis and might be nearly impossible at any level much lower than the single product or company reputation level. 

In some cases, a high level “time to market” metric might be considered, but again that is more a work group metric than a single individual measurement. 

Cost savings are almost never an actual measurement of productivity, though in principle it might be part of an evaluation in the short term. 

The bottom line is that we still have no idea whether knowledge worker productivity is rising, falling or remains unchanged.