Monday, December 5, 2022

"Blaming the Victim" When Surveys Don't Work as Expected

Some might call the effort people put into their survey responses as “satisficing.” As applied to survey response data, the term means some people are not thinking too much about the actual responses they are giving to the poll questions. That might be akin to "blaming the victim" of a crime for crime's commission.


Some of us might argue the term "satisficing" is quite misapplied. To the extent "satisficing" can be said to apply, most of it already has been applied in the design of the polls or surveys.


To be sure, the definition os “satisfice” is to “pursue the minimum satisfactory condition or outcome.” 


As used to describe survey respondent behavior, it connotes “choosing an alternative which is not the optimal solution but is a solution which is good enough.” 


source: FourweekMBA 


But that is precisely what multiple choice survey instruments require. As often stated, respondents are told to “pick the answer that most represents your views.” As most of us can attest, oftentimes none of the available options actually represents our “true” opinions. No matter. 


Also, unlike Simon’s search for understanding of decision making, he challenged the notion that human thinking actually could encompass all possible solutions. The whole point is that humans cannot do so, so a “good enough” solution always is chosen. 


In a multiple-choice survey instrument, the designers already have eliminated all but a set of choices. Respondents do not have to choose the “best possible” response, only that response presented to them, which is a handful of choices. 


The “satisficing” already has occurred, but it does not represent  respondent behavior: it represents all the simplifying decisions made by the designers of the survey instrument. 


One must indicate which answer “best” fits one’s views. The term “satisficing” was created in 1947 by Professor Herbert Simon  n his 1947 book Administrative Behavior


His argument was that humans cannot be fully rational when making decisions. So-called  rational choice theory, which asserts that this is how decisions are made, is unrealistic, Simon argued. 


Instead, what humans actually use is a process he called bounded rationality. What humans actually do, since they have limited data, limited time and limited capabilities, is seek a workable solution to a problem, not technically the “best possible” solution. 


The concept is that humans do not have unlimited time, resources or capability to rationally consider all possible solutions to any problem, and then choose the optimal solution. Given all the constraints, they search for a limited number of solutions that will work, that are “good enough” and proceed. 


As applied to survey design or survey response, bounded rationality--known as “satisficing”--already has been employed. Survey designers already have chosen a very-finite set of “solutions” or “answers” to problems, issues, attitudes or choices that might possibly be made in real life. 


Perhaps the real answer--from any respondent--is that they would choose “none of the above,” all of the time, for reasons they have no way to communicate to the survey design team. 


Perhaps it is understandable that survey instrument designers fault their respondents for providing “bad data.” Some of us would submit that is not the problem. The problem is the faulty architecture of thinking about the issues for which answers are sought; the explicit choices offered to respondents; the forcing of responses into a predetermined framework; using language not nuanced enough to capture actual choices, beliefs, preferences or possible actions. 


If the data does not fit one’s assumptions or existing beliefs, whose fault is that?


No comments:

Will AI Actually Boost Productivity and Consumer Demand? Maybe Not

A recent report by PwC suggests artificial intelligence will generate $15.7 trillion in economic impact to 2030. Most of us, reading, seein...