Arrow’s impossibility theorem suggests the impossibility of aligning human values with any single artificial intelligence model, for the simple reason that choices must be made about such values.
In other words, whose values should AI agents be aligned with? Take your pick: females or males; urban or rural; old or young; liberal or conservative; religious or atheist; rich or poor; European, African, Asian, Latin American or Native American?
Any model developer has to develop weights that make choices between such assumptions. The Arrow Impossibility Theorem suggests that universally aligning a single AI model to diverse human values is mathematically impossible, forcing AI developers to make structural trade-offs.
One can see this in the “paradox of voting,” dealing with ranked choices, discovered by the Marquis de Condorcet in 1785, for example.
That principle shows that possibilities for choosing rationally can be lost when individual preferences are aggregated into social preferences.
Likewise, Arrow’s theorem suggests that collective decisions cannot be made on the basis of a common preference that assimilates the tastes and values of all the individual men and women who make up a society. Not hard, “impossible.”
The question that Arrow’s theorem answers is: which procedures are there for arriving at a social ordering of some given alternatives, on the basis of some given people’s preferences among them, no matter what these preferences turn out to be?
And the answer turns out to be that there are no reliable procedures. Systems of ranked choice voting can produce outcomes that reflect “nobody’s” actual personal preferences, for example. Given three choices, A, B and C, it is possible that the ranked choice system produces an outcome nobody actually desired.
Say there are three alternatives: A, B and C, and three choosers. Each ranks their own preferences::
ABC
BCA
CAB.
In this scenario, there is no way to reconcile the differing “highest” value choices. One examines the second and third choices, looking for preference there. It cannot be found. So then one tries to assess the rankings any other way that attempts to assess which alternative “best” reflects a blended choice.
Looking at the sum total of first, second and third choices, no “clear” pattern emerges.
Suppose, taking the alternatives pair by pair, we put the matter to a vote: we count one alternative as socially preferred to another if most voters prefer it. We determine in this way that A is socially preferred to
B, since two voters (1 and 3) prefer A to B, but only one (voter 2) prefers B to A.
Similarly, B is socially preferred to C. We might then expect that A is socially preferred to C.
But there also are two voters who prefer C to A. So one winds up with a choice nobody actually supports.
In the context of model training, similar choices have to be made, when there are competing and conflicting views.
Since people, in addition to having different values and opinions, also have different intensities of feeling about their many values, there is no way of accounting for how much more strongly one must prefer one thing to another in order to make up for the fact that someone else’s preference is just the other way around.
So the problem is that there is no way to satisfy all of four conditions:
Universal domain: the aggregation rule should work for every possible profile of individual preferences. We cannot exclude unusual preferences in advance.
Pareto efficiency: if every individual strictly prefers a to b, the social ordering should too. This is the bare minimum of respecting individual preferences.
Independence of irrelevant alternatives: the social ranking of a versus b should depend only on individual preferences over a versus b, not on preferences over some third alternative c.
Non-dictatorship: there should be no single individual whose preferences always determine the social ordering.
The upshot is that models cannot be trained in a way that represents the full and whole group. So model developers use workarounds that are compromises.
Some of us might argue that only Grok is explicit about its own choices. The others might claim they are neutral or objective, when in fact they must make some value choices when training their models, just as
Grok does.



