Farmers’ scenario included simplifications, some of which are not too important for our discussion, and some of which do matter, thus, making our deep answers incomplete. Let’s list the cosmetic ones first.
1) Utility cannot be measured in potatoes. We could try to come up with better measures of happiness. I think the model holds for reasonable choices of human preferences. An example of preference that would break it is one that dictates that people care about happiness of others as much as about their own. I am afraid assuming this would be insufficiently cynical.
2) Humans are not completely selfish. But I think they are sufficiently selfish for the reasoning to be valid. It may feel monstrous to imagine killing Bob just to get his potatoes but scenarios of this kind have been played out too many times in human history. On that note, killing animals is pretty monstrous and most people are fine with this. (Though they may spend some energy to justify why it is okay to be monstrous towards animals.)
To sum up our model so far. We know that even completely selfish societies have reasons to establish moral codes and punish those who break the rules. The resulting morality is effective because following it is actually beneficial to the agents who negotiated the contract. The code and enforcement of it modify the pay-offs for different actions. Stealing potatoes is no longer a high-reward action, if you have to spend years in jail when caught.
Let’s move to simplifications in the scenario that make a big difference, that is, having them in our model leads to inaccurate predictions about morality. Each point is a correction of an assumption made previously.
1) Moral codes cannot be enforced perfectly.
Some attacks on other players are difficult to detect. As a result, it is challenging to generate incentives which discourage those attacks. Think about a skilled thief or problem of internet piracy. With the punishment removed, rational agents are less inclined to act morally.
This tells us that we probably will not be able to solve all of the Prisoner’s Dilemma-like coordination problems. If some crimes are difficult to detect, the law enforcer may not be able to generate sufficient disincentive. An agent may calculate that breaking the law has positive expected reward because likelihood of getting caught is low. Similarly, it may be that some criminals are really skillful at avoiding being detected. Thus, even though we have established rules that prevent most people from acting anti-socially, those particular players will keep breaking them because we cannot disincentivise them.
Should a rational agent adhere to such rules nonetheless? No. By definition, it should grab any extra utility whenever possible. Do humans adhere to such unenforcable rules? Some do and some don’t. Overall, I think people are much more lax about their moral codes when they think no one will catch them. Think about society’s leniency with respect to internet piracy when compared to physical theft.
We often like to view criminal behaviour as a form of disease. Something that people resort to only when they are damaged. However, many crimes, for example robberies, are often committed with a profit in mind and involve deliberation about risks. This points to rationality of perpetrators. Acting against moral codes may be the optimal action.
2) Real-life is an iterative game.
Game theoretic problems in real-life are rarely one-shot. Typically, the actions a player took affect his reputation and dynamics of future games. This slightly decreases the value that comes from enforcement of the rules by centralised institutions. If a merchant does not pay to his suppliers, it may be not necessary to threaten him with police force. If he knows that not paying can hurt his reputation to the point where nobody trades with him, he may feel compelled to be a fair player.
Reputation plays an important role in interactions between agents. People do not to trust those who are considered morally corrupt. If someone lied to one person today, he will likely lie to you tomorrow. With this in mind, we can expect that people will try to inflate their reputations. If an agent acted immorally, it will hide this information from others, even if they would not report him to the law enforcer. If someone is caught stealing at a supermarket, they will try to diminish gravity of the crime — “This is the first time!” or “The corporation earns so much that it wouldn’t even notice!” At other times people can dishonestly advertise willingness to help others when they expect no one will ask for it.
In a society where reputation is a resource that can be converted into allies and, thus, a greater political power, we should expect rational agents to try to convince others to credit them high reputation. However, it is in the interest of listeners not to get fooled. Usually to maintain a high reputation the agent has to convince a large number of agents freely exchanging information and judging it. In a situation when the listeners expect attempts of deception societies may discover signalling. We will return to this concept later, when considering evolutionary origins of morality.
We should expect that in societies where reputation is important, not all of the rules need to be explicitly stated in the code and enforced by a centralised institution. Potential damage to reputation may be a sufficient deterrent from immoral behaviour. For example, in US it is perfectly legal to object to and advocate against legalisation of gay marriages. However, reputation of someone who publicly takes such a political position may be attacked by those who consider such act a moral transgression.
3) The game is not symmetric.
In the real world, the competing players are not clones. Some are better at assassinations, others have many friends to call to arms, others still are not great at anything. Thus, the resulting game is asymmetrical and so we should not expect symmetrical moral contracts. Each of the farmers decided to fund the police force for selfish reasons. If there is a better utility maximising strategy the players will try to implement it. If the society is non-uniform, gangs may form to dominate the weaker farmers. More generally, the moral contracts established by a society reflect preferences of agents with more political power (those in possession of armies, resources and strong allies).
Citizens of an invaded country or slaves from far away colonies may cry “Unfair!” as their oppressors mistreat them. Without backing of political power their voices go unheard. Somewhat puzzling is the fact that those in power typically present a moral excuse for their acts. It need not be very believable. Perhaps the invaders are the true heirs of the lands? Perhaps the slaves are not truly human but from a different, lower, species? Why do the oppressors make an effort to find excuses? Are they trying to fool the other players that they are actually noble and deserve high reputation?
These acts sound truly atrocious and we tend to think that humanity is now far above them. That our societies would not abuse the weak only because they cannot fight back. Well, how do we treat animals? As equals? Clearly no. Still, justice is on our side. You see, animals are not as intelligent as we are and, thus, their pain does not matter. Or they are not conscious in the relevant way because ability to self-model is the only cosmically important kind of computation. I doubt we would be happy to grant validity to those excuses should an advanced alien civilisation show up on Earth tomorrow.
Again, why go into trouble of explaining ourselves? Do we do it to trick the vegetarian 3% of the population? Not sure if it is working. They usually seem convinced that it is them who holds the moral high ground.
Let’s restate correction applied to our model in this step. Moral contracts established by a society reflect preferences of the agents with political power rather than of entire society. If that is most of the story, then we expect that morality should change as the power balance shifts. Over the course of history we observe something more similar to a continual moral progress than an oscillation. One can argue that this is because the power spreads over a higher proportion of society. But the direction of causality is not so obvious.
Gay rights are on the rise recently. Does this imply that homosexuals have recently gained significant political power? Did they suddenly got richer or made lots of friends? Or rather, did many non-homosexuals started including gay rights in their preferences and truly fight for their rights? Pretending to care about others to boost one’s reputation may be a reasonable strategy for rational agents. Actually changing one’s preferences seems more far fetched.
Our current model does not explain this behaviour too well.
4) Morality enforcers are selfish agents as well.
Whoever enforces the law is not free from selfish desires. And so we see authoritarian regimes which bend the laws to jail political opponents. Or drunk-driving rich kids whose criminal problems can be straightened out by powerful parents bribing authorities. In these settings acting selfishly may be misaligned with acting morally. It seems that our civilisation keeps improving on this problem by increasing institutional transparency and by never putting all of the enforcement power in a limited number of hands.
Incorporating this complication means that enforcement institutions are made of agents. Thus, they are as selfish as any agent. Other agents may take advantage of their egoism and, for example, try to bribe them.
In dictatorships all of the political power is focused in the hands of a few agents who can easily coordinate to dominate the society. Modern democracies, on the other hand, are run by thousands of politicians with conflicting goals. Additionally, since governors are selected by the population, democracies have high transparency because acting transparently can serve as a good signal of trustworthiness.
5) Humans are not perfectly rational.
While rationality dictates a specific (not necessarily easy to predict) behaviour, there are many ways to be irrational. It is informative to consider our evolutionary history to understand what kind of decision-making algorithms are run by our brains. This is a bigger issue, so I’ll explain it in a separate post. For now let’s assume humans are mostly rational, i.e. they understand consequences of actions up to some complexity and use this knowledge to select actions which guide them towards their goals.
We still have a couple of missing parts in our model. Nonetheless, let’s revisit the question the core question: “Why should a rational agent, that only cares about maximising his own utility, be moral?”
“In large part, he should behave morally because a society of rational agents sets up the world in such a way that acting immorally is punished. Thus it is in agent’s self interest to act morally. An agent which earns a high reputation can reap long term rewards. However, a good reputation can be gained by both truly being charitable and by looking charitable. The second option is usually cheaper. Thus, the agent may loudly promote moral rules to which he is not himself committed.”
“Moreover, sometimes the society may not have enough power to align moral and selfish behaviours of the agent. Maybe the agent can conceal his suffering generating actions (e.g. a skilled thief), or he has an army which renders him untouchable (e.g. a dictator), or maybe the society does not care about those who suffer enough to confront the agent (e.g. PETRL). In such cases the rational agent should not act morally because it is against his self-interest.”
Such an agent appears to be a perfect Machiavellian. He can be nice to others but all of this is just a mask. The moment you take eyes of him he lies, manipulates, backstabs and does whatever can bring him closer to the goal. All with zero honest remorse. While this is similar to some fictional characters and brings to mind dark triad traits, it doesn’t feel very human. People usually do not engage in such completely selfish calculations. They seem to genuinely care about others and not just pretend. On the other hand who they care about can be extremely selective and self-serving.
However, do not let this overshadow how well this model explains contracts between different groups of people now and in the past. I realise that, for example, utilitarianism was not devised to predict human behaviour. But you can question how well it approximates our moral intuitions, if for years we had no regrets about enslaving, torturing and killing others. And wherever dictators rise to power they remind us that mass cruelty is not as inhuman as one would like to think.