How can we assess the impact of deliberation on people's votes?

Consensus conferences, citizens' assemblies, neighbourhood councils, etc. Since the 1980s, deliberation has emerged as a response to the crisis of democracy. But how do you measure its value and impact? For the first time, a team of researchers has studied the role of the tools used to synthesize participants' opinions.

Guilty or innocent? In a famous American film, 12 jurors deliberate the fate of a man accused of murder. If they find him guilty he’ll end up in the electric chair. If they have valid doubts, the man will be acquitted1. The first vote was by a show of hands: 11 jurors found him guilty. But Juror 8 has his doubts. In the stifling heat of the jury room, a long and difficult debate begins. Throughout the film, arguments and votes follow one another, with voting procedures varying. At the end of the process, the jurors reach a verdict of not guilty by reasonable doubt. So what happened? Is the difference in verdicts the consequence of discussion or is it due to the change in voting methods? Beyond the cinema screen, does deliberation offer a solution for creating a consensus based on the elimination of poor arguments? This is precisely the type of question that the team of researchers Mariam Maki Sy, Charles Figuières, Helene Rey-Valette, Richard Howarth and Rutger de Wit have been looking into. The researchers wondered whether the choice of tools used to synthesize participants' opinions during a collective deliberation had an impact on the outcome of that deliberation.

• 1" 12 angry men", Sidney Lumet, 1957.

The black box of deliberative democracy

The stakes are high. Since the 1980s, the growing disaffection of citizens with representative democracy has led to the encouragement of new forms of participation. The use of decision-making processes based on exchange and discussion is increasingly popular. This deliberative democracy seems more likely to produce solutions that serve the common good and are supported by the population2. In France, a recent example is the Citizens' Climate Convention, made up of citizens drawn by lot. At the outset, these citizens had a wide range of ideas about the climate policies they wanted to see implemented. However, after debate, the 149 proposals drawn up by the various "commissions" were adopted by all the members of the Convention by near consensus.

Now that collective deliberation is being encouraged, it is important to understand how it works and to avoid possible abuses or perverse effects. Researchers in political science, for example, have studied the conditions for inclusive and fair deliberation in societies marked by inequalities and relationships of domination. This work focused on the modalities of the debates themselves. For example, how can we ensure that no one takes over the floor during discussions? But until now, no one has looked at the way in which participants' opinions are aggregated. However, every deliberation explicitly or implicitly uses a method to summarise the opinions of the participants. This is what social choice theory, which studies the processes involved in arriving at a coherent collective decision, calls a "social preference".

• 2See for example Habermas Jürgen, « Discourse ethics: notes on a program of philosophical justification ». Polity Press Cambridge, 1990.

Picture by Manny Becerra on Unsplash

One vote does not equal one vote

For a long time now, this theory of social choice has shown that voting methods are not equivalent to one another. For example, for a presidential election, organising a two-round majority vote or asking voters to rate all the candidates and then aggregating the ratings can lead to divergent results. And at the risk of killing any suspense, you should know that there is no ideal method, but rather a multiplicity of methods, each with its own qualities and faults. Studies of collective deliberation have been surprisingly silent on this dimension: the aggregation methods used to measure the impact of deliberation are rarely explained. However, we can assume that the choice of method influences the result obtained. For example, we know that deliberations can lead to the amplification of a previously dominant opinion - this is known as the polarisation effect. This tempers the optimistic message of the film "12 Angry Men", in which deliberation ultimately leads to the evolution of the jurors' decision. 

The research team decided to open this black box of aggregation experimentally. There's no murder and no electric chair here, just a reflection on the Palavas-les-Flots lagoon complex. These lagoons which stretch for around twenty kilometres between Montpellier and Sète, are fragile areas that are the subject of protection policies. To ensure that these measures are as well adapted as possible and enjoy the support of the population, consultation processes with the people involved are encouraged. In the experiment, 42 people were invited to collectively determine the priorities for protecting the lagoons. For example, should priority be given to water quality? Maintaining biodiversity? Fish stocks? Recreational use? Visitors' sense of relaxation? Twenty or so ecosystem services were suggested to the participants for prioritisation3. Two different methods of aggregating these individual priorities were used and then compared.

• 3Philippe Méral, Denis Pesche, Les services écosystémiques - Repenser les relations nature et société. Librairie Quae, 2016

Picture by Philippe Graille on Adobe Stock

Deliberate, but to what end?

Firstly, the preferences of the 42 people taking part in the experiment were aggregated using two methods. The first, known as RESPA, involves initially defining a subset of services deemed to have priority, and then classifying these services within this subset. Imagine 6 possible choices: if your priority is to maintain "leisure activities", you would allocate 6 points to it. The "regulate the microclimate" service, which doesn't interest you at all, will get 1 point. The sum of the points awarded to each service by all the participants is then calculated. The service at the top of the social ranking is the one with the highest sum, the second service the one with the second highest sum and so on. This type of aggregation, formally studied and defended by the Chevalier de Borda in the 18th century, was already used by Roman senators. It is now used in Europe... in the Eurovision Song Contest.

©CC / Representation of a Senate session « Cicero denounces Catiline », Cesare Maccari, 1889.

The other method used in the experiment is the "majority judgement", a system put forward in 2010 by two CNRS researchers4. It consists of each participant assigning, to each service, a qualitative assessment chosen from a predefined and ordered set of assessments, independently of his or her judgement of the other options. For example, a participant may feel that protecting biodiversity is a "high priority" whereas maintaining the beauty of the place is "not a priority". We then determine the median rating for each service, which we call its majority rating. For example, if the majority mention of the regulation service is "low priority", this means that half of the participants think that this service deserves greater importance than "low priority" and the other half think it deserves less importance. Finally, the social preference follows the order indicated by the majority ratings: the service ranked first is the one with the highest median rating, and so on. This method tends to prioritise the most consensual options. It also has the advantage of being relatively robust to strategic manipulations5.

Divided into 8 groups, the participants then had time for discussion and debate, accompanied by experts. At the end of the deliberation process, they repeated their rankings using the same two methods.

A comparison of the results speaks for itself. Prior to deliberation, the collective preferences of the participants in the experiment were broadly the same whether one or other of the aggregation methods was used. However, after collective deliberation, the priorities for policies to protect the Palavas lagoon areas were not longer ranked in the same way, by the two aggregation methods depending on whether one method or the other was used. The method least impacted was the majority decision method, perhaps because it was designed from the outset to bring out the most consensual options. On the other hand, RESPA-type aggregation led to strong divergences before and after deliberation. The impact of a deliberation therefore depends on the voting method used. This result calls for a more detailed study of the methods of aggregating individual preferences used in deliberative democracy. And perhaps lead to fairer collective decisions, like in the movies...

• 4Balinski M, Laraki R (2007) A theory of measuring, electing, and ranking. Proc Natl Acad Sci 104:8720–8725. See also Edelman, Paul. (2011). Michel Balinski and Rida Laraki “Majority judgment: measuring, ranking, and electing”, Public Choice. 151. 807-810.
• 5The French presidential election, in which the first round is the subject of complex strategies of anticipation of the second round, is a good example of tactical voting, often referred to as the "useful vote".