Evaluating Education Systems Through Inequalities Between Families

Is the French education system a rather average performer? According to the PISA ranking, this seems to be the case. But this ranking, which ranks education systems based on the average scores of their 15-year-old pupils, has its limitations. The economists Nicolas Gravel, Edward Levavasseur, and Patrick Moyes have therefore taken a new look at the PISA ranking by accounting for inequalities in family background.

What returns every three years, becomes a media sensation, and creates an uproar among education specialists, journalists, and politicians alike? PISA, or the Program for International School Assessment. The aim of this survey carried out by the Organisation for Economic Co-operation and Development (OECD) is to identify “what 15-year-old students know, and what they can do with what they know.” So, every three years, it evaluates 600,000 students from 75 countries in three different subjects: reading, math, and science.

In every report, one can find the results of these evaluations with grades, information about the students, averages, and, most notably, a ranking of the participating countries based on the average scores of their students. China, Singapore, Estonia, and Canada are often in the lead. France occupies a rather modest – but not abysmal - 20th position in the ranking.

A (Socially) Incomplete Ranking

France’s performance, which is judged mediocre by some, is occasionally interpreted as revealing the inefficiency of the country’s education system. But using the PISA ranking as the only criterion for evaluating an education system is shortsighted and perhaps even oversimplified.

This ranking only evaluates the average performance of pupils on a “core” group of subjects. Non-cognitive skills—such as the ability to interact with others, control one’s emotions, participate in society, and acquire manual and athletic skills—are not assessed. Furthermore, the cultures and education systems of the countries tested are widely different, making it difficult to truly compare the performance of each.

This ranking also ignores another important feature of any good education system: its ability to mitigate the consequences of economic and social inequality. Children from highly educated and well-off families have an advantage over those coming from less privileged families. For this reason, the economists N. Gravel, Edward Levavasseur, and Patrick Moyes take another look at the PISA data by integrating the extent to which education systems are able to reduce the effects of such inequalities. They focus their attention on two variables recorded in the PISA survey: the results in math for every student and the distribution of HISEI (Highest parents’ International Socio-Economic Index) data. Why? Math is a standardized subject that promotes the understanding of quantitative reasoning, induction, the development of critical thinking, and, of course, logic. Because this discipline is technical and standardized in nature, it is less likely to be influenced by cultural and societal differences. The HISEI index provides information on the highest of the two parents’ socio-economic status. It provides therefore a synthetic summary of the child’s family background.

Photography by Nikhita S on Unsplash

The Vietnamese Surprise

Some countries, like Singapore, the Netherlands, and Denmark, are ranked high for results in both math and family background average. On the other side of the spectrum (with low math scores and low family backgrounds), one will find Morocco, Algeria, Thailand, and Mexico. France is situated somewhere in the middle, between Austria and Portugal.

However, some countries show quite a surprising gap between their score in math and their HISEI index. The most astonishing case comes from the United Arab Emirates. Though it has the highest average of parent socio-economic status among all the countries studied, it nevertheless holds one of the lowest scores in math. This demonstrates that the country’s education system does not seem to allow children from generally well-off families to develop high math skills. Lebanon, Jordan, and Israel find themselves in a similar situation.

Photography by Tinnakorn on Adobe Stock

A surprising case for the other side of the coin is Vietnam. It has the lowest national HISEI index average, yet it retains a higher math score average than France as well as several other countries. The Vietnamese education system is therefore successful in fostering highly developed math skills in children coming largely from modest family backgrounds.

An Innovative Methodology

The PISA ranking is based on the average score of each of the evaluated skill. While direct and easy to understand, an average does not provide any information about the distribution of the variables in the population. Hence, a country in which the scores are all nearly similar to the mean can be considered equivalent to another system that exhibits the same mean but in which the scores are much more spread out. Another shortcoming of averaging students’ scores is that it does not provide any information on the correlation between the student’s grade and the student’s family background. However, this information is crucial for evaluating the ability of the school system to reduce inequalities of opportunities between children from different family backgrounds.

To go beyond these limitations, the researchers used an innovative approach and relied on three different principles to analyze the distribution of the math scores and HISEI index for every country. The first principle requires the favorable recording of any improvement in the cognitive skill of a child with a given family background when this improvement does not affect the cognitive skill of other children. The second principle demands that any child’s cognitive skill be all the more favourably appraised as the child is coming from an unfavourable background. The third principle states that when two different skills and family backgrounds are allocated between two children, it is preferable that the high skill be given to the low background child than the other way around.

These three principles alone do not allow all countries to be ranked. Instead, the authors show how the countries can be compared by “dominance tree” showing which countries perform indisputably better than others.

France: Elitist & Low-Performing

Who are therefore the countries who dominate the others according to these three principles? Who are the countries who are the best at improving grades of students from the most disadvantaged backgrounds? It turns out that Vietnam performs very well, since it dominates 18 countries without being dominated by any. Among developed countries, Japan and Poland are at the top. Neither are dominated by another country, and they dominate nine and twelve countries respectively (with France being one of them).

One surprise comes from Finland, which is often described as the global leader in education: it dominates only five countries and is dominated by none. This (relative) counter-performance may be explained by the somewhat strong correlation between the parents’ socio-economic status and the student’s academic results observed in this country. Among the countries with the lowest performance, one will find Jordan, Lebanon, and the United Arab Emirates, respectively dominated by 18, 25, and 29 countries.

How does France do? Though Poland is the only country that dominates it, France dominates only five countries (including the United States). This rather modest performance can be partially explained by the fact that, just like Finland, France exhibits an important correlation between the student’s parents’ HISEI index and their math results.

In fact, France has one of the highest correlation coefficients between math results and background—just behind Hungary, Peru, and Belgium. What does this study imply for the French education system? It demonstrates that it is very elitist, with many students either well above the average, or well below the average! But, more importantly, it shows that its education system fails at correcting the inequality of opportunities among children from different family backgrounds.