Who makes it into PISA?

by Nicholas Spaull
Former Thomas J. Alexander Fellow, OECD

Unlike earlier PISA reports, the 2015 PISA report (Volume I  and Volume II) highlights differences in sample coverage – how many students were eligible to participate in PISA – between countries. Why is this important? Because you can’t really appreciate the magnitude of improvement in a country’s performance in PISA unless you also understand how access to education has expanded over time, too.

Take the case of Turkey. Of the OECD countries that participate in PISA, Turkey has one of the lowest levels of performance and the highest rates of improvement in PISA scores. Between 2003 and 2012, Turkey managed to improve its mathematics score by 25 points while also narrowing the achievement gap between rich and poor students (in other words, improving equity in education). Over the same period, Turkey also managed to keep students in education longer and see them progress more steadily through grades – achievements that, until now, have gone largely unrecognised.

In order to be eligible to sit the PISA assessment, a student must be between 15 years, 3 months and 16 years, 2 months old, still enrolled in school, and in grade 7 or higher. These details may sound like trivial technicalities, and in most OECD countries they are. But in Turkey and Mexico, and partner countries like Viet Nam and Indonesia, these details can make a big difference. This is because early school-leaving, dropout and slow progress through grades are widespread and substantially affect PISA sample coverage.

A recently-released paper shows that in 2003, fewer than one in two Turkish 15-year-olds was eligible to sit the PISA test. Household survey data from the 2003 Turkish Demographic and Health Survey show that 33% of 15-16 year-olds had already left school, and another 18% were still enrolled in grades 1 through 6. Thus the 2003 PISA results were only truly representative of less than half (45%) of the population of Turkish 15-16 year-olds.

But between 2003 and 2012, there was a significant increase in students remaining in school and a considerable decline in excessively delayed grade progression. The figure immediately below, based on data from the Turkish Demographic and Health Surveys of 2003, 2008 and 2013, shows that the percentage of students who were eligible to sit the PISA test increased from 45% in 2003 to 80% in 2012.

The paper explains that if one takes into account these large expansions in access and attainment, the improvement in Turkey is much greater than is usually thought. For example, taking into account both improvement in test scores and the expansion of the population of 15-year-olds who are eligible to sit the PISA test, the paper shows that the increase between 2003 and 2012 in the percentage of 15-16 year-olds attaining baseline Level 2 in mathematics and reading is more than twice as large as previously considered. While this paper only covers the PISA cycles from 2003 to 2012, in PISA 2015 Turkey’s performance declined in the three core domains of assessment, and coverage expanded only minimally compared to 2012.

Although all students benefited from the improvements in Turkey, girls and the poorest 40% of students benefited the most. This was largely because the dropout and early school-leaving rates among girls halved from 38% in 2003 to 20% in 2013.  During the same period, the percentage of disadvantaged 15-16 year-olds who attained Level 2 in reading increased more than threefold, from 13% to 46%. Since the percentage of advantaged students who attained Level 2 also increased – from 50% in 2003 to 82% in 2012 – the performance gap between rich and poor remains large.

Analyses reported in the paper also find that advantaged students are considerably more likely to be eligible to sit the PISA test – and to acquire basic proficiency in mathematics and reading – than disadvantaged students.

This new research shows the importance of accounting for who makes it into the PISA sampling frame. After all, survey results are only as representative as the students that make it into the sample. To address the problem of coverage, the OECD is piloting a survey for out-of-school 15-year-olds through the PISA for Development programme. The programme is currently conducting the field trial of this component, which will become available to all countries participating in the PISA 2021 survey.

Links
Who makes it into PISA? Understanding the impact of PISA sample eligibility using Turkey as a case study (PISA 2003 – PISA 2013) 

Who makes it into PISA? Illustrative Charts

The Programme for International Student Assessment (PISA)

PISA for Development

Join our OECD Teacher Community on Edmodo

Photo credit: @shutterstock