Listened TER #109 – How large-scale tests affect school management with Marten Koomen – 04 March 2018 by Cameron Malcher from Teachers' Education Review
Cameron Malcher speaks with Marten Koomen about his research into the process by which large-scale tests like PISA and NAPLAN affect school management and curriculum.

Marten Koomen frames the conversation around a discussion of collectivism, neoliberalism and skepticism. For collectivists, school is the responsibility of the state, whereas neoliberals consider it as another product to be consumed. While without effective governance, skepticism ends up in tragedy. Our current climate is very much in response to neoliberalism, however:

We are all part collectivist, individualists neoliberals and skeptics, so to identify in one corner is disingenuous.

The key question that Koomen tries to address is: How did Victoria go from a state that was a leader in content knowledge and democratic values to the launch of a content-free platform driven by the terror of performativity? As he explains,

They had this idea of the net, but no idea of the content … a complete infatuation with the technology.

Discussing PISA, Koomen provides some background to computer-based testing and the ‘Koomen Model’. The model involved providing schools with standardized devices for the consistency of data. It failed based on pressure.

In part, Koomen’s model tells us something about the data and what it tells us. There are groups out there that want the outcomes without the content or context. Koomen returns again and again to the difference between entity realism vs. constructivism:

Entity Realism = things are real

Constructivism = things agreed upon

Realists ignore context as it is not mapped back to a central curriculum. It also allows for the insult of the human spirit through comparison of outcomes, ratio and market results. For example, NAPLAN uses Item Response Theory, a format that does not allow any direct recall or reference to learning and development. This leads to the situation where a student can ‘improve’ yet remain on the same score. Margaret Wu explains this in her chapter in National Testing in Schools, while Sam Sellar, Greg Thompson and David Rutkowski elaborate on it in The Global Education Race.

For Koomen our decline in these scales comes back to a focus on the market:

Neoliberalism considers content as: self-evident, real, axiomatic, socially constructed and marketable. In a way that supports the status quo.

This leads to conversations with students in regards to points on a scale, rather than aspects of context and development. For example, it is easier in the media to talk about a change in ratios or job rates, rather than the collapse in the car industry and what impact that has for the state. This allows for the rise of education conferences based around data with little reference to the local context.

The answer Koomen closes with is to work together though associations to make systemic change.

Bookmarked PISA-shock: how we are sold the idea our PISA rankings are shocking and the damage it is doing to schooling in Australia by Aspa Baroutsis (EduResearch Matters)
The PISA-shock type media coverage has huge policy effects. Governments make decisions that have lasting fallout on our education systems as a result of this coverage. However the deep inequities of performance based on socio-economic background that show up in detailed PISA results and the differences between the jurisdictional schooling systems is where the media should be shining the spotlight. This is where the real story of what is happening in school education in Australia can be uncovered. This is where policy makers should be searching for policy changing data.
Aspa Baroutsis and Bob Lingard provide a summary of their analysis of the PISA reporting in the media and the subsequent ‘shock’ that it induced. This provides a good starting point in understanding some of the challenges assocaited with PISA. For extended response, read Sam Sellar, Greg Thompson and David Rutkowski’s book, The Global Education Race, Taking the Measure of PISA and International Testing.
Liked PISA for personality testing – the OECD and the psychometric science of social-emotional skills by Ben Williamson (code acts in education)
SSES extends the reach of datafication of education beyond school walls into the surveillance of home contexts and family life, treating them as a ‘home learning environment’ to be assessed on how it enables or impedes students’ development of valuable socio-emotional skills

Ben Williamson provides a (very partial) overview of some of the key features of SSES. However, it does raise a few headline points:

SSES extends international-large scale assessment beyond cognitive skills to the measurement of personality and social-emotional skills

SSES will deliver a direct assessment instrument modelled on psychological personality tests

SSES enacts a psychological five-factor model of personality traits for the assessment of students, adopting a psychometric realist assumption that personality test data capture the whole range of cross-cultural human behaviour and emotions in discrete quantifiable categories

SSES extends the reach of datafication of education beyond school walls into the surveillance of home contexts and family life, treating them as a ‘home learning environment’ to be assessed on how it enables or impedes students’ development of valuable socio-emotional skills

SSES normalizes computer-based assessment in schools, with students required to produce direct survey data while also being measured through indirect assessments provided by teachers, parents and leaders

SSES produces increasingly fine-grained, detailed data on students’ behaviours and activities at school and at home that can be used for targeted intervention based on analyses performed at a distance by an international contractor

SSES involves linking data across different datasets, with direct assessment data, indirect assessments, school admninistrative data, and process metadata generated during assessment as multiple sources for both large-scale macro-analysis and fine-grained micro-analytics–with potential for linking data from other OECD assessments such as PISA

SSES uses digital signals such as response times and keystrokes, captured as process metadata in software log files, as sources for stealth assessment based on assumptions about their correlation with specific social-emotional skills

SSES promotes a therapeutic role for education systems and schools, by identifying ‘success’ factors in SELS provision and encouraging policymakers to develop targeted intervention where such success factors are not evident

SSES treats students’ personalities as malleable, and social-emotional skills as learnable, seeking to produce policy-relevant psychometric knowledge for policymakers to design interventions to target student personalities

SSES exemplifies how policy-relevant knowledge is produced by networks of influential international organizations, connected discursively and organizationally to think tanks, government departments and outsourced contractors

SSES represents a psycho-economic hybridization of psychological and psychometric concepts and personality measurement practices with economic logics relating to the management of labour market behaviours and human resources