What we really need, both in schools and in the real world, is a shift away from arguing to win, and towards rhetoric as understanding. It’s a genre that Associate Professor Larissa McLean Davies at The University of Melbourne has labelled the “treaty” genre. Under this genre of writing students must show empathy, try to understand other points of view, and find a solution rather than just win an argument.
Last October, Dr Perelman was commissioned to conduct a review of ACARA’s planned automated essay-scoring known as “robot marking”.
His review was critical, sparked concern among education ministers, and finally led to the scrapping of the plan.
Cameron Malcher speaks with Marten Koomen about his research into the process by which large-scale tests like PISA and NAPLAN affect school management and curriculum.
Marten Koomen frames the conversation around a discussion of collectivism, neoliberalism and skepticism. For collectivists, school is the responsibility of the state, whereas neoliberals consider it as another product to be consumed. While without effective governance, skepticism ends up in tragedy. Our current climate is very much in response to neoliberalism, however:
We are all part collectivist, individualists neoliberals and skeptics, so to identify in one corner is disingenuous.
The key question that Koomen tries to address is: How did Victoria go from a state that was a leader in content knowledge and democratic values to the launch of a content-free platform driven by the terror of performativity? As he explains,
They had this idea of the net, but no idea of the content … a complete infatuation with the technology.
Discussing PISA, Koomen provides some background to computer-based testing and the ‘Koomen Model’. The model involved providing schools with standardized devices for the consistency of data. It failed based on pressure.
In part, Koomen’s model tells us something about the data and what it tells us. There are groups out there that want the outcomes without the content or context. Koomen returns again and again to the difference between entity realism vs. constructivism:
Entity Realism = things are real
Constructivism = things agreed upon
Realists ignore context as it is not mapped back to a central curriculum. It also allows for the insult of the human spirit through comparison of outcomes, ratio and market results. For example, NAPLAN uses Item Response Theory, a format that does not allow any direct recall or reference to learning and development. This leads to the situation where a student can ‘improve’ yet remain on the same score. Margaret Wu explains this in her chapter in National Testing in Schools, while Sam Sellar, Greg Thompson and David Rutkowski elaborate on it in The Global Education Race.
For Koomen our decline in these scales comes back to a focus on the market:
Neoliberalism considers content as: self-evident, real, axiomatic, socially constructed and marketable. In a way that supports the status quo.
This leads to conversations with students in regards to points on a scale, rather than aspects of context and development. For example, it is easier in the media to talk about a change in ratios or job rates, rather than the collapse in the car industry and what impact that has for the state. This allows for the rise of education conferences based around data with little reference to the local context.
The answer Koomen closes with is to work together though associations to make systemic change.
We have had nearly 10 years of Labor’s MySchool website, which encourages parents to play the school system like the stock market. Low scores are punished with low enrolments, as privileged families flock to high-performing schools, and the least socially mobile remain at schools with the least resources to support them.
As a result, when public schools in Victoria have received meagre funding increases, these are too often wasted on programs that principals think will boost scores and reputation – even if they undermine real learning. Despite plenty of evidence that streaming actually reduces student achievement, select-entry programs are breaking out like algae plagues around the state. As are uniform policies that mimic private schools in pettiness and pricing.
English tasks were to be marked by computers this year, but the proposal caused rancour among teachers’ unions who launched a campaign against it.
Now, the Education Council, which is comprised of all state and territory education ministers, has announced the move towards automated essay scoring will be halted.
Collection deals with NAPLAN in Australia, but our introductory and concluding chapters seek to situate the research reported here in a broader global context, aware of the circulation today of globalised education policy discourses and the significance of international testing as a complement to national testing such as NAPLAN.
1 National testing from an Australian perspective
Unlike other national testing regimes such as the National Assessment of Educational Progress (NAEP) in the US or the Pan-Canadian Assessment Program (PCAP), NAPLAN is a census test, not a sample test.
NAPLAN data are thus used for a variety of purposes, including governing school systems, accountability purposes, managing staff within systems and schools, and making educational decisions regarding curriculum and pedagogy in systems, schools and classrooms.
Together, NAPLAN, MySchool and the raft of programs and contractual arrangements between governments and schools that reference testing data illustrate the pervasiveness of technocratic rationality in Australian schooling
NAPLAN was established to improve teaching and learning outcomes, but one significant effect has been that much teaching is now aimed at improving NAPLAN scores.
NAPLAN data were useful in providing a common language for communication between principals, teachers and parents about student progress and achievement.
2 What national testing data can tell us
In summary, we would say that a NAPLAN test only provides an indicative level of the performance of a student: whether the student is struggling, on track, or performing above average. The NAPLAN tests do not provide fine grading of students by their performance levels because of the large uncertainties associated with the ability measures.
If teachers do not change the way they teach, the school mean scores for a year level can vary within a range of 32 NAPLAN points for 90% of the time if we have the opportunity to repeatedly allocate random samples of potential students to this school. Compare this margin of error with the expected annual growth rates of 44 points at Year 3, 28 points at Year 5, and 21 points at Year 7; the fluctuation in school mean scores due to a particular cohort of students has a magnitude close to one year of growth. This means that for many schools with a year level size of 50 or fewer, the average school performance could change significantly from one calendar year to another.
We need to always remember that using student assessment data to evaluate teachers is making an inference, since we have not directly measured teacher performance. The validity of making this inference needs to be checked in every case.
One should never jump to conclusions of ineffective schools whenever NAPLAN results are low. NAPLAN results indicate where further investigations are warranted.
As teacher effect accounts for only a small portion of the student achievement variance, individual teacher effect is likely to be swamped by the large variations in student abilities in a class. This is a reliability issue.
In conclusion, national testing data can inform us about performances of large groups of students, but not tell us a great deal about individual students or schools. National testing data cannot provide teacher performance measures, so there should not be any link between student test results and teacher appraisal or pay. National testing data have the potential to inform teaching and learning, and to frame education policies. However, we need to ensure that evidence-based decision making is backed by sound data and valid inferences.
3 The performative politics of NAPLAN and MySchool
Focusing on NAPLAN and MySchool as interesting objects – as actors in their own right, rather than as effects or products of neoliberal governance strategies – provides the opportunity to explore the technologies and mechanisms through which such objects serve to delegate trust, create new intimacies and reorganise relations.
By providing access to much more detail about each school, it brought parents closer to knowing their child’s school. It also revealed to schools themselves information that they previously did not have about themselves and about other schools.
Here I take NAPLAN and MySchool to be calculative objects – objects that resulted from policy decisions, to be sure, but which also became participants in the policy arena, actively rearranging the goals of schools, parents, teachers and policy makers and bringing to the forefront new issues and problems. I present four specific features or functions of interesting objects: creating new intimates, translating interests, displacing trust and creating informed publics.
Not only did MySchool become a technology through which the government entered intimate spaces of schools, schools themselves entered intimate spaces of living rooms and kitchens through discussions between parents
By involving parents in the job of keeping schools accountable and in continually improving their performance, parents and the government were cast as intimates – partners in the shared enterprise of school improvement.
By inserting itself between parents and their child’s school, MySchool attempted to enrol parents as canny stakeholders, casting the schools as secretive actors who were reluctantly being forced to reveal information they would rather have kept to themselves
NAPLAN and MySchool thus changed the original goals, motivations and plans of various actors
NAPLAN and MySchool thus created relations of distrust and suspicion between schools and the government, as well as schools and the public. They displaced trust from local actors with immediate knowledge and delegated trust instead to distant and impersonal actors.
NAPLAN and MySchool produce an abstract, impoverished and interested version of the very complex phenomenon of schooling in Australia. However, these interested observations of NAPLAN and MySchool are not merely providing useful, detailed accounts of Australian schooling; rather, they are actually changing the very nature of Australian schooling, so that it is beginning to more closely resemble the abstract version presented on the MySchool website. Rather than NAPLAN and MySchool reflecting an abstract version of Australian schooling, they are perhaps remaking Australian schooling in their image.
4 Questioning the validity of the multiple uses of NAPLAN data
As Strathern (1997) states: ‘When a measure becomes a target it ceases to be a good measure’ (308).
In the context of NAPLAN, while the tests may measure attainment in numeracy or literacy, it is questionable whether the information from these tests can be used validly for explaining how well the school has performed. Yet the aggregation of test scores across students to provide composite measures of educational effectiveness for teachers, schools, states or even the nation are commonly used in education for accountability purposes.
5 Local experiences, global similarities: teacher perceptions of the impacts of national testing
What policymakers intend is always mediated by how policy ‘hits the ground’, or is enacted, by individuals in diverse, complex community and institutional settings.
It must be stressed that NAPLAN is designed to change practice and behaviour through the emphasis on test-based accountabilities. However, not all change is desirable
The most dangerous possibility of testing data is that it distorts and corrupts the very processes it intends to measure. As education policy makers seem intent on continuing to use test data to steer practice from a distance, it remains to be seen how this distortion can be prevented.
6 NAPLAN and student wellbeing: teacher perceptions of the impact of NAPLAN on students
In the case of schools, the use of NAPLAN results as a blunt accountability instrument through their publication on the MySchool website has significantly increased the pressure on schools to treat NAPLAN results as more than just a snapshot of student achievement at a particular point in time
First, rather than NAPLAN itself being the central issue of concern in this instance, it is the use of NAPLAN results in largely inappropriate ways that is likely to be generating serious negative consequences
Second, these types of findings, and the likely reasons behind them, suggest a serious lack of knowledge amongst some policy makers, bureaucrats, principals, teachers and parents about the limitations of NAPLAN results (and indeed, any single test score)
Overall, it seems evident that the NAPLAN program is generating stress-related responses amongst substantial numbers of students across Australia. While there is a need for further research to elucidate the reasons behind this, it is highly likely that the use of NAPLAN results in inappropriate ways is contributing to student stress through the messages sent to students in the words and actions of principals, teachers and parents. Blaming these groups is not the way forward – rather, the time has come to discuss the relevance of NAPLAN, whether the benefits are worth the substantial costs (including psychological), and if NAPLAN is to continue, what the appropriate, statistically defensible and reasonable use of student test results might look like.
7 Literacy leadership and accountability practices: holding onto ethics in ways that count
The common agreement for literacy is a school-based policy, collaboratively developed between teachers and leaders, that prescribes what should be included in the daily uninterrupted literacy block. The block includes: guided reading, Jolly Phonics (Reception – Year 2), explicit teaching of comprehension strategies, daily reading practice (Choosing to Read), shared reading, handwriting, writing, spelling program, grammar and punctuation, as well as the locally mandated assessments to be undertaken over the year and the SMARTA (Specific/Student focused; Measurable; Achievable; Relevant; Time-lined; Agreed) targets for reading endorsed by the region. All teachers are given copies of the literacy agreement in their induction folders at the beginning of the school year and they were posted prominently on the notice board in the staff room.A locally generated text, the literacy agreement has come into existence as a result of very low NAPLAN results. It not only reflects the programs that teachers considered to be valuable, but the shaping force of NAPLAN. In this way, NAPLAN regulates the school’s common literacy agreement, constitutes the literacy problem and coordinates everyday classroom work in more or less obvious ways. For instance, the literacy component of NAPLAN includes a reading comprehension test, a writing test (genre writing), a spelling test and a grammar and punctuation test
As we have seen, Sandford has engaged with the unavoidable accountability requirements associated with NAPLAN. We have shown the extent to which NAPLAN has evoked a narrow view of literacy as the practice of content-free skills and how this view is reproduced in the active and occurring text of the literacy agreement that shapes what happens in classrooms. Nevertheless, NAPLAN does not always dominate what can be said. The potential sedimentation of NAPLAN is unraveled and reworked, at least to some degree, in the literacy chats, a product of the school’s recognition of the teachers’ needs for professional mentoring conversations that take account of actual students and their learning trajectories. In these educative and dialogical spaces, the senior leader works with teachers to design pedagogical interventions for students whose progress in school literacy learning is cause for concern. However, it is not only a question of looking at data as an artefact of the student, as the excerpt of Carrie’s literacy chat indicates. In mediating translocal policies that might otherwise close down possibilities for engaging ethically with students, the senior leader offers teachers the possibility of creative and critical literacy pedagogies. Despite their value in turning teachers around to students’ knowledge and practices as resources for school literacy learning, such pedagogies are less and less visible in schools since the advent of NAPLAN.
8 Contesting and capitalising on NAPLAN
… a warm-up session to ensure students were ready to learn;an ‘I do’ session in which the teacher demonstrated the specific task which was the focus of the lesson;a ‘we do’ session in which teachers worked with students as a whole class to co-construct a model response;a ‘you do’ activity involving students working independently;and a ‘ploughing back’ session in which students revised the lesson objectives and outcomes
9 Understanding the politics of categories in reporting national test results
Strong average performance in numeracy by some LBOTE students is not simply ascribed to a cultural fixation on academic attainment but may be a reflection of numeracy skills attained through comprehensive educational backgrounds;this strong average performance clouds the heterogeneity of the LBOTE category;LBOTE classification encompasses a broad heterogeneous group of students, which in the absence of a measure of English language proficiency, is most evident when NAPLAN results are disaggregated according to visa status of LBOTE students. Visa, in turn, is informative about disadvantage related to prior educational opportunities because students of refugee background are performing far below those of other migration categories, particularly the skilled visa category;language proficiency levels and years of schooling are associated with NAPLAN outcomes; andstudents who are of refugee background, with reduced years of schooling, and in the early stages of acquiring English are most disadvantaged in NAPLAN test results, but are completely hidden in the LBOTE category.
NAPLAN data need to be interpreted and understood within the context of language learning, whereas, in its current form, the breadth of LBOTE can only render a shallow interpretation, which dangerously ignores understandings about academic second language development.
10 Students at risk and NAPLAN: the collateral damage
Evident in the above is how, over the years of NAPLAN administration, support for students with different needs – social and emotional, language background, learning difficulties – to participate in NAPLAN has narrowed to serve the priority of administrative consistency.
NAPLAN data were reported to have little utility compared to information already obtained: [NAPLAN] does not provide us with any information about students that we don’t already know ourselves. We profile our students. And it just gives us another piece of information that we would otherwise have anyway.(Principal, independent PY–12 school)
…teachers reported positive value from NAPLAN as confirming their own professional judgements
16 The life of data: evolving national testing
Following Simons (2014), international and national tests can be seen to function as global/national positioning devices, evidence of a new spatial disposition and, in Australia, evidence of the emergence of a numbers-based national system of schooling. While these developments provide some evidence of a world polity approach that talks about the global diffusion of modernity and also the global dissemination of a particular version of science and social science, they also reflect the global impacts of an Anglo-American model of school reform based very much on test-based, top-down modes of educational accountability.
There is a common perception that testing data are inert, lifeless objects that provide an unbiased and objective measure of educational process, practices and outcomes.
However, we must be careful in making this claim that there is a life of data. In its most extreme form, this can lead to positing data as an agentive actor that makes decisions and behaves in certain ways. This is clearly not the case – data are expressions of human subjectivity, an expression of the values, sensibilities, processes that lead to their creation, and then the paths that the data lay down for individuals in terms of their choices, actions and acts of enunciation. Data are thus part of new spaces of subjectivity that are not contained within human bodies, but instead extend into information systems such as testing regimes, but also other data-driven applications such as social media or mobile phone usage. To understand the life of data, then, is to recognise that data produce possibilities and are invoked through the behaviours and values that result from the production of data. We cannot see data as external to the production of subjectivity, rather as Guattari (1992) argues, there is a little piece of human subjectivity in each data point: the technologies that we use to engage with data ‘are nothing more than hyperdeveloped and hyperconcentrated forms of certain aspects of human subjectivity’ (18).
Data have a life, they are always and everywhere put to work, they are always and everywhere in motion. One demonstration of this principle was highlighted by Nichols and Berliner (2007). Their argument was that the higher the stakes attached to any single measure that is used to make important decisions about students, teachers and schools, the more liable it is that the initial measure becomes corrupted because the processes are distorted by the emphasis. This is called ‘Campbell’s Law’, which stipulates: …the more any quantitative social indicator is used in social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it was intended to measure.(Nichols & Berliner 2007: 27)For example, tests like NAPLAN, which are designed to measure student achievement in the constructs of basic literacy and numeracy skills, become corrupted when teachers devote excessive class time to preparing for the tests. In other words, the tests no longer measure constructs regarding literacy and numeracy, rather they begin to measure the construct of how well a teacher can prepare a class. Obviously this is a problem, if important decisions are being made about literacy and numeracy on data that do not measure what they purport to measure, such decisions may not drive the improvements that were intended.
If data have lives, they are enacted through the space and time of data, and notions like consequential validity advanced by test developers themselves speak to this life
The critical question then is ‘what ought to be the future orientation to data at all levels of schooling’? This is primarily a political question and it needs to trouble the thinking and work of politicians, policy makers, system leaders, principals, teachers, students, the broader community and also educational researchers. I
Given this, we are not opposed to national testing, but we do believe that our assessments of national testing clearly point to areas where action must be taken to reduce its negative effects in Australia and elsewhere.