Guest blog: What happens when we correlate NAPLAN data with Comparative Judgement?

Comparative Judgement is an innovative new assessment that has recently become available in Australia. Traditional marking, including NAPLAN Writing, involves the use of a rubric; Comparative Judgement relies on many educators comparing one script to another until a continuum of scripts is formed.

In the pursuit to understand if Comparative Judgement can be a useful predictor for national writing assessment, No More Marking are able to make some interesting correlations. In the following graph we compared a cohort of 60 Year 5 students. We have their NAPLAN data from 2023 and their averaged Comparative Judgement data from 2022 and 2021.

There are reasonable concerns from Australian teachers that the introduction of typed NAPLAN Writing in Year 5 is a disadvantage to many students. Inequitable IT resourcing, minimal typing focus and the possibility of students writing less, are some of the disadvantages that we imagine will impact NAPLAN data.

Here is a summary of what we found using the correlation of both NAPLAN and Comparative Judgement data:

• If Comparative Judgement is using the same construct as NAPLAN writing, then we would expect to see students tracking toward the top right-hand quadrant where alignment for both NAPLAN and Comparative Judgement is positive. 90% of the students are doing exactly this, as expected.

• There are no students in the bottom right-hand quadrant. This is good news. It shows that there are no students with high Comparative Judgement scores and low NAPLAN scores, allowing us confidence in the Comparative Judgement data received for students in the Year 2, 4, 6, 8 & 10 projects.

• Having 2 different standardised data sets showed that typing had less of an affect than predicted. Comparative Judgement was handwritten so for most of the cohort this was directly comparable and student performance was not impacted by typing or handwriting.

• There are 6 distinct outliers in this data set – teachers have an advantage here because knowing our students means insights can be provided into why scores may be inconsistent with typical growth. We will attempt to shed some light on this.

What do we know about the outlying students?

Student 16, 7 and 34 are funded students accessing tutoring and Tier 3 instruction. Each of their scores trends high for NAPLAN, yet low for Comparative Judgement. The intervention they receive is skill-based for the most part, so against a rubric this may result in a higher score if sentence skills are being measured. We also know the challenges these students face are physical. Typing their NAPLAN scripts provides the equaliser here and perhaps explains why these students engaged better with a keyboard writing task.

Here is a comparative sample to demonstrate this:

Student 14, 13 & 21 are confident writers who produce consistently strong writing pieces in the classroom. What can be said when examining their Comparative Judgement scripts, is that a weakness lies in an attention-grabbing beginning and solid ending. Judgements made against a more entertaining script will therefore impact choice. When marked against a rubric however, any solid writing piece will score points where holistic measurement is not the precursor. When writers with a solid skill base are demonstrating difficulty when they are released to write, it gives us some direction into what instruction is needed to help them apply their skills more aptly.

Overall, Comparative Judgement has resulted in an accurate and efficient measurement process allowing students to be tracked in ways that we have not had before. The results of the latest project included nearly 100 schools from across Australia. For 2024 we will be moving to typed tasks for Years 4, 6, 8 & 10 to mirror the national testing. Year 2 will remain handwritten.

As we continue to track cohorts across time, we hope to share more about the correlations in student writing from large data sets.

What can be said from this small sample:

• Typing and handwriting were equally impacted by sentence level knowledge.

• Typing provided equity for some students to be able to showcase their skills and knowledge in a more measurable way.