For an adaptive assessment to function, the difficulty level of every item (or question) in a bank must be definitively known, a process done using item response theory.

Modern adaptive testing relies on item response theory (IRT), in which a probabilistic model is used to estimate both how challenging items are and how able learners are. Learners’ ability levels and items’ difficulty parameters are computed on the same scale, allowing the difficulty level of an item to be determined independently of the ability of the learner who takes it, something that classical statistics are unable to do.

In the personalised assessments for Wales, learners begin with a question that is approximately aligned to their year group. Once they answer this question, an algorithm calculates an interim ability estimate, and uses this to select an appropriate question to deliver next. A learner who answers a question correctly will receive a question that is slightly more challenging; a learner who answers a question incorrectly will receive a slightly easier question.

The software estimates the learner’s ability on a scale, and the purpose of each subsequent question is to refine the estimate, so questions are chosen to optimise those measurements. This process is repeated after every question, with ability rising and falling as learners get questions right or wrong, and the algorithm selecting questions at an appropriate level for a learner’s current ability estimate.

This process continues until the assessment system has covered the relevant aspects of the curriculum and covered the expected number of questions. At this point we will have sufficient evidence available to accurately estimate the ability of the learner completing the assessment.

There are several additional constraints on the adaptive algorithm in these assessments:

• Curriculum category constraints ensure that learners do not receive questions solely on one topic.
• Test length constraints ensure that learners receive an assessment of acceptable length.
• Exposure controls prevent any one question from being seen by ‘too many’ learners (the threshold of which can be adjusted).
• Item bank constraints. While learners can access questions from any year if they are sufficiently able, the earlier questions in each assessment are aimed at that learner’s own year group, unless their teacher has set a different starting point for them. Secondary school content is withheld for the youngest learners, so that they cannot access content on advanced topics in the secondary curriculum they may not have encountered in class yet.

This tailored assessment provides information on the strengths and areas for further development for individual learners and whole classes, so that teachers and learners know where to focus their efforts to improve progress.

The graph below maps a year 3 learner’s journey through an adaptive assessment. The red triangles are incorrect answers and the blue circles are correct answers. The y axis scale is the learner’s ability estimate.

The learner answers the first few questions correctly and quickly climbs the ability scale. Through a series of correct and incorrect answers, the difficulty range of the questions narrows, converging on the final, steady outcome. As the algorithm has reached a sufficiently accurate estimate, the assessment is terminated.

Schools are free to schedule the assessments at any point during the academic year that they consider most appropriate to gain benefits for learning and teaching. They can be scheduled up to twice a year. For example, a school could choose to use the assessment at the very beginning and very end of each school year, to see how a learner has progressed throughout the year. It is a statutory requirement for the assessment to be used once per year, with results reported to parents or carers.

Schools can choose to assess whole classes, small groups or individual learners according to their facilities, and they can do so at a time that works for them and their learners. As the personalised assessments are different for each learner, there is no need for a whole class of learners to take the assessments at the same time over concerns about cheating.

The assessments are entirely online: they are web-based with no local server or software installation required. They can be taken on desktop computers, laptops or tablets such as iPads or Chromebooks.

Assessments are around 30 questions long, though this varies by subject, and generally take between 20 and 40 minutes to complete.

Support for schools is available through online webinars, a user guide, and videos demonstrating how the assessments should be administered and how to access learner and group reports. Schools are encouraged to use the webinars to gain an understanding of how the assessments work and how they can use the information from the assessments to support teaching and learning.

Here are some example of questions from the personalised assessments:

See presentations that showcase the use of personalised assessments in schools here.

The assessments are accessed via the Welsh Government’s digital learning platform Hwb. Hwb provides all maintained schools in Wales with access to a wide range of digital services and content via a Welsh Government operated national active directory. This means that all learners and teachers have a single username and password, which significantly simplifies the authentication process for services such as personalised assessments.

Before learners can take personalised assessments:

• the headteacher, or member of staff acting on their behalf, needs to assign assessment roles to relevant staff
• staff log into Hwb, navigate to the personalised assessments in the Hwb tools menu and schedule assessments for learners
• learners log into Hwb and work through a familiarisation assessment, so that they understand the different question formats.

Personalised feedback is made available on the website one day after the assessments are completed. This is considerably faster than for the paper tests, which required marking by teachers. This also supports the key purpose of formative assessment, so that any changes to teaching can be implemented in a timely manner. Teachers have the opportunity to review individual feedback before releasing it to learners.

Language

The content of the English and Welsh versions of both the procedural numeracy assessment and the numerical reasoning assessment is identical; the questions have been developed in both languages simultaneously. However, the banks are different for the English and Welsh reading assessments, given that they are based on texts written in the respective languages.

The numeracy personalised assessments can therefore be taken in English or Welsh, and the language is selected when the assessment is scheduled by staff. However, a learner can view the translation at any point during the assessment by clicking an icon at the bottom of the screen. The pop-up that appears in an image, so that the learner cannot answer the question in the wrong screen.

All administration support, including helplines, the administration website, user guides and FAQs are provided in English and Welsh.

The personalised assessments are specifically designed for formative benefit, so that teachers have accurate and timely information about the reading and numeracy skills and subskills of their learners and an understanding of strengths and areas for improvement in those skills. The assessments are focused on understanding learner progress, and not school performance or accountability. The assessments offer teachers an attainment measure against a national benchmark (a standardised score that compares performance against all other learners in Wales), but the outcomes are purposefully not used as an accountability measure.

Adaptive assessments enable learners to move up or down the ability range so that all learners get approximately 50% of the questions right and 50% wrong. Learners may therefore move up and down the bank to access content from higher or lower years – challenging the more able while not demotivating those who are struggling. Allowing learners from different years to ‘roam’ through a single item bank means that the assessments truly test what the learners can do, making no judgement of what was expected to be taught in a given year.

One of the key challenges of national testing is sustaining learners’ engagement. The national personalised assessments have two key benefits here:

• Because each learner attempts questions at or about their level, the tests are engaging for all. Paper tests, which are the same for all learners in a given school year, typically have significant floor and ceiling effects (too easy for some learners, too difficult for others), meaning that not all learners experience a test at an appropriate level (the range of abilities within a year is much wider than the difference between years).
• The adaptive assessment typically takes less time than an equivalent linear test to reach an equivalent measurement accuracy. Shorter tests or assessments can mean better learner engagement.

Aside from the engagement benefits, there are operational benefits for schools:

• The assessments are personalised, so there is no need for whole year groups to take the assessment at the same time.
• The assessments are marked automatically, while the previous paper-based tests were marked in schools.
• The assessments offer high-quality feedback, providing schools with information on the skills of individual learners and whole classes.
• As the assessments are automatically marked and calibrated, this feedback is available the day after the assessment. This supports the key purpose of formative assessment, as learners get the information while the assessment is still fresh in their minds and changes to teaching can be implemented in a timely manner.