May 18, 2022
  • May 18, 2022

ACER: developing statistical software for more than 25 years

By on July 3, 2020 0


Developed internally by ACER to support research and evaluation programs, ACER ConQuest is also available to the public. Version 5 brings new features such as the ability to use simulation methods to test the fit of items, the ability to treat item parameters as random effects, and to estimate models using the ‘Monte Carlo Markov Chain Estimate (MCMC).

ACER ConQuest 5 is the culmination of over 25 years of statistical software research and development. Along the way, many large-scale evaluations and other research studies have benefited from ACER software.

The story of ACER ConQuest and its predecessor, QUEST, begins in 1983 with the arrival of lead author Ray Adams. In 1983, ACER used three key software: I tan, internal software developed by Lindsay Mackay and Steve Farish to perform Classical Test Theory (CTT) analysis; BICAL, software for fitting the Rasch model to dichotomous data; and CREDIT, developed by Geoff Masters to fit partial credit and similar models to polytomous data. This software was run on a single DEC VAX-11 minicomputer. Analyzes were entered via a user on a terminal connected to the machine, and the output was printed by a line printer.

Figure 1. Digital Equipment Corporation (DEC) VAX-11/750 minicomputer. Rhode Island retro-computing company CC BY-SA 3.0

As ACER grew and more and more people undertook work on projects, more computing power was needed and it became evident that the development of new software could streamline projects. This included large projects like the NSW Basic Skills Testing Program (1989) – one of the state-based assessment programs that was a precursor to the National Assessment Program (NAP). QUEST was developed in 1990 by Ray Adams and Khoo Siek Toon. QUEST has integrated CTT Analysis and Element Response Theory (IRT) into a single interactive application that runs on desktop computers. This increased the number of people who would work simultaneously. QUEST has also extended the type of models that can be fitted, including credit scale and partial scoring models using joint maximum likelihood estimation (JML).

As the diversity of analytical work undertaken increased, it was necessary to adapt more varied models. To do this, a new statistical software, ACER ConQuest, was developed that could fit a highly generalized item response model that could be parameterized to fit specific models, including models conventionally named as the partial credit model. ACER ConQuest also implemented maximum marginal likelihood (MML) estimation, allowing plausible values ​​to be drawn – a method now central to all large-scale assessments – and could accommodate element response models. multidimensional. ACER ConQuest was first written in the FORTRAN programming language (and at this point was called MATS: Multi Aspect Test Software) and was later rewritten in C ++. MATS and ConQuest was written in collaboration with Mark Wilson at the University of California, Berkeley.

Figure 2. ACER ConQuest was developed by the Australian Council for Educational Research (ACER) in partnership with the University of California at Berkley.

The use of ACER ConQuest grew further when ACER partnered with the International Association for the Evaluation of Educational Achievement (IEA) and led the IRT scale-up of early trends in the international study of mathematics and science (TIMSS 1995). ACER ConQuest 2 focused on the user experience. After being rewritten by Ray Adams and Margaret Wu, ACER ConQuest 2 could do user-friendly things like automatically producing design matrices by iterating over data. ACER ConQuest 2 ran on the Windows operating system, including Windows NT, but remained compatible with 32-bit VAX machines for large jobs.

Version 3 of ACER ConQuest is the result of ACER’s work on the OECD Program for International Student Assessment (PISA). ACER led the international undertaking PISA consortium from 1998 to 2012 and continues to lead the Australian component of the study, including the most recent cycle in 2018. With the largest team of analysts to date, features Such as a graphical user interface (GUI) and the ability to generate valid syntax through the use of point-and-click menu commands were developed to support as many users as possible. Additionally, native plotting functions have been added to export a range of plots, including the eponymous Wright maps.

ACER ConQuest 4 appeared after this and added new features, including the ability to estimate paired comparison models (Bradley-Terry-Luce (BTL) model), directly manipulate matrices, and add more analyzes. CTT, in particular the estimation of the Mantel-Haenszel tests of differential functioning of the elements (DIF). Perhaps the most notable evaluation program to use ACER ConQuest 4 is NAPLAN, which uses a multidimensional item response model with a latent regression model to draw plausible values ​​(PV). This expansion of functionality was in part conducted to support new areas of research, including enabling ACER’s Global Education Monitoring Center to help countries report on their progress towards the Sustainable Development Goal. 4. ACER ConQuest 4 has also been used to evolve a new family or region assessments, including Pacific Island Numeracy and Literacy Assessment, Primary Learning Measures in South East Asia and monitoring trends in education growth in Afghanistan.

Although it has only just been released, ACER ConQuest 5 has already supported important research. The OECD international study on early learning and well-being took advantage of the possibility of estimating large-dimension IRT models to test the veracity of the simultaneous estimation of abilities in ten domains of learning. results. ACER is also undertaking research by applying a wider variety of flexible and descriptive models (eg explanatory IRT models, tree-based IRT models). Going forward, a new generation of assessments will be scaled up using version 5, while the next progression of ACER’s research program will explore the intersection of IRT and new frontiers of analysis, including machine learning.

Find out more:
For more information on ACER ConQuest 5, visit www.acer.org/conquest

If you’re interested in digging deeper into educational data analysis, our Understanding Rasch Theory of Measurement Theory course will teach you everything you need to know. For more information: https://www.acer.org/au/professional-learning/postgraduate/rasch.

Leave a comment

Your email address will not be published.