Implementing Computer Adaptive Testing for High-Stakes Assessment: A Shift for Examinations Council of Lesotho

Abstract

We examine the feasibility of implementing Computer Adaptive Testing (CAT) for high-stakes assessments in Lesotho, specifically through the Examination Council of Lesotho (ECoL). CAT, a cutting-edge testing method, enhances precision and efficiency by adjusting test items in real-time based on an examinee's ability. While CAT has gained widespread global adoption, its implementation in developing countries presents significant challenges, particularly regarding infrastructure, expertise, and resource limitations. Our research addresses a critical gap in understanding how CAT can be effectively integrated into educational systems in developing contexts such as Lesotho. We conducted a literature review across five databases—Google Scholar, ERIC, PsycINFO, JSTOR, and PubMed—retrieving 48 studies published between 2013 and 2023. Of these, 18 studies met our inclusion criteria, focusing on CAT's advantages, challenges, and real-world applications in educational assessments. We applied thematic analysis to identify key benefits and barriers, focusing on precision, fairness, and infrastructure requirements. We outline the essential stages of CAT development, including feasibility studies, item bank creation, pretesting and calibration, specification determination, and live CAT publication. We highlight critical tasks such as using Monte Carlo simulations to validate CAT feasibility and developing a robust item bank calibrated with Item Response Theory (IRT). We also address challenges like building a robust technological infrastructure, providing comprehensive stakeholder training, and securing adequate funding. We emphasize the importance of continuous evaluation and stakeholder engagement to ensure CAT's successful implementation and sustainability. Global trends indicate growing adoption, driven by advancements in psychometrics and technology. CAT has the potential to offer more equitable and accurate assessments, making it a promising solution to improve educational outcomes in Lesotho.