Validity of Log-Based Indicators of Algorithmic Competence in An Adaptive Programming Environment: A Simulation Study

Abstract

Background. Adaptive programming environments increasingly report students’ competence from log data such as attempts, hints, solution time and written reflections. Such “digital trace” indicators are attractive for formative assessment, but their validity is rarely examined before they are shown to teachers and students. Purpose. This study examines whether the four indicators of a process-based learner model, implemented in the AlgoAdapt system to estimate the cognitive, practical, metacognitive and reflective components of algorithmic competence, measure the constructs they are intended to measure. Methods. Following an evidence-centred perspective on assessment, the scoring code of the system was applied without modification to logs generated for 3,000 synthetic learners whose four latent component traits were known. Convergent and discriminant validity, the influence of a construct-irrelevant trait (verbosity), parallel-forms reliability for windows of 10–80 attempts and robustness to two response strategies (keyword padding of reflections and copying generated solutions) were analysed. Results. All indicators correlated substantially with their target traits (r = .78–.87) and were reliable for the default window of 40 attempts (r = .91–.96). However, the cognitive and practical indicators were practically identical (r > .99), the metacognitive indicator was partly driven by practical skill (β = .38), and verbosity explained part of the reflective indicator (β = .48). Copying generated solutions moved all copiers into the “high” band on three components. Conclusions. Reliable process indicators are not automatically valid. We propose concrete revisions—separating evidence sources, adjusting time for correctness, rubric-based reflection scoring and process checks that resist copying—that should be made before such indicators inform grades.