He, Q;
Borgonovi, F;
Paccagnella, M;
(2021)
Leveraging process data to assess adults' problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks.
Computers & Education
, 166
, Article 104170. 10.1016/j.compedu.2021.104170.
Preview |
Text
He_Borgonovi_paccagnella_Compuers&Education.pdf - Accepted Version Download (911kB) | Preview |
Abstract
This paper illustrates how process data can be used to identify behavioral patterns in a computer-based problem-solving assessment. Using sequence-mining techniques, we identify patterns of behavior across multiple digital tasks from the sequences of actions undertaken by respondents. We then examine how respondents’ action sequences (which we label “strategies”) differ from optimal strategies. In our application, optimality is defined ex-ante as the sequence of actions that content experts involved in the development of the assessment tasks identified as most efficient to solve the task given the range of possible actions available to test-takers. Data on 7462 respondents from five countries (the United Kingdom, Ireland, Japan, the Netherlands, and the United States) participating in the Problem Solving in Technology-Rich Environment (PSTRE) assessment, administered as part of the OECD Programme for the International Assessment of Adult Competencies (PIAAC), indicate that valuable insights can be derived from the analysis of process data. Adults who follow optimal strategies are more likely to obtain high scores in the PSTRE assessment, while low performers consistently adopt strategies that are very distant from optimal ones. Very few high performers are able to solve the items in an efficient way, i.e. by minimizing the number of actions and by avoiding undertaking unnecessary or redundant actions. Women and adults above the age of 40 are more likely to adopt sub-optimal problem-solving strategies.
| Type: | Article |
|---|---|
| Title: | Leveraging process data to assess adults' problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1016/j.compedu.2021.104170 |
| Publisher version: | https://doi.org/10.1016/j.compedu.2021.104170 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Problem-solving skills, Process data, Longest common subsequence, PIAAC, Sequence mining |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education > IOE - Social Research Institute |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10124063 |
Archive Staff Only
 |
View Item |
