Development and evaluation of a framework for the performance-based testing of ICT skills

  • This paper addresses the development of performance-based assessment items for ICT skills, skills in dealing with information and communication technologies, a construct which is rather broadly and only operationally defined. Item development followed a construct-driven approach to ensure that test scores could be interpreted as intended. Specifically, ICT-specific knowledge as well as problem-solving and the comprehension of text and graphics were defined as components of ICT skills and cognitive ICT tasks (i.e., accessing, managing, integrating, evaluating, creating). In order to capture the construct in a valid way, design principles for constructing the simulation environment and response format were formulated. To empirically evaluate the very heterogeneous items and detect malfunctioning items, item difficulties were analyzed and behavior-related indicators with item-specific thresholds were developed and applied. The 69 item’s difficulty scores from the Rasch model fell within a comparable range for each cognitive task. Process indicators addressing time use and test-taker interactions were used to analyze whether most test-takers executed the intended processes, exhibited disengagement, or got lost among the items. Most items were capable of eliciting the intended behavior; for the few exceptions, conclusions for item revisions were drawn. The results affirm the utility of the proposed framework for developing and implementing performance-based items to assess ICT skills.

Export metadata

Author:Lena Engelhardt, Johannes NaumannORCiDGND, Frank GoldhammerORCiDGND, Andreas FreyORCiDGND, Holger HorzORCiDGND, Katja Hartig, S. Franziska C. WenzelGND
Parent Title (English):Frontiers in education
Publisher:Frontiers Media
Place of publication:Lausanne
Document Type:Article
Date of Publication (online):2021/05/13
Date of first Publication:2021/05/13
Publishing Institution:Universitätsbibliothek Johann Christian Senckenberg
Release Date:2021/08/31
Tag:assessment framework; behavioral indicators; information and communication technology skills; performance items; validation
Issue:art. 668860
Page Number:15
First Page:1
Last Page:15
This work was supported by the German Federal Ministry of Education and Research (grant numbers: 01LSA010, 01LSA010A, 01LSA010B). The publication of this article was funded by the Open Access Fund of the Leibniz Association.
Institutes:Psychologie und Sportwissenschaften / Psychologie
Dewey Decimal Classification:1 Philosophie und Psychologie / 15 Psychologie / 150 Psychologie
3 Sozialwissenschaften / 37 Bildung und Erziehung / 370 Bildung und Erziehung
Licence (German):License LogoCreative Commons - Namensnennung 4.0