Refine
Document Type
- Article (7)
- Doctoral Thesis (1)
Has Fulltext
- yes (8)
Is part of the Bibliography
- no (8)
Keywords
- PISA (3)
- log data (2)
- paper-based assessment (2)
- Aufmerksamkeit (1)
- Aufmerksamkeitskomponenten (1)
- Complex problem solving (1)
- Computer-based assessment (1)
- ICT Engagement (1)
- Konzentration (1)
- Log data (1)
This paper addresses the development of performance-based assessment items for ICT skills, skills in dealing with information and communication technologies, a construct which is rather broadly and only operationally defined. Item development followed a construct-driven approach to ensure that test scores could be interpreted as intended. Specifically, ICT-specific knowledge as well as problem-solving and the comprehension of text and graphics were defined as components of ICT skills and cognitive ICT tasks (i.e., accessing, managing, integrating, evaluating, creating). In order to capture the construct in a valid way, design principles for constructing the simulation environment and response format were formulated. To empirically evaluate the very heterogeneous items and detect malfunctioning items, item difficulties were analyzed and behavior-related indicators with item-specific thresholds were developed and applied. The 69 item’s difficulty scores from the Rasch model fell within a comparable range for each cognitive task. Process indicators addressing time use and test-taker interactions were used to analyze whether most test-takers executed the intended processes, exhibited disengagement, or got lost among the items. Most items were capable of eliciting the intended behavior; for the few exceptions, conclusions for item revisions were drawn. The results affirm the utility of the proposed framework for developing and implementing performance-based items to assess ICT skills.
As a relevant cognitive-motivational aspect of ICT literacy, a new construct ICT Engagement is theoretically based on self-determination theory and involves the factors ICT interest, Perceived ICT competence, Perceived autonomy related to ICT use, and ICT as a topic in social interaction. In this manuscript, we present different sources of validity supporting the construct interpretation of test scores in the ICT Engagement scale, which was used in PISA 2015. Specifically, we investigated the internal structure by dimensional analyses and investigated the relation of ICT Engagement aspects to other variables. The analyses are based on public data from PISA 2015 main study from Switzerland (n = 5860) and Germany (n = 6504). First, we could confirm the four-dimensional structure of ICT Engagement for the Swiss sample using a structural equation modelling approach. Second, ICT Engagement scales explained the highest amount of variance in ICT Use for Entertainment, followed by Practical use. Third, we found significantly lower values for girls in all ICT Engagement scales except ICT Interest. Fourth, we found a small negative correlation between the scores in the subscale “ICT as a topic in social interaction” and reading performance in PISA 2015. We could replicate most results for the German sample. Overall, the obtained results support the construct interpretation of the four ICT Engagement subscales.
In this explorative study, we investigate how sequences of behaviour are related to success or failure in complex problem‐solving (CPS). To this end, we analysed log data from two different tasks of the problem‐solving assessment of the Programme for International Student Assessment 2012 study (n = 30,098 students). We first coded every interaction of students as (initial or repeated) exploration, (initial or repeated) goal‐directed behaviour, or resetting the task. We then split the data according to task successes and failures. We used full‐path sequence analysis to identify groups of students with similar behavioural patterns in the respective tasks. Double‐checking and minimalistic behaviour was associated with success in CPS, while guessing and exploring task‐irrelevant content was associated with failure. Our findings held for both tasks investigated, from two different CPS measurement frameworks. We thus gained detailed insight into the behavioural processes that are related to success and failure in CPS.
Complex problem solving (CPS) is a highly transversal competence needed in educational and vocational settings as well as everyday life. The assessment of CPS is often computer-based, and therefore provides data regarding not only the outcome but also the process of CPS. However, research addressing this issue is scarce. In this article we investigated planning activities in the process of complex problem solving. We operationalized planning through three behavioral measures indicating the duration of the longest planning interval, the delay of the longest planning interval and the variance of intervals between each two successive interactions. We found a significant negative average effect for our delay indicator, indicating that early planning in CPS is more beneficial. However, we also found effects depending on task and interaction effects for all three indicators, suggesting that the effects of different planning behaviors on CPS are highly intertwined.
The effects of aging on response time were examined in a paper-based lexical-decision experiment with younger (age 18–36) and older (age 64–75) adults, applying Ratcliff’s diffusion model. Using digital pens allowed the paper-based assessment of response times for single items. Age differences previously reported by Ratcliff and colleagues in computer-based experiments were partly replicated: older adults responded more conservatively than younger adults and showed a slowing of their nondecision components of RT by 53 ms. The rates of evidence accumulation (drift rate) showed no age-related differences. Participants with a higher score in a vocabulary test also had higher drift rates. The experiment demonstrates the possibility to use formal processing models with paper-based tests.
In this paper, we developed a method to extract item-level response times from log data that are available in computer-based assessments (CBA) and paper-based assessments (PBA) with digital pens. Based on response times that were extracted using only time differences between responses, we used the bivariate generalized linear IRT model framework (B-GLIRT, [1]) to investigate response times as indicators for response processes. A parameterization that includes an interaction between the latent speed factor and the latent ability factor in the cross-relation function was found to fit the data best in CBA and PBA. Data were collected with a within-subject design in a national add-on study to PISA 2012 administering two clusters of PISA 2009 reading units. After investigating the invariance of the measurement models for ability and speed between boys and girls, we found the expected gender effect in reading ability to coincide with a gender effect in speed in CBA. Taking this result as indication for the validity of the time measures extracted from time differences between responses, we analyzed the PBA data and found the same gender effects for ability and speed. Analyzing PBA and CBA data together we identified the ability mode effect as the latent difference between reading measured in CBA and PBA. Similar to the gender effect the mode effect in ability was observed together with a difference in the latent speed between modes. However, while the relationship between speed and ability is identical for boys and girls we found hints for mode differences in the estimated parameters of the cross-relation function used in the B-GLIRT model.
Die Arbeit befasst sich mit der Zusammenhangsstruktur einer Vielzahl von Aufmerksamkeitskomponenten, welche aus deutlich unterscheidbaren theoretischen Perspektiven postuliert werden. Untersucht wird insbesondere die Frage, inwieweit zu den auf konzeptueller Ebene differenzierten Aufmerksamkeitskomponenten entsprechende kognitive Fähigkeiten empirisch separiert und in ein gemeinsames, theorienübergreifendes Aufmerksamkeitsmodell integriert werden können. Zunächst wurde unter Fragestellung 1 untersucht, ob fünf häufig postulierte Aufmerksamkeitskomponenten in der Tradition Posners (s. Posner & Boies, 1971; Posner & Rafal, 1987), Alertness, räumliche Aufmerksamkeit, fokussierte Aufmerksamkeit, Aufmerksamkeitswechsel und geteilte Aufmerksamkeit, empirisch unterscheidbare kognitive Fähigkeiten darstellen. Im Anschluss daran wurde unter Fragestellung 2 analysiert, welchen Beitrag die Aufmerksamkeitskomponenten in der Tradition Posners zur Erklärung konzeptuell unterscheidbarer Aufmerksamkeitskomponenten aus Sicht der Handlungstheorie (Neumann, 1992), aus der Arbeitsgedächtnisperspektive (Baddeley, 1986) sowie aus Sicht der psychometrischen Konzentrationsforschung (z.B. Brickenkamp, 1994; Moosbrugger & Goldhammer, 2006) leisten. Unter Fragestellung 3 wurde eine faktorenanalytische Integration aller einbezogenen Aufmerksamkeitskomponenten in ein theorienübergreifendes, integratives Aufmerksamkeitsmodell vorgenommen. In Anlehnung an Theorien, welche Aufmerksamkeit sowohl auf perzeptive Prozesse als auch auf die exekutive Kontrolle bei der Bearbeitung komplexer Aufgaben beziehen (Bundesen, 1990; Logan & Gordon, 2001), wurden zwei Basisdimensionen bzw. grundlegende Aufmerksamkeitsfaktoren vermutet, nämlich perceptual attention und executive attention, welche interindividuelle Fähigkeitsunterschiede über die verschiedenen Aufmerksamkeitskomponenten hinweg erklären. Schließlich wurde unter Fragestellung 4 auf theoretischer wie auch auf empirischer Ebene untersucht, inwieweit sich Konzentration in den Erklärungsrahmen mehrdimensionaler Aufmerksamkeitsmodelle eingliedern lässt, indem eine konzeptuelle sowie statistische Rückführung von Konzentration auf mehrdimensionale Aufmerksamkeit vorgenommen wurde. An einer Stichprobe von 232 Studierenden wurde die Leistung in 13 Aufmerksamkeits- und Konzentrationstests erfasst. Konfirmatorische Faktorenanalysen zeigten, dass die fünf Aufmerksamkeitskomponenten in der Tradition Posners miteinander in Beziehung stehen, jedoch klar separierbar sind. Aus den getesteten Strukturgleichungsmodellen ging hervor, dass diese Aufmerksamkeitskomponenten signifikant und in unterschiedlicher Weise zur Erklärung von konzeptuell unterscheidbaren Aufmerksamkeitskomponenten beitragen. In hypothesenkonformer Weise erwiesen sich Aufmerksamkeitswechsel und geteilte Aufmerksamkeit als bedeutsam für die Erklärung von Konzentration sowie von handlungsorientierten Aufmerksamkeitskomponenten. Hinsichtlich des integrativen Aufmerksamkeitsmodells konnte die vermutete 2-Faktorenstruktur, welche einen Generalfaktor perceptual attention sowie einen spezifischen Faktor executive attention enthält, bestätigt werden. Das vielfach von Aufmerksamkeit getrennt behandelte Konzentrationskonstrukt konnte auf theoretischer Ebene durch Aufzeigen konzeptueller Entsprechungen in den Erklärungsrahmen mehrdimensionaler Aufmerksamkeit eingegliedert werden. In konsistenter Weise ließ sich Konzentration auf empirischer Ebene durch Aufmerksamkeitskomponenten nach Posner sowie die beiden Faktoren des postulierten integrativen Aufmerksamkeitsmodells substanziell erklären.