F5-T4-4 - An empirical evaluation of Reflective Writing Framework (RWF) for Reflective Writing in Computer Science Education
2. Research-to-Practice Work In Progress1 Huda Alrashidi
2 Thomas Daniel Ullmann
3 Mike Joy
Introduction
This research to Work in Progress Paper describes a manual annotation evaluation over four iterative annotation cycle of the proposed reflective writing framework to Computer Science (CS) education. The accuracy of an annotated reflective writing framework can be increased through the evaluation and revision of the annotation scheme to ensure a reliable and valid of the framework. Up to our knowledge, there is a lack of literature related to the accuracy of the reflective writing framework in CS education. This paper aims to describe a novel Reflective Writing Framework (RWF) that has been applied to annotate students’ reflective writing corpus for CS education.
The paper approaches this goal to (1) develop a framework for reflective writing in CS education through the manual iterative cycle and (2) empirically examine a framework indicator that can be manually assessed through annotators’ comments and suggestions. We focus on the following research question; RQ. What are potential reflective writing indicators can be assessed?
Inter-rater reliability (IRR) was undertaken to revise an iterative cycle of the initial RWF development over the four iterative annotation cycle with four independent annotators. Annotators were recruited based on their experience of assessing formative reflective writing knowledge of reflective writing to produce reliable and clear guidelines based on the annotators’ comments and suggestions. That aims to refine the description of the RWF’s levels and indicators to ensure the quality of producing annotated reflective writing corpus for CS education. The data were collected by the CS Department at the authors’ university as part of its normal assessment process and then provided to the researchers fully anonymized. Four iterative annotation cycle were employed to annotate 1200 sentences. The result shows that the four iterative annotation cycle accuracy of IRR increases from 0.5 to 0.8, which was substantial to an almost perfect agreement. This paper contributes to CS education on the annotated corpus that can be potentially used for generating an intelligent tutoring assessment using machine learning algorithms of students' reflective writing.