S5-AS3-3 - Estimating Programming Skills with Combined M-ERS and ELO Multidimensional Models

2. Research-to-Practice Full Paper
Fabiana Zaffalon1, 2 , Ricardo de Souza1, André Prisco1, Rafael Penna1, Jean Luca Bez3, Neilor Tonin3, Silvia Botelho1
1 Universidade Federal do Rio Grande - FURG
2 Instituto Federal de Educação, Ciência e Tecnologia Sul-rio-grandense - IFSul
3 Universidade Regional Integrada do Alto Uruguai e das Missões Erechim

This work presents a Full Paper in the Research-to-Practice category associated with a model for assessing multidimensional skills in online learning. In recent years, the most prominent educational assessment methods are those that aim to present accurate data on the construction of competences or skills acquired during academic education, not by means of punctual assessment, but by an assessment that tracks their progress over time and that take into account the various skills involved in solving problems, whether they are presential or distance learning students.  There are methods that estimate students' skills, including Item Response Theory (IRT) and ELO. IRT is a set of mathematical models that seeks to represent the probability that an individual will correctly answer a question about an item, due to a parameterization that establishes the relationship between the difficulty of an item and the ability of an individual to answer it properly. The relationship between the probability of an individual to give a certain answer to an item and their skills in the evaluated knowledge area is expressed in a way that the greater the individual's ability, the greater the probability of answering the item correctly. The IRT presents multidimensional models that contemplate cases in which more than one skill is required for the student to answer an item correctly. On the other hand, ELO, widely used for evaluation in international chess rankings, aims to classify players through their game histories, through a statistical classification that calculates ability for competitors or machines in competitions, this model deals with a player's skill compared to another player. There is an adaptation of the ELO that adapts the metric to relate the student to the problem with which he interacts and in previous works we have extended the ELO model, which is a scalar value for each student and for each learning object, to a multidimensional model where each dimension is a skill developed to solve programming problems. However, this multidimensional ELO model assumes that the relationship between the student and the learning objects comprises a set of independent, orthogonal skills. Considering that the programming exercises involve skills that are not independent, in this work we propose a Multidimensional ELO model for non-orthogonal skills, such proposal is the adaptation of the embedded ELO model to a multidimensional IRT model. This model aims to update several non-orthogonal skills simultaneously, assuming that the lack of one skill can be compensated for by the higher level of another skill. To validate the model, we used a database made available by an Online Judge from Brazil, a platform that contains 1.163 programming problems and 62.997 registered users. Comparing the results generated by the two models, we conclude that, for online platforms of programming problems, the multidimensional ELO model for non-orthogonal skills was more satisfactory, because through it is possible to observe the evolution of student’s skills individually.