Analytics & Validation
Using a new educational tool can be a challenging (en even a daunting) task for teachers, partially because when a teacher uses these tools in class may think that he/she loses control of what is going on during the class.
One mechanism to face this problem is to provide some analytics inside the tools (from just mere statistics or a more elaborated dashboard), which can provide enough information/metrics to the teacher to understand what is going on during the activity or just to have some metrics that can be accessed afterward.
IMPRESS provides three main tools (CodeDefenders, FormalZ, Quiz Repository), and two supporting tools (RAGE Analytics, SIMVA). During the project, we have analyzed how these tools can be integrated into a course, reaching a consensus about the needed analytics for each of the main tools:
- CodeDefenders provides leaderboards associated with a specific CodeDefender challenge that is useful and used in class (look to some of the pictures during the second multiplier event, https://impress-project.eu/impress/), but also provide more global statistics that are available if you run your installation of CodeDefenders.
- QuizRepository provides analytics at both the question and questionnaire levels, so a teacher can analyze the actual impact of the questions that you contribute to the quiz repository.
- FormalZ provides during class statistics provided through RAGE Analytics and summative statistics that can be accessed afterward.
The RAGE Analytics platform, one of the main results of the H2020 RAGE project has been extended to adequate its functionalities to the IMPRESS project needs (GitHub repository). For more details, have a look to the year 2 analytics summary.
Serious Games / Educational tools validation
Traditionally the validation of educational tools, including serious games, has been performed using pre-post tests. Still, additional preparation and tasks must be done to prepare the validation experiment.
To help with his process we have created Simva AKA Simple Validator, AKA SurveyManager is a tool created and designed to simplify the task of validating educational components (such as serious games or online material) by integrating survey tools, educational analytics systems, as well as a lot of flexibility to interconnect multiple resources that are part of the validation or learning process. Simva is specially designed to be simple and intuitive, so that the learning curve and validation process is much more robust, avoiding errors, and ensuring data security. SIMVA it is available in GitHub.
The management of Simva is characterized by the unification of the data of each participant coming from multiple platforms in a single platform. In this way, the analysis of the effectiveness of the tools to be validated is much easier. This is done through a unified user management, authorization and authentication. Simva is able to communicate with each system and specify who is participating and what task they should perform next.
Simva, although it can be integrated as another component, is available as a framework including all the components and services necessary for its operation, including an SSO, LimeSurvey, Minio and Kafka, RAGE Analytics, etc. Simva’s architecture can be seen in the following diagram:
Simva allows the researcher, in a very brief way: to create groups, to add participants to the groups, to create studies, test branches within those studies, activities within the test branches, and to assign groups to the studies. Once everything is set up, the experiment can be carried out, with the participants logging in to Simva to participate in the game.
For participant management, automatic creation of users with random anonymous identifiers (also known as tokens) is allowed. These are distributed digitally or physically (via a downloadable and printable PDF) to participants. For example: if a teacher wants to validate an educational game he is developing, he can also control his students by using the printable PDF and noting the names of his students, and can later relate the results to the students.
- Ivan Perez-Colado, Iván Martínez-Ortiz, Manuel Freire, Wishnu Prasetya, Baltasar Fernández-Manjón (2020). Simplifying serious games and analytics deployment in a virtual campus using LTI and xAPI. X International Conference on VIRTUAL CAMPUS. Tetouan (Morocco), 3-5 December 2020.
- Ivan Perez-Colado, Víctor Manuel Pérez-Colado, Iván Martínez-Ortiz, Manuel Freire, Baltasar Fernández-Manjón (2020): Simplifiying Serious Games Authoring and Validation with uAdventure and SIMVA. 2020 IEEE 20th International Conference on Advanced Learning Technologies (ICALT) IEEE DOI 10.1109/ICALT49669.2020.00039.
- Cristina Alonso-Fernández, Ivan Perez-Colado, Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2020): Applications of Simva to simplify serious games validation and deployment. IEEE Revista Iberoamericana de Technologias del Aprendizaje (RITA).
- Ivan Perez-Colado, Víctor Manuel Pérez-Colado, Iván Martínez-Ortiz, Manuel Freire, Baltasar Fernández-Manjón (2020): Simplifiying Serious Games Authoring and Validation with uAdventure and SIMVA. 2020 IEEE 20th International Conference on Advanced Learning Technologies (ICALT) 2161-377 DOI 10.1109/ICALT49669.2020.00039.
- Cristina Alonso-Fernández, Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Simplifying the validation and application of games with Simva. The 4th International Symposium on Emerging Technologies for Education In conjunction with ICWL 2019 (SETE 2019), 23-25 September, 2019, Magdeburg, Germany.
- Cristina Alonso-Fernández, Ivan Perez-Colado, Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Using Simva to evaluate serious games and collect game learning analytics data. Learning Analytics Summer Institute Spain (LASI-Spain), 27-28 June 2019, Vigo, Spain.
- Ivan Perez-Colado, Antonio Calvo-Morata, Cristina Alonso-Fernández, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Simva: Simplifying the scientific validation of serious games. 19th IEEE International Conference on Advanced Learning Technologies (ICALT), 15-18 July 2019, Maceió-AL, Brazil.
- Iván Martínez-Ortiz, Ivan Perez-Colado, Dan-Cristian Rotaru, Manuel Freire, Baltasar Fernández-Manjón (2019): From heterogeneous activities to unified analytics dashboards. IEEE Global Engineering Education Conference (EDUCON), 9–11 April, 2019, Dubai, UAE.
- Antonio Calvo-Morata, Cristina Alonso-Fernández, Ivan Perez-Colado, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Improving Teacher Game Learning Analytics Dashboards through ad-hoc Development. Journal of Universal Computer Science, Vol. 25, No. 12, pp. 1507-1530. [JCR-SCIE 2018, IF 0.910 Computer Science – Theory & Methods].