Analytics & Validation

Analytics

Using a new educational tool can be a challenging (en even a daunting) task for teachers, partially because when a teacher uses these tools in class may think that he/she loses control of what is going on during the class.

One mechanism to face this problem is to provide some analytics inside the tools (from just mere statistics or a more elaborated dashboard), which can provide enough information/metrics to the teacher to understand what is going on during the activity or just to have some metrics that can be accessed afterward.

IMPRESS provides three main tools (CodeDefenders, FormalZ, Quiz Repository), and two supporting tools (RAGE Analytics, SIMVA). During the project, we have analyzed how these tools can be integrated into a course, reaching a consensus about the needed analytics for each of the main tools:

  • CodeDefenders provides leaderboards associated with a specific CodeDefender challenge that is useful and used in class (look to some of the pictures during the second multiplier event, https://impress-project.eu/impress/), but also provide more global statistics that are available if you run your installation of CodeDefenders.
  • QuizRepository provides analytics at both the question and questionnaire levels, so a teacher can analyze the actual impact of the questions that you contribute to the quiz repository.
  • FormalZ provides during class statistics provided through RAGE Analytics and summative statistics that can be accessed afterward.

The RAGE Analytics platform, one of the main results of the H2020 RAGE project has been extended to adequate its functionalities to the IMPRESS project needs  (GitHub repository). For more details, have a look to the year 2 analytics summary.

Serious Games / Educational tools validation

Traditionally the validation of educational tools, including serious games, has been performed using pre-post tests. Still, additional preparation and tasks must be done to prepare the validation experiment.

To help with his process we have created Simva AKA Simple Validator, AKA SurveyManager is a tool created and designed to simplify the task of validating educational components (such as serious games or online material) by integrating survey tools, educational analytics systems, as well as a lot of flexibility to interconnect multiple resources that are part of the validation or learning process. Simva is specially designed to be simple and intuitive, so that the learning curve and validation process is much more robust, avoiding errors, and ensuring data security. SIMVA it is available in GitHub.

The management of Simva is characterized by the unification of the data of each participant coming from multiple platforms in a single platform. In this way, the analysis of the effectiveness of the tools to be validated is much easier. This is done through a unified user management, authorization and authentication. Simva is able to communicate with each system and specify who is participating and what task they should perform next.

Simva, although it can be integrated as another component, is available as a framework including all the components and services necessary for its operation, including an SSO, LimeSurvey, Minio and Kafka, RAGE Analytics, etc. Simva’s architecture can be seen in the following diagram:

Simva allows the researcher, in a very brief way: to create groups, to add participants to the groups, to create studies, test branches within those studies, activities within the test branches, and to assign groups to the studies. Once everything is set up, the experiment can be carried out, with the participants logging in to Simva to participate in the game.

For participant management, automatic creation of users with random anonymous identifiers (also known as tokens) is allowed. These are distributed digitally or physically (via a downloadable and printable PDF) to participants. For example: if a teacher wants to validate an educational game he is developing, he can also control his students by using the printable PDF and noting the names of his students, and can later relate the results to the students.

Selected Publications