The main aspects of analytics on which we focused during Y3 were:
Improve the stability and facilitate the deployment of the project’s results.
Create a tool to help with the validation of serious games and other educational activities.
As described in last year results, we performed a full integration between FormalZ and the RAGE Analytics platform, in order to provide near-realtime analytics to teachers during gameplay sessions. This integration was successful, and proved to be effective helping teachers to better understand what students were doing while playing the game. Although the idea of the FormalZ analytics dashboard was to provide an in-class tool, allowing teachers to oversee students from their computers, and thus provide proactive feedback, dashboards are even more important in a remote-based teaching scenario, such as the one we are currently immersed in as of this writing due to the global COVID-19 pandemic.
Thinking about the future of the project and its maintainability once IMPRESS is formally finished, and in order to simplify use of project results to potential users, this year we have focused on software quality and stability, including both initial deployment and actual usage. As a result, we have improved both the stability of FormalZ and the metrics offered in the analytics panel through RAGE Analytics. Another important aspect was to simplify the deployment of our results by third parties, and to enable ongoing feedback from potential users of the tools. In this regard, FormalZ has been extensively adapted to facilitate deployment using well-known best practices for software application development, such as use of Docker containers. The FormalZ installer now allows customization of tool installation, so that it is possible to either perform a minimal deployment on a PC or laptop for testing purposes, or to perform a full-fledged installation on a server for sustained use or to be accessed by a significant number of users. Moreover, in order to facilitate deployment, the integration with RAGE Analytics can be disabled during deployment, so that, if the educational setting does not require immediate feedback (e.g. FormalZ is used as a homework / assignment or without teacher supervision), a simpler installation can be performed. Finally, to streamline the configuration of RAGE Analytics and its integration with FormalZ, we have created a specific installer that allows deployment of an installation of RAGE Analytics pre-configured for integration with FormalZ.
In addition, to facilitate communication with potential users and following the usual conventions of open source projects, both FormalZ and RAGE Analytics are hosted in a well known open repository, GitHub. Potential users can not only download the results from GitHub but also start discussions there or request for help (e.g., opening an issue or requesting a new functionality). This aspect also increases the discoverability of our results for the software development community.
The agile methodology used in the project with some early-released working prototypes has allowed us to carry out different formative tests during the project to improve the final products. During these formative tests, we have collected both user feedback and evidence-based interaction information needed to carry out a more solid scientific studies about the validity and impact of the tools (Teaching Software Testing with the Code Defenders Testing Game: Experiences and Improvements). In both cases, it has been necessary to use other complementary tools such as user questionnaires to collect such information. This approach is aligned with the experience of project participants in the area of serious game evaluation. A research methodology based on pre-post tests is widely accepted as the gold standard, both for the formal evaluation of educational tools (gamified or not) in general and for serious games in particular.
Although the described methodology may seem simple at first glance, it usually requires significant preparatory work by the researchers conducting the formal evaluation, in order to prepare the experiment and the activities to be carried out by the participants during the experiment. For example, one or more questionnaires must be created for each experimental group, participants must be issued anonymized identifiers to avoid storing personally-identifiable information (if the internal regulations of the organization require it or to comply with the EU GDPR regulation), different activities must be orchestrated, and the information from the different activities performed during the experiment must finally be linked together after it is collected to prepare it for further analysis. In addition, if the tool under study generates analytical information, it must also be collected and linked with the aforementioned identifier to allow the appropriate analyses to be carried out. Based on this experience and to help researchers and other projects with similar goals to IMPRESS, we have created a tool called SIMVA to help with the entire evaluation process.
We have tested SIMVA inside the context of the IMPRESS project by integrating both SIMVA and FormalZ. Additionally, the e-UCM team has extensively used SIMVA in other contexts (Simplifying serious games and analytics deployment in a virtual campus using LTI and xAP). During this integration we identified certain limitations in SIMVA, regarding the complexity of its integration with other web-based educational tools such as FormalZ. Therefore, instead of just performing an ad-hoc integration between both tools, to make the process easily generalizable, we included support for the IMS LTI 1.3 standard by extending both tools (see figures 1 and 2). Adopting IMS LTI to facilitate the integration of both tools provides the additional benefit of allowing FormalZ, SIMVA or both to be launched from an LTI-compliant Learning Management System such as Moodle with a single click. This has allowed us to greatly decrease the burden of both deploying and using our tools, and help teachers to maintain their workflow and the tools they already know.
