IMPRESS Analytics wrap-up

Analytics has been a key concern of the IMPRESS project since its inception. During the first and second years of the project, we settled the foundations needed to provide a scaffold for other project results. In particular, facilitating both a tight or a loose analytics integration. By tight integration, we mean that the tool has to use the API that provides RAGE Analytics or one of the libraries available, so it is possible to orchestrate the integration of both tools. By loose integration we mean that it is possible to upload coarse grained analytics (e.g. score, progress, etc.) by using a CSV or Excel file. This allowed us to get the analytic data that we can get from third part tools such as Kahoot! and Quizziz that do not provide an API or we do not control in order to follow a tight integration.

 

Nevertheless, the experience of both designing and developing the tools and using them in academic environments, showed us that analytics must be carefully and explicitly designed to be included in the tools during the tools design and development phases. In addition, experience shows that certain tools can take advantage of more advanced analytics, at a correspondingly higher cost, such as for example FormalZ and near-realtime analytics; while other tools such as the Quiz Repository and CodeDefenders already had access to their own analytics and/or results gathering (e.g. leaderboards, tool integrated feedback), which were more cost-effective than attempting to integrate an external analytics tool. In summary, the real goal is not to shoehorn the same analytics into all tools, but to provide the precise metrics and indicators required for the educational scenarios where the tools are used.

 

For example, the Quiz repository includes metrics and statistics about the usage of both questions and quizzes. Teachers can check statistics about each question, seeing how many times it has been used in a quiz and their overall correct reply rate. The system also collects and presents statistics about students, regarding their use of the tool (i.e quizzes answered separated by type) and % of correct answers. This information is presented to teachers and students. Students can only see their personal information, and the teachers have an overall view of the whole class, but can see the information of a single student as well.

 

 

Moreover, CodeDefenders provide a metrics’ dashboard that administrators can access so they can understand what is the impact of the challenges and puzzles that are proposed using the platform. Moreover, the challenges themselves also offered a score board during the gameplay that serve as a feedback for both instructors and students.

 

 

All this work has been reflected in publications of our results, published in several scientific venues and highlighting both in the aspects of analytics and personalized dashboards, and in the formal evaluation of serious games.

1. Personalized Dashboards and Unified Analytics

 

Description:

 

This category includes the publications related to improving analytics dashboards: enhancing their understanding for teachers, providing personalized dashboards (e.g. dashboards created for location-based serious games), and unified analytics.

It additionally includes related works of learning analytics applications, such as those carried out with the serious games Conectado or DownTown, or the learning analytics integration in the tool uAdventure to simplify creation of serious games.

 

Journal publications:

  1. Ana Rus Cano, Alvaro Garcia-Tejedor, Baltasar Fernández-Manjón (2018): Using Game Learning Analytics for Validating the Design of a Learning Game for Adults with Intellectual Disabilities. The British Journal of Educational Technology (in press) [Q1, 2017 Impact factor: 2.588 ISI Journal Citation Reports © Ranking: 2018:31/243 (Education & Educational Research)].

  2. Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2018): Conectado: A Serious Game to Raise Awareness of Bullying and Cyberbullying in High Schools. Educational Communications and Technology, Issue 3, 2018, pages 59 to 64 (original published in Chinese).

  3. Antonio Calvo-Morata, Cristina Alonso-Fernández, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Game Learning Analytics, Simplificando el Uso de Juegos Serios en la Clase. Versión Abierta Español Portugués de IEEE Revista Iberoamericana de Tecnologías del Aprendizaje (VAEP-RITA), Noviembre 2019. Volumen 7, Número 4 Pags.175-184.

  4. Antonio Calvo-Morata, Cristina Alonso-Fernández, Ivan Perez-Colado, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Improving Teacher Game Learning Analytics Dashboards through ad-hoc Development. Journal of Universal Computer Science, Vol. 25, No. 12, pp. 1507-1530. [JCR-SCIE 2018, IF 0.910 Computer Science – Theory & Methods].

  5. Antonio Calvo-Morata, Cristina Alonso-Fernández, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Game Learning Analytics, Facilitating the Use of Serious Games in the Class. IEEE Revista Iberoamericana de Tecnologías del Aprendizaje (RITA), November 2019. DOI: 10.1109/RITA.2019.2952296.

  6. Ana Rus Cano, Alvaro Garcia-Tejedor, Cristina Alonso-Fernández, Baltasar Fernández-Manjón (2019): Game Analytics Evidence-Based Evaluation of a Learning Game for Intellectual Disabled Users. IEEE Access doi:10.1109/ACCESS.2019.2938365 [JCR 2018, Impact Factor: 4.098, Q1, 23/155 Q1 in COMPUTER SCIENCE, INFORMATION SYSTEMS].

  7. Cristina Alonso-Fernández, Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Applications of data science to game learning analytics data: a systematic literature review. Computers & Education, Volume 141, November 2019, 103612 [2019 IF 5.296 JCR, Q1 in COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS]. DOI: 10.1016/j.compedu.2019.103612.

  8. Cristina Alonso-Fernández, Ana Rus Cano, Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Lessons learned applying learning analytics to assess serious games. Computers in Human Behavior, Volume 99, October 2019, Pages 301-309 [2019 IF 5.003, Q1 in PSYCHOLOGY, EXPERIMENTAL]. DOI: 10.1016/j.chb.2019.05.036.

Conference publications:

  1. Cristina Alonso-Fernández, Ivan Perez-Colado, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2018): Improving serious games analyzing learning analytics data: lessons learned. Games and Learning Alliance conference (GALA Conf), December 5-7, 2018, Palermo, Italy.

  2. Cristina Alonso-Fernández, Ana Rus Cano, Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2018): Applications of Learning Analytics to assess Serious Games (extended abstract). 2nd Annual Learning & Student Analytics Conference (LSAC), October 22-23, 2018, Amsterdam, The Netherlands.

  3. Ivan Perez-Colado, Dan C. Rotaru, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2018): Multi-level Game Learning Analytics for Serious Games. VS Games 2018. 10th International Conference on Virtual Worlds and Games for Serious Applications. 5. – 7. September, Würzburg, Germany.

  4. Antonio Calvo-Morata, Cristina Alonso-Fernández, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2018): Making understandable Game Learning Analytics for teachers. 17th International Conference on Web-based Learning (ICWL 2018), August 22nd – 24th, 2018, ChiangMai, Thailand.

  5. Víctor Manuel Pérez-Colado, Dan C. Rotaru, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2018): Learning analytics for location-based serious games. IEEE Global Engineering Education Conference (EDUCON), April 18-20, 2018, Santa Cruz de Tenerife, Canary Islands, Spain.

  6. Ivan Perez-Colado, Cristina Alonso-Fernández, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2018): Game Learning Analytics is not informagic!. IEEE Global Engineering Education Conference (EDUCON), April 18-20, 2018, Santa Cruz de Tenerife, Canary Islands, Spain.

  7. Antonio Calvo-Morata, Cristina Alonso-Fernández, Iván Martínez-Ortiz, Manuel Freire, Baltasar Fernández-Manjón (2019): Game Learning Analytics for Educators. IEEE Global Engineering Education Conference (EDUCON), 9–11 April, 2019, Dubai, UAE.

  8. Iván Martínez-Ortiz, Ivan Perez-Colado, Dan-Cristian Rotaru, Manuel Freire, Baltasar Fernández-Manjón (2019): From heterogeneous activities to unified analytics dashboards. IEEE Global Engineering Education Conference (EDUCON), 9–11 April, 2019, Dubai, UAE.

2. Serious games evaluation 

Description:

 

This category includes the publications related to the evaluation of serious games (usually carried out with pre-post tests) and the work done on this field to: improve such validation process of serious games, simplifying the experiments required to conduct such validations (including the tool developed SIMVA) and the later larger-scale deployment simplifying players’ automatic assessment.

 

Journal publications:

  1. Cristina Alonso-Fernández, Ivan Perez-Colado, Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2020): Aplicaciones de Simva para simplificar la validación y el despliegue de juegos serios. Versión Abierta Español Portugués de IEEE Revista Iberoamericana de Tecnologías del Aprendizaje (VAEP-RITA).

  2. Cristina Alonso-Fernández, Ivan Perez-Colado, Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2020): Applications of Simva to simplify serious games validation and deployment. IEEE Revista Iberoamericana de Tecnologías del Aprendizaje (RITA), Volume 15 , Issue 3. DOI: 10.1109/RITA.2020.3008117.

  3. Cristina Alonso-Fernández, Iván Martínez-Ortiz, Rafael Caballero, Manuel Freire, Baltasar Fernández-Manjón (2020): Predicting students’ knowledge after playing a serious game based on learning analytics data: A case study. Journal of Computer Assisted Learning, vol. 36, no. 3, pp. 350-358, June 2020. [JCR 2019, Impact Factor: 2.126, Q2, 79/263 in Education & Educational Research]. DOI: 10.1111/jcal.12405.

  4. Cristina Alonso-Fernández, Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2020): Evidence-based evaluation of a serious game to increase bullying awareness. Interactive Learning Environments, 2020. [JCR 2019, Impact Factor: 1.938, Q2, 96/263 in EDUCATION & EDUCATIONAL RESEARCH]. DOI: 10.1080/10494820.2020.1799031.

Conference publications:

 

  1. Dan-Cristian Rotaru, Sara García-Herranz, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón, Mª Carmen Díaz-Mardomingo (2018): Using Game Technology to Automatize Neuropsychological Tests and Research in Active Aging. GOODTECHS 2018 – 4th EAI International Conference on Smart Objects and Technologies for Social Good.

  2. Ivan Perez-Colado, Antonio Calvo-Morata, Cristina Alonso-Fernández, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Simva: Simplifying the scientific validation of serious games. 19th IEEE International Conference on Advanced Learning Technologies (ICALT), 15-18 July 2019, Maceió-AL, Brazil.

  3. Cristina Alonso-Fernández, Ivan Perez-Colado, Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Using Simva to evaluate serious games and collect game learning analytics data. Learning Analytics Summer Institute Spain (LASI-Spain), 27-28 June 2019, Vigo, Spain.

  4. Cristina Alonso-Fernández, Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Simplifying the validation and application of games with Simva. The 4th International Symposium on Emerging Technologies for Education In conjunction with ICWL 2019 (SETE 2019), 23-25 September, 2019, Magdeburg, Germany.

  5. Ivan Perez-Colado, Víctor Manuel Pérez-Colado, Iván Martínez-Ortiz, Manuel Freire, Baltasar Fernández-Manjón (2020): A scalable architecture for one-stop evaluation of serious games. Games and Learning Alliance conference, GaLA 2020 (9th edition).

  6. Ivan Perez-Colado, Víctor Manuel Pérez-Colado, Iván Martínez-Ortiz, Manuel Freire, Baltasar Fernández-Manjón (2020): Simplifiying Serious Games Authoring and Validation with uAdventure and SIMVA. 2020 IEEE 20th International Conference on Advanced Learning Technologies (ICALT) IEEE DOI 10.1109/ICALT49669.2020.00039.

  7. Ivan Perez-Colado, Iván Martínez-Ortiz, Manuel Freire, Wishnu Prasetya, Baltasar Fernández-Manjón (2020). Simplifying serious games and analytics deployment in a virtual campus using LTI and xAPI. X International Conference on VIRTUAL CAMPUS. Tetouan (Morocco), 3-5 December 2020..

Leave a Reply