Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
British Journal of Educational Technologydoi:10.1111/bjet.12940
Vol 51 No 4 2020 1019–1038
© 2020 British Educational Research Association
Evaluation of a learning analytics tool for supporting teachers in the creation and evaluation of accessible and quality open educational resources
Cecilia Avila , Silvia Baldiris , Ramon Fabregat and Sabine Graf
Cecilia Avila is a professor and researcher at the Fundación Universitaria Konrad Lorenz in Bogotá (Colombia). She has a PhD in Technology from the University of Girona. Her research interests are semantic web, web accessibility, usability, learning analytics and open educational resources. Silvia Baldiris is a full professor and researcher at the International University of La Rioja, Logroño, Spain. She has a PhD in Technology from the University of Girona. She is interested in the field of technology enhanced learning and on topics such as authoring tools, open educational resources, universal design for learning and web accessibility. Ramon Fabregat is co-director of the BCDS research group in the University of Girona and is an associate professor at the Institute of Informatics and Applications (IIiA). He has a PhD in computer science from the same university. His research interests are e-learning, adaptive systems, user modeling and inclusive learning. Sabine Graf is presently a full professor (tenured) at the School of Computing and Information Systems, Athabasca University, Canada. She has a PhD from Vienna University of Technology, Austria. Her research areas are: e-learning, personalization, learning analytics and academic analytics, user modeling and game-based learning. Address for correspondence: Cecilia Avila, Faculty of Mathematics and Engineering, Fundación Universitaria Konrad Lorenz, Bogotá, Colombia. Email: [email protected]
IntroductionThe UNESCO guidelines for Open Educational Resources (OERs) in higher education promote the investment in the systematic production, adaptation and use of OER that address students’ learning needs (Commonwealth of Learning, 2011). Thus, teachers are challenged with provid-ing OERs that meet characteristics such as web accessibility and quality as two key features of OERs, which make them suitable for all students (Almendro & Silveira, 2018; Chen, Sanderson,
AbstractThe learning analytics (LA) field seeks to analyze data about students’ interactions, and it has been applied in the development of tools for supporting both learning and teaching processes. Recent research has paid attention on how LA may benefit teachers in the creation of educational resources. However, most of the research on LA solutions is carried out to support teachers in analyzing students’ behavior data collected as they interact with virtual learning environments. While solutions to support teachers in different virtual learning and teaching contexts have become important, to date little research has been done on how LA solutions can help teachers to create and evaluate Open Educational Resources (OERs). This study aims at presenting the evaluation of a LA tool for supporting teachers in the creation and evaluation of accessible and quality OERs considering that both processes fall within the competences that teachers can acquire and strengthen by participating as authors (creation) and evaluators (evaluation) of OERs. The study was conducted with Colombian teachers and the results obtained highlight the positive effect the tool had on the teachers’ acquisition of the competences and the positive attitude they had toward using the tool.
© 2020 British Educational Research Association
1020 British Journal of Educational Technology Vol 51 No 4 2020
Kessel, & Królak, 2015; Marín, Orellana, & Peré, 2019; Moise, Vladoiu, & Constantinescu, 2014). Research on how to support teachers in this is limited. According to Iniesto, McAndrew, Minocha, and Coughlan (2016), “accessibility does not appear to have been considered in a consistent way when designing online learning resources”. Some studies have also stated that evaluating digital contents demands technical or specialized knowledge (Restrepo, Amado-Salvatierra, Otón, & Hilera, 2018; Vlachogiannis, Darzentas, & Koutsabasis, 2010). Moreover, authoring tools should provide feed-back about the learning contents that need to be improved at design time (Chen et al., 2015).
Solutions based on learning analytics (LA) have demonstrated to be suitable for supporting teachers in designing educational resources, a process in which decision making and feedback tools (eg, dashboards) are crucial for the improvement and evaluation of learning contents and activities (Albó, Barria-Pineda, Brusilovsky, & Hernández-Leo, 2019; Hernández-Leo, Martinez-Maldonado, Pardo, Muñoz-Cristóbal, & Rodríguez-Triana, 2019; Mangaroska & Giannakos, 2017). However, most of the LA tools focus on informing teachers about students’ interaction with learning activities or contents (Brooks, Erickson, Greer, & Gutwin, 2014; Fernandez-Delgado, Mucientes, Vazquez-Barreiros, & Lama, 2014; Leacock & Nesbit, 2007) and not on tracing the activities performed by teachers while creating OERs. (Bodily, Nyland, & Wiley, 2017) pointed out that LA may help identifying improvements for OER contents and continuous improvement drives high quality contents.
The purpose of this paper, therefore, is to investigate the use of LA as an opportunity to support teachers in learning how to create and evaluate OERs by tracing their activities in these processes, all the while keeping web accessibility and quality in mind. Because this entails a learning process, we consider the creation and evaluation processes to form part of a teacher’s digital competences. The study was conducted with 19 Colombian school and university teachers and it seeks at ana-lyzing: (1) whether the LA tool helps teachers acquire competences in the creation and evaluation of OERs, and (2) the teachers’ perceptions about the tool.
The paper is structured as follows. Second section describes some theoretical background and related work on web accessibility and quality in OERs, competences and, LA. Next, third section presents an overview of the LA tool. In fourth section, the methodology of the evaluation
Practitioner NotesWhat is already known about this topic
• Most of the Learning Analytics (LA) solutions focus on tracing students’ interactions with learning environments and few focus on learning design processes.
• Teachers require specialized knowledge and skills for participating in the creation and evaluation of OERs that meet with web accessibility and quality requirements.
What this paper adds
• The evaluation of a LA tool for supporting teachers in the creation and evaluation of OERs.
• Demonstrates that LA is a useful tool for teachers to know how well they create and evaluate accessible and quality contents of OERs.
Implications for practice and/or policy
• The current use of authoring tools for creating OERs is focusing on solutions that help teachers to work without the need of expert guidance.
• Tools based on LA may contribute to the continuously improvement of OERs.
© 2020 British Educational Research Association
Learning analytics for creating and evaluating OER 1021
of the LA tool is described. Fifth section presents the results and discussion, whereas sixth section presents the threats to validity for this study. Finally, seventh section presents the conclusions and future work.
Theoretical background and related workWeb accessibility and quality in OERsCreating OERs that address students’ diverse learning needs is a challenge for teachers who should consider characteristics such as web accessibility (to avoid barriers to content access), and quality (to make learning contents more appropriate for the learning context in which they are intended). On the one hand, quality is one of the most discussed features of OERs (Marín et al., 2019) and it is recognized to contribute to a better retrieving and sharing of OERs (Almendro & Silveira, 2018). Moreover, Moise et al. (2014) stated that the lack of quality assurance hinders the uptake and usage of OERs. On the other hand, accessible OERs contribute to flexible learning experiences for all (CAST, 2018) and this feature is also considered when evaluating quality in OERs. In their lit-erature review about standards and trends for OERs, Cueva & Rodríguez (2010) pointed out three aspects for future research in the field of OERs, namely accessibility standards, social components and semantic web. As for accessibility standards, they highlight the use of standards such as the Web Content Accessibility Guidelines (WCAG) specification.
Most of the research on OER web accessibility and quality has been focused on evaluating the contents after their creation (Kumar & Owston, 2016; Kurilovas, Bireniene, & Serikoviene, 2010; Mishra & Kanwar, 2015). However, evaluating an OER while it is still being created contributes to a better adoption once the resource is published in a virtual learning environment or in a reposi-tory (Marín et al., 2019).
CompetencesA competence is “the proven ability to use knowledge, skills and personal, social and/or methodological abilities, in work or study situations and in professional and personal development” (Ferguson et al., 2016). In the Information and Communication Technologies(ICT) context, digital competences are a “set of knowledge, skills, attitudes (thus including abilities, strategies, values and awareness) that are required when using ICT and digital media to perform tasks; solve problems; communicate; manage information; collaborate; create and share content; and build knowledge effectively, efficiently, appropri-ately, critically, creatively, autonomously, flexibly, ethically, reflectively for work, leisure, participation, learning, socialising, consuming, and empowerment” (Ferrari, 2012). In this research, the creation and evaluation of OERs are considered as part of teachers’ digital competences.
Learning analytics and OERsLA is a tool for measuring, collecting, analyzing and reporting data about learners and their con-text (Siemens et al., 2011). Adopting LA solutions to create and evaluate educational resources, often entails providing teachers with feedback about the learning context (eg, student interaction traces) and teachers can, in turn, use this feedback to improve the learning content (Haya, Daems, Malzahn, Castellanos, & Hopp, 2015; Holtham, Martin, Brown, Jawaheer, & Dove, 2012).
Some recent studies highlight the use of LA as a mean for managing learning designs (Er et al., 2019; Hernández-Leo et al., 2019; Holmes, Nguyen, Zhang, Mavrikis, & Rienties, 2019). For instance, Dyckhoff, Zielke, Bültmann, Chatti and Schroeder (2012) present a LA toolkit for teach-ers (eLAT), which is a conceptual toolkit designed to support teachers in the analysis of improve-ments for the learning scenarios by using graphical information regarding content usage, student behavior and assessment results. Lockyer, Heathcote, and Dawson (2013) analyze how analytics
© 2020 British Educational Research Association
1022 British Journal of Educational Technology Vol 51 No 4 2020
can facilitate pedagogical actions for learning designs. Another study highlights that LA may pro-vide insights on how to assess students based on the OERs’ impact (Prasad, Totaram, & Usagawa, 2016). However, Persico and Pozzi (2015) argue that most of the research in learning analyt-ics has concentrated attention to the contribution of learning analytics during the enactment of learning experiences and not in a design phase.
Some contributions reported in the literature about using analytics to support the design of edu-cational resources (involving processes such as the creation and evaluation) are: analytics can support teachers in decision making when completing a course design (Fritz & Whitmer, 2017); Cooper, Ferguson, and Wolff (2016) state that analytics is an approach that help identifying accessibility deficits in learning contents; reports based on analytics (usually presented in the form of dashboards) provide objective information that may be used by teachers in the design of OERs in order to improve learning contents (Bodily et al., 2017).
Therefore, we hypothesize LA may be used to provide teachers with feedback on how well they cre-ate and evaluate OERs, and how they can use the feedback to improve the learning content itself.
An overview of the ATCE toolThe Analytics Tool to trace the Creation and Evaluation of OERs (ATCE) was developed as a mod-ule in the ATutor LMS (ATutor, 2002). The ATCE tool was first introduced in the paper by Avila, Baldiris, Fabregat, & Graf (2017). However, in this section, some additional details are described.
ATCE differs from others that support teachers in OER creation and evaluation, in that it allows teachers to identify what they need to improve on their role as authors and evaluators of OERs. ATCE provides authors with visualizations to help them easily identify accessibility failures and quality items that need to be improved in each OER before its delivery in a real scenario with stu-dents, as well as feedback about their role as evaluators. Figure A1 (in supplementary material) depicts the flow diagram of the ATCE tool and its functionalities. The main functionalities of the ATCE tool are:
• Management options
The administrator of the tool assigns evaluators and experts and change the states of an OER. Possible states are: edit (by an author), evaluate (by an evaluator) and verify (by an expert). These states are used to ensure that an OER is not created/edited, evaluated or verified at the same time. Figure 1 depicts the management options.
Figure 1: ATCE—management options [Colour figure can be viewed at wileyonlinelibrary.com]
© 2020 British Educational Research Association
Learning analytics for creating and evaluating OER 1023
• Storing HTML elements
The ATCE tool stores each HTML element included in the web pages of an OER as a separate record in the database. The LMS did not include this functionality because it stored the complete HTML source code of each content page. Thus, the content editor of the LMS was modified so that it could store each HTML element in a separated record.
• Automatic accessibility evaluation
Once the HTML elements have been stored, the ATCE tool carries out an automatic accessibility evaluation. For this automatic evaluation, we integrated AChecker (2012), which is an automatic tool to check accessibility criteria in web contents and works in accordance with different acces-sibility standards, among which are the WCAG (W3C-WAI, 2018). We used the API provided by AChecker which facilitates its integration in web environments. The workflow of this API is depicted in Figure 2.
• Manual evaluation
Teachers as evaluators use the ATCE evaluation module to evaluate the web accessibility and quality of each OER. The web accessibility is evaluated for each HTML element in a web page by using the questions pre-sented in Table A1 and the quality is evaluated with the items presented in Table A2 (in supplementary material). Figure 3 depicts the options presented to the evaluator when answering one of the accessibility questions.
Figure 2: AChecker API—rest service [Colour figure can be viewed at wileyonlinelibrary.com]
Client(ATutor)
Server(AChecker)
HTTP URIrequest
REST responseXML format
XML to JSON
GET accessibility answer
Figure 3: ATCE—manual evaluation [Colour figure can be viewed at wileyonlinelibrary.com]
© 2020 British Educational Research Association
1024 British Journal of Educational Technology Vol 51 No 4 2020
• Dashboard
It provides teachers with feedback related to their role as both authors and evaluators of OERs. This infor-mation is coming from the creation and evaluation process. Figure 4 depicts the welcome interface of the dashboard (a) and the main view with information about the contents added to the OER (b). Figure 5 depicts the accessibility view with a general overview of the accessibility in the selected OER (a), the accessibility over the time (b), the percentage of accessibility reached in each principle (c) and the detail of accessibility failures
Figure 4: Dashboard—main interface [Colour figure can be viewed at wileyonlinelibrary.com]
Figure 5: Dashboard—accessibility view [Colour figure can be viewed at wileyonlinelibrary.com]
© 2020 British Educational Research Association
Learning analytics for creating and evaluating OER 1025
identified in one of the principles (d). Figure 6 depicts the quality view with the general (a) and the detailed (b) results of the quality for the selected OER. Finally, Figure 7 depicts the evaluator view with information of the level reached by the teacher in the role of evaluator and comments to improve in this role for both compe-tences accessibility evaluation (a) and quality evaluation (b). Table A3 (in supplementary material) presents a summary of the metrics and visualizations of the dashboard.
EvaluationMethodThis study was carried out as comparative experimental research using an experimental group within-subjects design, in which the same group of participants are measured in different points in time (Randolph, 2008). Thus, the purpose of this study is twofold:(1) to investigate whether
Figure 6: Dashboard—quality view [Colour figure can be viewed at wileyonlinelibrary.com]
Figure 7: Dashboard—evaluator view [Colour figure can be viewed at wileyonlinelibrary.com]
a
b
2019-01-07
© 2020 British Educational Research Association
1026 British Journal of Educational Technology Vol 51 No 4 2020
the ATCE tool benefits teachers in the acquisition of competences in the creation and evaluation of OERs and (2) to identify the perceptions teachers had with regard to the use of the ATCE tool. These objectives were addressed by answering the following research questions(RQ):
• RQ1: Does the use of the ATCE tool benefit teachers in the acquisition of competences in the creation and evaluation of OERs?
• RQ2: What are the perceptions teachers have about the use of the ATCE tool?
The research study was divided into two parts. To answer RQ1, an observational process was de-fined to evaluate the ATCE tool involving a training course, in which teachers learned how to create OERs in terms of web accessibility and quality. To answer RQ2, we analyzed the perceptions teachers had when using the ATCE tool.
MaterialsThe following materials, tools and systems were used for this study:
• Initial questionnaire: This questionnaire gathered data related to demographic information and the background teachers had in the use of ICT tools. The questionnaire consisted of four parts: (1) demographic information, (2) ICT experience and skills, (3) specific use of ICT and (4) ICT in the creation of educational resources as part of teaching practices. This questionnaire was created using Google forms.
• Training course about the creation of OERs: This virtual training course aimed at teaching teachers how to create OERs contemplating web accessibility and quality. The course included four units. Three units with theoretical foundations related to OERs (Unit 1: Inclusive Learning, Unit 2: Universal Design for Learning and Unit 3: Open Educational Resources). As part of Unit 3, teachers learned how to add an open license to their OERs in the ATutor LMS using a Creative Commons License. Unit 4 was about web accessibility and quality.
• ATutor LMS and the ATCE tool: ATutor LMS was the platform used to deliver the content of the training course on creating OERs and integrating the ATCE tool.
• Final questionnaire: The final questionnaire consists of two parts: questions from the LAAM (Learning Analytics Acceptance Model) instrument (Ali, Asadi, Gašević, Jovanović, & Hatala, 2013) and some open questions to collect the opinions the teachers had about the ATCE tool. The authors of LAAM suggest that the questions can be adapted according to the LA tool being assessed. Some open questions were also included in the questionnaire for gathering additional feedback about the functionalities of the ATCE tool such as the ATCE dashboard (comments about the dashboard), the ATCE web editor used for creating the web pages (comments about the web editor) and the ATCE evaluation module (comments about the evaluation module).
ParticipantsInitially, 35 Colombian university and school teachers volunteered to participate in this study, although only 19 teachers completed the study. The reason for this experimental mortality was that participants who dropped out of the study, stated that they did not have enough time to finish the activities in the research study.
According to the answers given to the initial questionnaire, all the participants have teaching experience in fields such as Arts, Accounting, Biology, English, Ethics, Informatics, Literature, Management, Math, Social sciences and Technology. In terms of ICT use and experience, the teachers had participated in courses related to: how to use a computer (74%), Microsoft Office tools (89%), management of virtual learning environments (32%), creation of web pages (58%), inclusion of ICT in education (21%) and web accessibility (16%). Besides that, all the participants
© 2020 British Educational Research Association
Learning analytics for creating and evaluating OER 1027
reported that they had participated in at least one activity related to the creation of educational resources: they had asked experts for help (42%), organized the content by themselves (74%), cre-ated the content by themselves (53%), shared the content with others (53%), used a methodol-ogy to create the educational resources (5%), used a license to share educational resources (5%), published resources on the Internet (11%), evaluated web accessibility (5%) or evaluated content quality (26%).
The teachers participated as authors and evaluators of OERs, while the researchers of this study participated as experts by carrying out additional evaluations of the OERs created and verifying the evaluations carried out by teachers.
ProcedureFigure 8 depicts the research design to answer RQ1 and RQ2. The evaluation process encompassed five phases in which the teachers participated as OER authors and evaluators. The teachers took part in a training course on the creation of OERs during phases 2 and 3. In all the phases, the par-ticipants used the web editor and evaluation module of ATCE. In phase 5, both authors and evalu-ators used the ATCE dashboard. For measurements 1 and 2, an expert used the ATCE verification module to verify how well the teachers had created and evaluated the OERs (in phases 4 and 5).
The activities carried out in the evaluation process are described as follows:
• Initial questionnaire: At the beginning of the evaluation process, teachers were asked to answer the initial questionnaire.
• Creation phase 1: Before starting the training course, each teacher in their role of author was asked to create an OER (OER1i, where i identifies the author) as a virtual course in the ATutor LMS using the ATCE web editor.
• Evaluation phase 1: Each teacher, in their role of evaluator, was asked to evaluate the web ac-cessibility and quality of an existing OER (evaluation_OER) entitled, The Kingdoms of Nature using the ATCE evaluation module.
• Training course part: The teachers were asked to read the first three units of the training course: (1) Unit 1: Inclusive Learning, (2) Unit 2: Universal Design for Learning and (3) Unit 3: Open Educational Resources.
• Creation phase 2: After reviewing the aforementioned units, each teacher in their role as author, was asked to create a new OER (OER2i, where i identifies the author) using the ATCE web editor.
• Evaluation phase 2: Each teacher, in their role as evaluator, was asked to once again evaluate the web accessibility and quality of the evaluation_OER. By this stage, the teachers possessed more knowledge on what criteria to consider when evaluating the web accessibility and quality of the OER.
• Training course part 2: The teachers were asked to read Unit 4: Web accessibility and quality.• Creation phase 3. After reading Unit 4, teachers were asked to develop some exercises focused
on creating accessible contents using the ATCE web editor. To do these practical exercises each teacher created another OER (OER3i).
• Creation phase 4: Each teacher, in their role as author, was asked to continue editing the OER2i created in creation phase 2.
• Evaluation phase 4: Each teacher, in their role as evaluator, was asked to once again evaluate the web accessibility and quality of the evaluation_OER.
• Measurement 1: An expert used the ATCE evaluation module to evaluate the OERs edited by the teachers in creation phase 3 and the ATCE verification module to verify the evaluations made by teachers in evaluation phase 4.
© 2020 British Educational Research Association
1028 British Journal of Educational Technology Vol 51 No 4 2020
Figu
re 8
: P
hase
s of
the
rese
arch
des
ign
[C
olou
r fig
ure
can
be v
iew
ed a
t wile
yonl
inel
ibra
ry.c
om]
© 2020 British Educational Research Association
Learning analytics for creating and evaluating OER 1029
• Creation phase 5: The teachers were introduced to the ATCE dashboard. After that, each teacher in their role as author, was asked to continue editing OER2i but this time using the ATCE web editor and the information provided through the author’s view of the ATCE dashboard.
• Evaluation phase 5: The teachers were asked to participate in a peer-review process in which each teacher assumed the role of evaluator and evaluated an OER created by another teacher (OER5ij, where i identifies the evaluator and j the OER’s author) using the ATCE evaluation module and the feedback provided in the evaluator’s view of the ATCE dashboard.
• Measurement 2: An expert evaluated the OERs edited by the teachers in creation phase 4 and also verified the evaluations made by the teachers in evaluation phase 4.
• Final questionnaire: The teachers were asked to answer the final questionnaire with questions based on the LAAM instrument and some open questions.
Data analysisThe methods used to analyze the data to answer the two research questions are described as follows:
RQ1: Does the use of the ATCE dashboard benefit teachers in the acquisition of competences in the creation and evaluation of OERs?
The dependent variables for RQ1 are:
• Accessibility (creation): This variable shows how well authors can create accessible OERs in terms of their competence level (from 0 to 100).
• Quality (creation): This variable shows how well authors can create quality OERs in terms of their competence level (from 0 to 100).
• Accessibility (evaluation): This variable shows how well evaluators can evaluate the web acces-sibility of OERs in terms of their competence level (from 0 to 100).
• Quality (evaluation): This variable shows how well evaluators can evaluate the quality of OERs in terms of their competence level (from 0 to 100).
To answer RQ1, the results obtained in measurements 1 and 2 were compared, considering the dependent variables. A t test was applied to compare the variables with a normal distribution and the Mann–Whitney U test for those with a non-normal distribution. The Excel tool was used to organize the data collected, the RStudio tool to carry out the statistical tests and the G*Power tool to calculate the effect size and power.
RQ2: What are the perceptions teachers have about the use of the ATCE tool?
To answer RQ2, the answers given by the teachers to the final questionnaire were analyzed. Mean and standard deviation were calculated for the answers given to the questions of the LAAM in-strument and a qualitative description was applied for the answers given to the open questions. The Excel tool was used to organize the data collected and to calculate the descriptive statistics.
Results and discussionOn the one hand, this section presents general results about the web accessibility and quality of the OERs created by teachers and, on the other hand, results from both RQ1 and RQ2.
Overview of web accessibility and qualityFigure 9 shows results of web accessibility of OERs in the second expert evaluation. The level of web accessibility by type of HTML element varies between 79% and 98%.
© 2020 British Educational Research Association
1030 British Journal of Educational Technology Vol 51 No 4 2020
Since web accessibility was addressed using the WCAG guidelines, the level of accessibility for each one of the three WCAG principles considered in the ATCE tool (Perceivable, Operable and Understandable) was also analyzed. The robust principle was not included here because criteria in this principle are related to the HTML source code of the web pages and the teachers do not work directly with the source code of the web pages. Figure 10 shows the level of web accessibility reached for each principle. These results revealed that the web accessibility of the OERs still need to be improved but the level obtained in each principle was good (higher than 75% in all three principles).
Figure 11 shows the results for the OERs’ level of quality. These results represent the rounded average value of stars obtained by each OER in all the quality items. According to these results, most OERs created in creation phase 4 reached between 4 and 5 stars. These results suggest that the level of quality in most of the OERs was good (3 to 5 stars) with and none of them obtaining 1 or 2 stars.
At the end of the evaluation process, the ideal situation would be that, in the last measurement, all the OERs reached 100% in the level of accessibility and 5 stars in the quality rating for all OERs. That said, the results for the accessibility by type of HTML element and by WCAG principle revealed a good level of web accessibility (higher than 75%) and, in terms of quality, the OERs obtained a good level with 4 and 5 stars for almost all of them.
RQ1: Teacher Competences in the Creation and Evaluation of OERsThe results obtained from the measurements 1 and 2 revealed that teachers’ competences in OER creation and evaluation improved. Data used for answering RQ1 come from measurements
Figure 9: Level of web accessibility by type of HTML element [Colour figure can be viewed at wileyonlinelibrary.com]
0102030405060708090
100
paragraph heading image table elements list video link
Accessible Not accessible
Figure 10: Level of web accessibility by WCAG principle [Colour figure can be viewed at wileyonlinelibrary.com]
© 2020 British Educational Research Association
Learning analytics for creating and evaluating OER 1031
carried out in the evaluation process before teachers used the ATCE dashboard (measurement 1) and after they used it (measurement 2). The results for RQ1 were calculated based on statistical tests conducted for each of the four dependent variables defined and are described as follows:
• Accessibility (creation): Data collected about how well authors can create accessible OERs in measurement 1 (ACM1) and measurement 2 (ACM2) was normally distributed according to the Shapiro–Wilk test with p > 0.05 (ACM1 p = 0.08746; ACM2 p = 0.569). The standard parametric t test was used to identify if there was any difference when teachers did not use the ATCE dashboard (ACM1) and when they did use it (ACM2). Results from the t test revealed that the level of web accessibility in ACM2 (M = 85.46; SD = 4.19) was significantly higher than in ACM1 (M = 65.98; SD = 4.19), with t(18) = −6.0638, p < 0.05. The effect size was large, Cohen’s d = 1.43, and the Power was 1.
• Quality (creation): Data collected for how well authors can create quality OERs in measure-ment 1 (QCM1) and measurement 2 (QCM2) was not normally distributed according to the Shapiro–Wilk test with p < 0.05 (QCM1 p = 0.003226; QCM2 p = 0.02163). The Wilcoxon signed-rank test was used to identify if there was any difference when teachers did not use the ATCE dashboard (QCM1) and when they did (QCM2). Results from the Wilcoxon test revealed a significant difference between QCM1 and QCM2: V = 0, p < 0.001. The sum of ranks assigned to the differences with positive sign 0, and the sum of ranks with negative sign was 210. The effect size was large, r = 3.65, and Power was 1.
• Accessibility (evaluation): Data collected for how well evaluators evaluated the web accessibility of the OERs in measurement 1 (AEM1) and measurement 2 (AEM2) was normally distributed according to the Shapiro–Wilk test with p > 0.05 (AEM1 p = 0.9362; AEM2 p = 0.2351). The standard parametric t test was used to identify if there was any difference when teachers did not use the ATCE dashboard (AEM1) and when they did use it (AEM2). Results from the t test revealed that the level of web accessibility in AEM2 (M = 79.99; SD = 10.16) was significantly higher than in AEM1 (M = 68.97; SD = 11.47), with t(18) = −5.1136, p < 0.05. The effect size was large, Cohen’s d = 1.11, and the Power was 1.
• Quality (evaluation): Data collected for how well evaluators evaluated the quality of the OERs in measurement 1 (QEM1) and measurement 2 (QEM2) was not normally distributed accord-ing to the Shapiro–Wilk test with p < 0.05 (QEM1 p = 0.001224; QEM2 p = 0.0002435). The Wilcoxon signed-rank test was used to identify if there was any difference when teachers did not use the ATCE dashboard (QEM1) and when they used it (QEM2). Results from the Wilcoxon test revealed a significant difference between QEM1 and QEM2: p < 0.001. The sum of ranks
Figure 11: Level of quality [Colour figure can be viewed at wileyonlinelibrary.com]
© 2020 British Educational Research Association
1032 British Journal of Educational Technology Vol 51 No 4 2020
assigned to the differences with positive sign 0 and the sum of ranks with negative sign was 153. The effect size was large, r = 1.74, and Power was 1.
Differences observed in the two measurements were positive and statistically significant. These results suggest that teachers might benefit from using the ATCE dashboard because their compe-tences improved after they used the feedback provided through the dashboard. These results are consistent with the findings of other studies that found feedback from dashboards guides users in their progress and performance in learning activities (Aljohani & Davis, 2013; Corrin & de Barba, 2014; Na & Tasir, 2017).
RQ2: Teacher Perceptions on the Use of the ATCE ToolThe results from the answers given by teachers (N = 19) for both parts of the final questionnaire (LAAM instrument and open questions), revealed the positive perceptions the teachers had to-ward the use of the ATCE tool.
Table 1 shows the descriptive statistics for the answers given to the LAAM instrument. The possi-ble answers for each question range from 1 to 5, where 1 indicates very low agreement with the statement and 5 indicates very high agreement. It is important to note that Q8 is a reverse ques-tion, which means that the question is a negatively worded question and the numerical scoring scale runs in the opposite way. For the Q8, the result obtained in the mean score (2.17) is equiva-lent to 3.84 so that it can be compared to the other questions.
The mean scores of all the questions in the questionnaire were greater than 3. Overall, this indicates that participants reported positive perceptions on the use of the tool. The highest score among the questions was 4.84 in Q1a. This result suggests that the information pre-sented in the accessibility section of the Author’s view of ATCE, seems to be the most remark-able feature of the tool which is related to the web accessibility of OERs. The lowest scores were in Q8 (3.84) and Q13 (3.84). A potential explanation for this result might be that some of the teachers may have used other similar tools which are more intuitive in feedback provisioning and less overwhelming. This might suggest that some teachers did not fully understand the visualizations and information presented in the dashboard. However, in the comments they gave to the open questions, the teachers seemed to be satisfied with the information presented in the dashboard.
Qualitative feedback on the use of the ATCE tool was gathered through the comments provided for each of the open questions included in the final questionnaire, main comments from teachers are described next and a complete description of this qualitative feedback is presented in the sup-plementary material:
• Comments about the dashboard: The dashboard allowed teachers to “… observe the evolution of the creation process”, and that “the analytics dashboard is an ideal and excellent tool with which to obtain data that allow us to improve the quality of our resources”.
• Comments about the evaluation module: The evaluation module served to help teachers im-prove as OER evaluators (eg, “at the beginning for me it was difficult to understand some ques-tions or categories presented in the evaluation module but little by little I was able to understand them”).
• Comments about the ATCE web editor: Teachers found the web editor to be an adequate sup-port tool to improve the OER’s web accessibility (eg, “an excellent tool that allows us to improve the webpages of the OERs and also to improve their accessibility”).
© 2020 British Educational Research Association
Learning analytics for creating and evaluating OER 1033Ta
ble
1:
Res
ults
for
the
LAA
M in
stru
men
t
Cat
egor
yQ
uest
ions
in th
e qu
esti
onna
ire
Mea
nSt
anda
rd d
evia
tion
Perc
eive
d va
lue
of v
isu
aliz
atio
ns
pres
ente
d in
th
e A
TCE
dash
boar
dQ
1a:
I u
se th
e in
form
atio
n p
rese
nte
d in
the
acce
ssib
ility
se
ctio
n o
f th
e A
uth
ors
view
of
ATC
E as
feed
back
to s
up-
port
my
proc
ess
as a
n a
uth
or o
f O
ERs
4.8
40
.37
Q1
b: I
use
the
info
rmat
ion
pre
sen
ted
in th
e qu
alit
y se
ctio
n
of th
e A
uth
or’s
vie
w o
f A
TCE
as fe
edba
ck to
su
ppor
t my
proc
ess
as a
n a
uth
or o
f O
ERs
4.5
80
.51
Q1
c:I u
se th
e in
form
atio
n p
rese
nte
d in
the
Eval
uat
or's
view
of
ATC
E as
feed
back
to s
upp
ort m
y pr
oces
s as
an
eva
lua-
tor
of O
ERs
4.5
80
.51
Perc
eive
d u
sefu
lnes
s (u
sage
bel
ief)
of
the
tool
fo
r im
prov
ing
the
OER
s' w
eb a
cces
sibi
lity
and
qual
ity
Q2
: ATC
E en
able
s m
e to
gai
n in
sigh
t in
to m
y pr
oces
s in
the
crea
tion
an
d ev
alu
atio
n o
f O
ERs
4.7
40
.45
Q3
: Th
e in
form
atio
n A
TCE
prov
ides
hel
ps m
e id
enti
fy w
hat
n
eeds
to b
e im
prov
ed in
my
OER
s4
.68
0.4
8
Q4
: ATC
E pr
ovid
es r
elev
ant i
nfo
rmat
ion
reg
ardi
ng
the
ac-
cess
ibili
ty a
nd
qual
ity
of m
y O
ERs
4.4
70
.51
Q5
: Th
e in
form
atio
n p
rovi
ded
by A
TCE
hel
ps m
e de
term
ine
how
to im
prov
e in
my
role
of
auth
or a
nd
eval
uat
or o
f O
ERs
4.6
30
.50
Q6
: ATC
E h
elps
me
iden
tify
the
mai
n a
cces
sibi
lity
and
qual
-it
y is
sues
in m
y O
ERs
4.6
80
.48
Perc
eive
d G
UI (
ease
of
use
) of
the
tool
Q7
: ATC
E’s
GU
I (G
raph
ical
Use
r In
terf
ace)
is in
tuit
ive
enou
gh4
.37
0.6
0
Q8
: ATC
E’s
GU
I is
over
burd
ened
wit
h in
form
atio
n. (
reve
rse
ques
tion
)2
.17
(3.8
4)
0.7
6
Q9
: ATC
E’s
GU
I has
a g
ood
desi
gn4
.37
0.5
0G
ener
al p
erce
ptio
n o
f th
e to
olQ
10
: All
in a
ll, I
fou
nd
ATC
E a
han
dy to
ol fo
r fe
edba
ck
prov
isio
n4
.63
0.5
0
Q1
1: I
wou
ld li
ke to
be
able
to u
se A
TCE
in m
y te
ach
ing
prac
tice
4.7
40
.56
Q1
2: A
TCE
prov
ides
me
wit
h m
ore
use
ful f
eedb
ack
than
ot
her
sim
ilar
tool
(s) t
hat
I h
ave
use
d/tr
ied
4.2
10
.71
Q1
3: A
TCE
is m
ore
intu
itiv
e th
an th
e ot
her
tool
s ca
pabl
e of
fe
edba
ck p
rovi
sion
that
I h
ave
use
d/tr
ied
3.8
40
.83
© 2020 British Educational Research Association
1034 British Journal of Educational Technology Vol 51 No 4 2020
Our observations are in agreement with prior research on teachers’ behavioral intentions when using a LA tool (Papamitsiou & Economides, 2015) because the information provided through the tool was perceived by the users as useful and easy-to-use and users declared their intention to use the tool in the future.
Threats to validityThere are some threats to validity of the evaluation conducted. While teachers learn about OERs and the use of the tool, they participated in different moments of creation and evaluation of OERs. Sometimes using the same OER. However, the measurements focused on what teachers did in the phase 4 (measurement 1) and the phase 5 (measurement 2). On the one hand, for the creation process in phase 4 teachers improved the OER created in phase 2 (OER2i), which was not modified in phase 3. The OER was improved based on what they practiced and learned in previous phases. They did the same in phase 5 but focusing on the feedback given through the dashboard, which was used by teachers only in phase 5. On the other hand, the fact that teachers evaluated the same OER in phase 1, phase 2 and phase 4 made teachers to be more focused on what they needed to improve each time they evaluated the OER and this might have influenced the results obtained in measurement 1 for the competency of evaluating accessibility and quality of OERs. However, in phase 5 teachers evaluated a different OER considering the recommendations given through the evaluator’s view of the dashboard.
In relation to the LAAM instrument, the instrument was adapted in this study by considering the information given to teachers through the ATCE tool about their role as authors and evaluators of OERs. Since the evaluation was conducted in a Spanish speaking country, questions were trans-lated into Spanish by a native speaker with English as a second language. Then, he and one more person discussed each translated question in order to verify their understandability. Then, another person translated the questions again into English to compare them with the original version of the instrument and make the corresponding revisions.
ConclusionsThis study sought to analyze whether the ATCE tool helped teachers in their acquisition of cre-ation and evaluation competences in the context of OERs, and to analyze the perceptions they had about using the ATCE tool. The general results from the evaluation of the ATCE tool suggest that the OERs created reached a good level of accessibility and quality. Moreover, the use of the ATCE dashboard furthered teachers in their acquisition of the competences and they also had positive perceptions about using the ATCE tool. The use of ATCE also allowed teachers to improve in their OER creation and evaluation competences. As authors, teachers were informed about how to im-prove the web accessibility and quality of their learning content and, as OER evaluators, they were informed about how to improve their evaluation in terms of web accessibility and quality.
As discussed by Navarrete and Luján-Mora (2017), people with disabilities are faced with barriers when they try to access OERs. They also state that one way to deal with this issue is to create authoring tools and learning contents that cope with such barriers. In line with this need, support tools like the one evaluated here can make teachers more aware of and can contribute to the creation of OERs with learning content that foster an equal access for all students.
Based on the study conducted, some benefits that can be highlighted in the use of LA in an author-ing tool to create and evaluate OERs are:
• Storing the detail of each element added or modified in the OERs allows to better analyze the traces of activities done by teachers when creating and evaluating OERs.
© 2020 British Educational Research Association
Learning analytics for creating and evaluating OER 1035
• Supporting the manual evaluation with an automatic evaluation of web accessibility and qual-ity drives better analyses of content elements that may present issues.
• Teachers are given with feedback about improvements needed for an OER (eg, in terms of web accessibility and quality) before it is delivered in real scenarios with students or in a public repository.
• The feedback is shown through visualizations in a dashboard that is updated as teachers add or modify contents.
Future work may involve an evaluation focused on teachers integrating their OERs into real sce-narios, so as to observe the effect providing students with accessible and quality OERs has on their learning process. Another future research direction is to foster collaborative work among teachers and stakeholders in the creation and evaluation of OERs and the trace actions coming from such collaborative work.
AcknowledgementsAuthors acknowledge to all teachers taking part in this process. Cecilia Avila and Ramon Fabregat acknowledge the support from the BCDS research group (ref. GRCT40), which is part of the DURSI consolidated research group Smart IT Engineering and Services (SITES) (ref. 2017 SGR-1551). Cecilia Avila had financial support under the grant BR2014 of the University of Girona. Silvia Baldiris acknowledges the support from the School of Engineering and Technology from the International University of La Rioja. Sabine Graf acknowledges the support from NSERC.
Statements on open data, ethics and conflict of interestThis research was approved by the Broadband Communications and Distributed Systems group from the University of Girona.
The involved participants were given with the informed consent for their participation in this study. Data can be accessed for research purposes upon request to the corresponding author.
No conflict of interest declared.
ReferencesAChecker. (2012). IDI web accessibility checker : Web accessibility checker. Retrieved from https://achec ker.ca/
check er/index.phpAlbó, L., Barria-Pineda, J., Brusilovsky, P., & Hernández-Leo, D. (2019). Concept-level design analytics for
blended courses. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11722 LNCS, 541–554. https://doi.org/10.1007/978-3-030-29736-7_40
Ali, L., Asadi, M., Gašević, D., Jovanović, J., & Hatala, M. (2013). Factors influencing beliefs for adoption of a learning analytics tool: An empirical study. Computers & Education, 62, 130–148. https://doi.org/10.1016/j.compe du.2012.10.023
Aljohani, N. R., & Davis, H. C. (2013). Learning analytics and formative assessment to provide immediate detailed feedback using a student centered mobile dashboard. In 2013 Seventh International Conference on Next Generation Mobile Apps, Services and Technologies, 262–267. Prague, Czech Republic. https://doi.org/10.1109/NGMAST.2013.54
Almendro, D., & Silveira, I. F. (2018). Quality assurance for open educational resources: The OER trust framework. International Journal of Learning, Teaching and Educational Research, 17(3), 1–14. https://doi.org/10.26803/ ijlter.17.3.1
ATutor. (2002). ATutor home. Retrieved from https://atutor.github.io/
© 2020 British Educational Research Association
1036 British Journal of Educational Technology Vol 51 No 4 2020
Avila, C., Baldiris, S., Fabregat, R., & Graf, S. (2017). ATCE: An analytics tool to trace the creation and eval-uation of inclusive and accessible open educational resources. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference (LAK ’17). Vancouver, British Columbia, Canada, 183–187. New York, NY: ACM. https://doi.org/10.1145/30273 85.3027413
Bodily, R., Nyland, R., & Wiley, D. (2017). The RISE framework: Using learning analytics to automati-cally identify open educational resources for continuous improvement. International Review of Research in Open and Distance Learning, 18(2). Retrieved from http://www.irrodl.org/index.php/irrod l/artic le/view/2952/4085
Brooks, C., Erickson, G., Greer, J., & Gutwin, C. (2014). Modelling and quantifying the behaviours of stu-dents in lecture capture environments. Computers & Education, 75, 282–292. https://doi.org/10.1016/j.compe du.2014.03.002
CAST. (2018). UDL on campus: Accessibility and open educational resources. Retrieved from http://udlon cam-pus.cast.org/page/media_oer
Chen, W., Sanderson, N. C., Kessel, S., & Królak, A. (2015). Heuristic evaluations of the accessibility of learning management systems (LMSs) as authoring tools for teachers. First Monday, 20(9). https://doi.org/10.5210/fm.v20i9.5430
Commonwealth of Learning. (2011). Guidelines for open educational resources (OER) in higher education. Commonwealth of Learning, Vancouver and UNESCO. Retrieved from http://unesd oc.unesco.org/image s/0021/00213 6/21360 5e.pdf
Cooper, M., Ferguson, R., & Wolff, A. (2016). What can analytics contribute to accessibility in e-learning systems and to disabled students’ learning? In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge Edinburgh, United Kingdom, 99–103. New York, NY: ACM. https://doi.org/10.1145/28838 51.2883946
Corrin, L., & de Barba, P. (2014). Exploring students’ interpretation of feedback delivered through learning analytics dashboards. In 31st Annual Conference of the Australian Society for Computers in Tertiary Education (ASCILITE 2014), ‘Rhetoric and Reality: Critical perspectives on educational technology’ (pp. 23–26). Dunedin, New Zealand.
Cueva, S., & Rodríguez, G. (2010). OERs, standards and trends. RUSC. Universities and Knowledge Society Journal // International Journal of Educational Technology in Higher Education, 7(1), 1–8. http://dx.doi.org/10.7238/rusc.v7i1.361
Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012). Design and implementation of a learning analytics toolkit for teachers. Educational Technology and Society, 15(3), 58–76.
Er, E., Gómez-Sánchez, E., Dimitriadis, Y., Bote-Lorenzo, M. L., Asensio-Pérez, J. I., & Álvarez-Álvarez, S. (2019). Aligning learning design and learning analytics through instructor involvement: A MOOC case study. Interactive Learning Environments, 27(5–6), 685–698. https://doi.org/10.1080/10494 820.2019.1610455
Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittelmeier, J., … Castaño, J. (2016). Research evidence on the use of learning analytics: Implications for education policy. Joint Research Centre Science for Policy Report; EUR 28294 EN. https://doi.org/10.2791/955210
Fernandez-Delgado, M., Mucientes, M., Vazquez-Barreiros, B., & Lama, M. (2014). Learning analytics for the prediction of the educational objectives achievement. In 2014 IEEE Frontiers in Education Conference 1-4. Madrid, Spain. https://doi.org/10.1109/FIE.2014.7044402
Ferrari, A. (2012). Digital competence in practice: An analysis of frameworks (No. JRC68116; JRC Technical Reports). Retrieved from http://ftp.jrc.es/EURdo c/JRC68 116.pdf
Fritz, J., & Whitmer, J. (2017). Learning analytics research for LMS course design: Two studies. Retrieved from https://er.educa use.edu/artic les/2017/2/learn ing-analy tics-resea rch-for-lms-course-design-two-studies
Haya, P. A., Daems, O., Malzahn, N., Castellanos, J., & Hoppe, H. U. (2015). Analysing content and patterns of interaction for improving the learning design of networked learning environments. British Journal of Educational Technology, 46(2), 300–316. https://doi.org/10.1111/bjet.12264
Hernández-Leo, D., Martinez-Maldonado, R., Pardo, A., Muñoz-Cristóbal, J. A., & Rodríguez-Triana, M. J. (2019). Analytics for learning design: A layered framework and tools. British Journal of Educational Technology, 50(1), 139–152. https://doi.org/10.1111/bjet.12645
© 2020 British Educational Research Association
Learning analytics for creating and evaluating OER 1037
Holmes, W., Nguyen, Q., Zhang, J., Mavrikis, M., & Rienties, B. (2019). Learning analytics for learning de-sign in online distance learning. Distance Education, 40(3), 309–329. https://doi.org/10.1080/01587 919.2019.1637716
Holtham, C., Martin, R., Brown, A., Jawaheer, G., & Dove, A. (2012). Evaluating engagement to address undergraduate first year transition: A case study. CSEDU 2012—Proceedings of the 4th International Conference on Computer Supported Education, 1, 223–228. Retrieved from http://www.scopus.com/inwar d/record.url?eid=2-s2.0-84864 87134 6&partn erID=tZOtx3y1
Iniesto, F., McAndrew, P., Minocha, S., & Coughlan, T. (2016). Accessibility of MOOCs: Understanding the provider perspective. Journal of Interactive Media in Education, 2016(1), 1-10. https://doi.org/10.5334/jime.430
Kumar, K. L., & Owston, R. (2016). Evaluating e-learning accessibility by automated and student-centered methods. Educational Technology Research and Development, 64(2), 263–283. https://doi.org/10.1007/s11423-015-9413-6
Kurilovas, E., Bireniene, V., & Serikoviene, S. (2010). Methodology for evaluating quality and reusability of learning objects. Electronic Journal of E-Learning, 9(1), 39–51.
Leacock, T. L., & Nesbit, J. C. (2007). A framework for evaluating the quality of multimedia learning re-sources. Journal of Educational Technology & Society, 10(2), 44–59.
Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459. https://doi.org/10.1177/00027 64213 479367
Mangaroska, K., & Giannakos, M. (2017). Learning analytics for learning design: Towards evidence-driven decisions to enhance learning. In É. Lavoué, H. Drachsler, K. Verbert, J. Broisin, & M. Pérez-Sanagustín (Eds.), Data driven approaches in digital education, 428–433. Tallinn, Estonia: Springer International Publishing.
Marín, V. I., Orellana, M. L., & Peré, N. (2019). Open educational resources for research training: Quality assurance through a collaborative evaluation. Research in Learning Technology, 27, 1-16. https://doi.org/10.25304/ rlt.v27.2271
Mishra, S., & Kanwar, A. (2015). Quality assurance for open educational resources: What’s the difference? (Pre-print).
Moise, G., Vladoiu, M., & Constantinescu, Z. (2014). MASECO: A multi-agent system for evaluation and classification of oers and ocw based on quality criteria. Studies in Computational Intelligence, 528, 185–227. https://doi.org/10.1007/978-3-642-41965-2_7
Na, K. S., & Tasir, Z. (2017). A systematic review of learning analytics intervention contributing to student success in online learning. In 2017 International Conference on Learning and Teaching in Computing and Engineering (LaTICE) (62–68). Hong Kong, China. https://doi.org/10.1109/LaTiCE.2017.18
Navarrete, R., & Luján-Mora, S. (2017). Bridging the accessibility gap in open educational resources. Universal Access in the Information Society, 17(4), 755–774. https://doi.org/10.1007/s10209-017-0529-9
Papamitsiou, Z., & Economides, A. A. (2015). Temporal learning analytics visualizations for increasing awareness during assessment. International Journal of Educational Technology in Higher Education, 12(3), 129–147. https://doi.org/10.7238/rusc.v12i3.2519.
Persico, D., & Pozzi, F. (2015). Informing learning design with learning analytics to improve teacher inquiry. British Journal of Educational Technology, 46(2), 230–248. https://doi.org/10.1111/bjet.12207
Prasad, D., Totaram, R., & Usagawa, T. (2016). Development of open textbooks learning analytics sys-tem. The International Review of Research in Open and Distributed Learning, 17(5), 215-234. https://doi.org/10.19173/ irrodl.v17i5.2541
Randolph, J. (2008). Multidisciplinary methods in educational technology research and development. HAMK, University of Applied Sciences. Retrieved from http://justu srand olph.net/artic les/multi disci plina ry_metho ds.pdf
Restrepo, F., Amado-Salvatierra, H., Otón, S., & Hilera, J. (2018). Training engineering educators on ac-cessible and inclusive learning design. International Journal of Engineering Education, 34(5), 1538–1548.
Siemens, G., Gasevic, D., Haythornthwaite, C., Dawson, S., Shum, S. B., & Ferguson, R. (2011). Open learn-ing analytics: An integrated & modularized platform Proposal to design, implement and evaluate an open platform
© 2020 British Educational Research Association
1038 British Journal of Educational Technology Vol 51 No 4 2020
to integrate heterogeneous learning analytics techniques. SoLAR (Society for Learning Analytics Research). https://solar esear ch.org/wp-conte nt/uploa ds/2011/12/OpenL earni ngAna lytics.pdf
Vlachogiannis, E., Darzentas, J. S., & Koutsabasis, P. (2010). Beyond specifications: Towards a practical methodology for evaluating web accessibility. Journal of Usability Studies, 5(4), 157–171.
W3C-WAI. (2018). Web content accessibility guidelines (WCAG) overview. Retrieved from https://www.w3.org/WAI/stand ards-guide lines/ wcag/
Supporting Information
Additional supporting information may be found online in the Supporting Information section at the end of the article.