Compendium of the assignments on Quality Criteria in Blended Learning

Background and objectives

In the framework of the MOOC “Quality in Digital Learning” participants formed working groups. The groups got the task to develop a set of 3 to max. 10 core quality criteria and a description on how the criteria might be applied in an overall quality assurance scheme such as the Open ECB-Check.

Table of Contents

1. Blended Learning Design and Methodology........................................................31.A Proposed definition of blended learning..........................................................................................31.B Learning design and methodology: brief literature review......................................................31.C Criteria for improvement of quality in e-learning........................................................................41.D Concluding remarks: focus on motivations.....................................................................................7References..............................................................................................................................................................7

2. Few criteria to evaluate blended learning quality................................................82.A Blended learning – a proven and recognized tool........................................................................82.B. Review of chosen ECB-Check criteria – Improvement suggestions.....................................8References:..........................................................................................................................................................15

3. Quality of the digital blended learning training programs...............................16Summary and Objectives of the assignment.........................................................................................163.A Definition of the Task and of the Instructions for the Assignment.....................................163.B Criteria and Evidence..............................................................................................................................17CRITERIA TO ADD TO ECB CHECK related community building in online trainings/communities for subjects on peace and security / human rights..........................23References........................................................................................................................................................... 24

4. Quality Assessment criteria for Blended Learning..............................................254.A. The Task......................................................................................................................................................254.B. Chosen topic “QA criteria for Blended Learning”......................................................................254.C. How the criteria might be applied in an overall quality assurance scheme such as the Open ECB-Check...............................................................................................................................................33References........................................................................................................................................................... 35

5. Application and Update of existing ECB-Check quality criteria...........................36Introduction........................................................................................................................................................36Step 1: Core Quality Criteria (and sub-criteria from the ECB-Check).......................................37Step 2: Assessment of the Quality in Digital Learning MOOC by applying the above core quality criteria................................................................................................................................................... 38Step 3: Additional descriptors concerning our chosen core quality criteria.........................39References: ?.......................................................................................................................................................40

6 Quality Criteria for Social Media in Blended Learning.........................................41Introduction........................................................................................................................................................41Suggested Criteria............................................................................................................................................41References: ?.......................................................................................................................................................43

List of Authors

1. Fatma Ibnouf, Marianela Salas, Mevognon Joseph, Teklit Gebregiorgis ,Toni García

2. Gláucia Queiroz , Eleonora Deschamps3. Anton (Sippyan), Hedda Vaassen, Saad Sulttan Hussein4. Jan, Maggy Beukes-Amiss, Albrecht Tiedemann, Obed Sindy, Mel Tan, Ana

Riza Mendoza, Katrin Backhaus, Andrew Mina, Bahran Atabeyli, Olimpius Istrate, Anton, Nigel, Gino Schiavone, Katharina Rohde, Nelly Sargsyan, Tsetseg-Ulzii, Fabrizio Mellado, Alkautsar AL, Carlos Würschmidt,

5. Uranchimeg Tudevdagva, Kerynn Dahl, Angela Falkensteiner and Eleonora Deschamps

6. Mohamed Moustafa, Hilkka, Shikongo Nathanael, Carlos Würschmidt

2

1. Blended Learning Design and Methodology*

Prepared by: Fatma Ibnouf, Marianela Salas, Mevognon Joseph, Teklit Gebregiorgis and Toni García

Short summaryThis paper focuses on core quality criteria to assess the effectiveness of b-learning initiatives. It concentrates on the e-learning portion of b-learning activities, and proposes an open check-list of simple and measurable criteria and guidelines, that can be applied before, during, and after the delivery of the given learning session.

1.A Proposed definition of blended learning

Blended learning is an approach of facilitation of a course, module or programme, in which from 20 per cent to 70 per cent of the content is facilitated face to face, with learners away from their homes in a ‘facilitation room’, and the remaining part of the course is facilitated at a distance, with learners having control over time, place, pace and path of learning, and content presented in print or digital format.

1.B Learning design and methodology: brief literature review

Kanuca, H. (2006) mentions early literature on the use of instructional design for eLearning. Such literature maintains that eLearning’s success is largely due to the link that programmes of instructional design have made between the design of learning materials embedded in learning theory and the effective selection and use of technology.

As mentioned by Kanuca (2006), work by Bates (2005) in the area of eLearning concludes that, without a team of instructional design experts, facilitation of effective eLearning is highly unlikely. For example, given the wide variety of Internet communication technologies and social software available (e.g. computer conferencing, podcasting, blogs, learning management systems, audio/video technologies, email, instant messaging, social bookmarking, peer-to-peer networks, non-immersive virtual reality, etc.), most academics will need to consult with instructional designers to ensure that the technologies they choose and use will teach the concepts effectively and meet their students’ needs.

* (*) Main inspiration for this writing comes from the ideas and work done in “Instructional Design and eLearning: A Discussion of Pedagogical Content Knowledge as a Missing Construct available”, by Heather Kanuka, to whome we express our appreciation, recognition and respect.

3

Many problems and concerns that have been identified in the literature related to eLearning— such as, for example, low rates of participation, learner resistance, high non-completion rates, poor learner performance (Bates, 2005; Kanuka et al., 2006)—can be addressed by working with a team of instructional design experts.

Research by Twigg (2003) provides additional examples of the benefits of using effective technology applications and effective instructional design. While the use of instructional designers with expertise in pedagogical strategies and technology is becoming widespread within institutions of higher education, the conventional wisdom suggests that the content-knowledge encompasses the ‘structure of knowledge’: the theories, principles, and concepts of a particular discipline. Hence, the educator (e.g., teacher or instructional designer) must identify the ways in which each unique body of knowledge (or discipline) should be structured so that it can be more readily understood by the learners.

Finally, Kanuca (2006) highlights that the content-knowledge that deals with the learning designs is of crucial importance; this includes the most useful forms of representing and communicating content and how students best learn the specific concepts and topics of a subject.

In conclusion, this literature review suggests that if instructional designers are to be effective, they must struggle with issues of both content and pedagogy. This means that instructional designers need to develop a repertoire of teaching strategies that reflects the uniqueness of each disciplinary culture.

1.C Criteria for improvement of quality in e-learning

The proposed criteria for improving quality in e-learning include, though not being limited to:

Design: set a “SMART” e-learning plan, by identifying and analyzing learners’ needs and learning content. In particular, analyzing whether or not the e-learning program is designed for a professional or an educational setting. In addition, clear objectives, attractive instructions and tools, and the criteria for evaluation need to be considered to ensure all learn and achieve at the highest levels possible.

Content: provide content and tools that are as relevant and as useful as possible to your target audiences. Interactive content, in particular, to achieve the desired learning objectives.  Course materials should be made easily accessible to all learners. There is a requirement in digital learning that the content reflect the requisite standards … this may require that the program needs to be reviewed frequently.

Methods:  successful eLearning programs depend on a needs assessment approach. To target audiences and their needs, both synchronous and asynchronous communication tools can be used. You will be required to

4

develop and deliver an actual lesson using an e-learning development tool (software and technical tools).

Target audience: What do they want?  What are their specific learning needs/gaps?  Working under these guidelines will help trainers better define a content strategy and prioritize content to be brought to the forefront.

Motivation: motivation may be enhanced by being given the opportunity to participate effectively. Building e-learners participation is not an easy task; it can be achieved, for instance, by attracting them through continuous encouragement, by broadening ownership of the program, sharing experiences, sharing thoughts, views and opinions, etc. Continual feedback regarding the progress of learners creates, in our view, scope for open space in digital learning (see next criterion).

Open space: this is the place where e-learners can engage in open discussions, and share ideas or topics of their choice with e-course facilitators and experts. This space is not necessarily connected to the course contents. The aim of the open space is ultimately to open up constructive debates and to motivate e-learners for more participation.

Evaluation: e-learning evaluation should not be a one-time event. It should occur at all times throughout the e-learning program. We can evaluate, for example, e-learners’ reactions to the program; continuous feedbacks; measure what they have learned through quizzes and assignments, as well as through final assessment forms, to determine whether or not the learning objectives and goals were met. Evaluation includes follow-up actions to assess the extent to which e-learners have been able to take action in integrating course contents in their work, the possible impact of these actions, and the challenges met. This can be done through on-line questionnaires, which can be sent to all participants who attended the course. It would be better to send it after six months of the end of the course.

Quality assurance: quality assurance in digital learning is an ongoing process, not a final destination. Trainers need to take a systemic approach to quality assurance, such as, for instance: before starting the learning activity, needs assessments can be conducted, including budget and time framework to better plan e-Learning. The formative evaluation can be done during the e-Learning to measure progress according to plan for continuous improvements, and the summative evaluation to determine the e-Learning outcomes.

5

VIEW THESE CRITERIA THROUGH THIS CHART.

1.D Concluding remarks: focus on motivations

Where and when participants learn it is mostly up to them. A regular interest in the course it therefore not to be taken for granted. Instead, e-learning requires a high degree of self motivation to make computer learning an effective tool. Some participants can’t deal with this degree of responsibility and drop out of e-learning course.

It is the task of tutors to keep up the motivation for the course, the learning content and the learning as a group. It helps if you “know” your participants, i.e., find out when and where they study, what degree of pressure they undertake and what kind of motivation enhancement they may need.

A learning unit is to be able to stand in its own right and constitute a complete and intelligible statement without the need for additional information. The course could be established indicating the internal connections between individual learning units. All units must use a uniformed structure to facilitate the orientation of learners. This includes:

Constantly writing simple and coherent texts, and using simple words. Always keeping contact with the participants, by asking them how they

are doing with the content, but also how they are doing personally. Sharing with the participants your own personality. Giving the participants the certainty that you and them are on this project

together and constantly co-operating.

Being ready to support the participants’ understanding with advice is also important. Trying to build on your personal experience and help the learning process along with a great understanding of the matter at hand. Advice concerning the learning process itself such as learning strategies are also very helpful and can greatly facilitate e-learning.

Finally, making sure that the participants talk to each other is key. The difficulties that arise for the individual learners during the course are usually very similar, so they can be brought up with a view to detecting similarities in challenges and relevant solutions. This shall also help to foster contacts within the group of learners.

References

Kanuca, H (2006) Instructional Design and eLearning: A Discussion of Pedagogical Content Knowledge as a Missing Construct. E-Journal of Instructional Science and Technology, v9 n2 Sep 2006.

2. Few criteria to evaluate blended learning qualityPrepared by: Gláucia Queiroz & Eleonora Deschamps

Short summary: This paper focuses in three core criteria: Learning Objectives, Learning design and Methodology, and Media Design.

2.A Blended learning – a proven and recognized tool

Blended learning has proved its potential to support and enhance both the effectiveness and efficiency of a meaningful learning experience (Garrison & Kanuka, 2004; Bremer, 2010). Some researchers such as Graham (2006) affirmed that blended learning approaches increase the level of active learning strategies, peer-to-peer learning strategies, and learner centred strategies. Blended learning assembles several event-based activities such as face-to-face classrooms, e-learning, and self-paced learning. Singh (2003) defined blended learning as a mix of traditional instructor- led training, synchronous online conferencing or training, asynchronous self-paced study, and structured on-the-job training.

Worldwide, several institutions such as FAO and ECB-Check are currently interested in the improvement of e-learning quality. Usually, each of these institutions has listed the criteria to check the quality of e-learning programmes. Our group discussed and agreed on three core criteria to be commented: Learning Objectives, Learning design and Methodology, and Media Design.

2.B. Review of chosen ECB-Check criteria – Improvement suggestions

In this essay, we chose to present criteria, comments and suggestions in form of a table. In the first column are the criteria suggested by ECB-check to evaluate each of these three aspects. In the second column, we present comments on the “description” and / or “Guidance on how to evaluate” related to the aspect provided by ECB-Check. Most of the comments are based on the document “Open ECB Check Quality Criteria for Programmes” (2012). In the case that the comment is based on other document/article, the reference can be found immediately before/after the idea or citation. In the third column we suggest a criterion that could be added in order to improve the ECB-Check criteria. 

8

ECB-Check‘s criterion:Learning Objectives

Comments 

Suggestion of criterion to be added

A.1.1 All information potential learners need to orient their decision about taking the course is available.

A.1.2 Learning objectives state clearly the expected improved competency. All learning objectives are measurable (e.g. stated according to the SMART criteria).

A.1.3 The methodological approach is clearly explained. It presents how the methods/approaches selected lead to the achievement of the learning objectives. Learners are able to understand the added value of technology enhanced learning.

A.1.4 Information on contact persons is available, mentioning name, area of responsibility, and contact details.

Learning objectives defines the desired measurable learning objectives students will achieve upon course completion. Clearly informed objectives facilitate decision making. For having a learner centered learning methodology approach, blended learning must support and empower participants to define part of their own learning objectives and learning pace. Students need to know what they can get from the course (Tucker, n.d.). Therefore, they need flawless information and guidance about the course program goals, methodological approach, target group, and time table. Having all needed information, a participant can check, for instance, whether all the pre-conditions for an efficient learning are met and to develop his/her study plan. On the other hand, alerted Bremer (2010), only the client knows his/her personal objectives, demands and current situation. Therefore, it is recommendable to involve potential users in the early phase of determining learning objectives of a programme. According to Ghirardini (2011), in an appropriate designed blended-learning course, previously determined learning objectives must be reflected in the programme topics and tasks as well as in the sequencing of topics, in the choice of learning strategies and delivery formats. Bonk & Graham (2012), however, alerted that “few things may be best taught in a traditional classroom. Therefore, it is crucial to define the learning objectives as a whole, and at the same time, outline which objectives can be achieve during the distant learning phases in which during the F2F part. 

For understanding well the learning objectives, learners are able to evaluate their own shortfalls and search for a way to overcome them in time in order to obtain plenty benefits from the programme.

ECB-Check‘s criterion:Learning Objectives

Comments

A.1.1 All information potential learners need to orient their decision about taking the course is available.

A.1.2 Learning objectives state clearly the expected improved competency. All learning objectives are measurable (e.g. stated according to the SMART criteria).

A.1.3 The methodological approach is clearly explained. It presents how the methods/approaches selected lead to the achievement of the learning objectives. Learners are able to understand the added value of technology enhanced learning.

A.1.4 Information on contact persons is available, mentioning name, area of responsibility, and contact details.

Learning objectives defines the desired measurable learning objectives students will achieve upon course completion. Clearly informed objectives facilitate decision making. For having a learner centered learning methodology approach, blended learning must support and empower participants to define part of their own learning objectives and learning pace. Students need to know what they can get from the course (Tucker, n.d.). Therefore, they need flawless information and guidance about the course program goals, methodological approach, target group, and time table. Having all needed information, a participant can check, for instance, whether all the pre-conditions for an efficient learning are met and to develop his/her study plan. On the other hand, alerted Bremer (2010), only the client knows his/her personal objectives, demands and current situation. Therefore, it is recommendable to involve potential users in the early phase of determining learning objectives of a programme. According to Ghirardini (2011), in an appropriate designed blended-learning course, previously determined learning objectives must be reflected in the programme topics and tasks as well as in the sequencing of topics, in the choice of learning strategies and delivery formats. Bonk & Graham (2012), however, alerted that “few things may be best taught in a traditional classroom. Therefore, it is crucial to define the learning objectives as a whole, and at the same time, outline which objectives can be achieve during the distant learning phases in which during the F2F part. 

Suggestion of criterion to be added

For understanding well the learning objectives, learners are able to evaluate their own shortfalls and search for a way to overcome them in time in order to obtain plenty benefits from the programme.

10

ECB-Check‘s criterion:Learning Design and Methodology

Comments Suggestion of criterion to be added

 D.1.1 The blending of learning methods (online, face-to-face, self-learning, tutor-facilitated, asynchronous, synchronous) is adequate and meets the needs of the learners.

D.1.2 A learner centred learning design facilitates the development of the desired skills and expected competencies described in the learning objectives.

D.1.3 Social and collaborative activities are included in the programme methodologies and contribute to the achievement of the learning objectives.

D.1.4 The learning methodologies offer opportunities for learners to determine their learning pace.

D.1.5 Through the learning activities learners are encouraged to consider and use systems thinking and to

The learning specifications processes are a crucial guide for both participants and tutors. A lesson to be never forgotten is that the blended learning methods must be adequately aligned with the learning needs of the specific target group. Regarding Learning Design and Methodology, ECB-Check’s determine several criteria. First, the blending of learning methods (online, face-to-face, self-learning, tutor facilitation, asynchronous and synchronous activities) must be adequate and meets the needs of the learners. The utilization of blended approaches must be explained and incorporated into the programme methodological approach. The blended learning design must facilitate the development of the desired skills and expected competencies described in the learning objectives. Training settings/environment must focus on interactive strategies. The design ought to include social and collaborative activities that contribute to the achievement of the learning objectives. Ehlers (2004) advised that a platform should provide the possibility of synchronous communication such as chat and video conferencing. The content of the programme must be progressively organized in a logical sequence from the simpler to the most complex concepts. However, the programme must be presented in a flexible manner, allowing the participants to determine self-directed learning paths and pace, to analyse their learning progress, and to identify their learning gaps. Bremer (2010) recommended asking potential users to test e-learning modules in advance. Learning times and give feedback on navigation, design, and ergonomics can be measured and evaluated. The results of the evaluation generally go back into the conception phase and implementation, enabling necessary changes before the programme is delivered to regular users.The utilization of blended approaches ought to take into account the learners characteristics. Blended learning design must consider professional and work context, previous experience or prior learning of the participants. Lim & Morris

Level of demand and time available to the study are in balance.

When necessary, learners get from tutor extra guidance to develop an increase his/her efficient learning strategies and tempo of studying.

When necessary, learners become extra information / exercise to develop their critical thinking capacity. (Although this capacity is indispensable for an efficient learning, but it not always assured   given, for instance, cultural backgrounds)

11

view issues from different perspectives.

D.1.6 The content of the programme is organized in a logical sequence from the simpler to the most complex concepts. The modules/ lessons/ units build progressively on each other. 

(2009) asserted that age, prior experiences with distance learning opportunities, preference in delivery format, and average study time have a strong influence on the learning outcomes among participants of blended learning programmes. Such aspects must be considered when reflecting on Learning Design and Methodology for the development of a blended learning programme.Accessibility and usability are the key words concerning Media Design. The expected standards of both must be met to guarantee good quality in blended learning. Accessibility means the degree to which product (environment, service, device, etc.) is available to as many people as possible. Usability focuses on the easiness of the learning environment and its components, including the way materials and content are presented. When navigating through the mandatory learning materials, the participant must easily get information about his/her progress and position in relation to the overall content. Besides, screens, table of content, and learning materials, including additional resources must be printable. Besides of enabling participants to save interesting parts of the programme material, printable material plays an important role in cases with limited or no connectivity. Participants can use this material during the times the connectivity is low or absent and keep themselves in time with the course (Peres & Pimenta, 2009).Environment is also a key word when talking about Media Design. Ehlers (2004) adverted that   “a learning process is not something that is delivered to a learner by an e-learning-provider but rather constitutes a process of co-production between the learner and the learning-environment”. This assertion is valid for both face-to-face (F2F) and distance learning phases.

12

ECB-Check‘s criterion:E Media Design

Comments Suggestion of criterion to be added

 E.1 Accessibility standards have been applied.

E.2 Usability standards are met.

E.3 The navigation (through the mandatory learning materials) allows learners to know about their progress and position in relation to the overall content

E.4 Screens, table of content, and learning materials, including additional resources are printable. 

It is necessary to assure that all components of the blended learning programme are appropriately designed and developed. Besides, participants’ experiences of the on-line part and their experience of face-to-face phases must be so well integrated that each participant experiences the programme as a unit, customized to his/her own needs.Efficient e-learning projects always demand a good design and planning. For combining face-to-face instruction with technology-mediated learning, blended learning programmes have an even bigger challenge: both components must be efficiently designed, flow into each other smoothly and together meet the learning needs of the participants. Graham (2006) explained that since live interaction plays an important role in learning processes, many participants have a preference for the live (or F2F) components of a blended experience. It is recommendable to assure that participants’ experiences of the on-line part of their course are associated with their experience of the course as a whole. When working by themselves in the distance learning phases, participants have to feel save navigating in the virtual environment. Therefore, technical issues must be adequately addressed in advance, when creating the learning environment and the tools to deliver the learning programme (Singh, 2003). Ideally, all expected technical standards must be met. For instance, suitable learning management system and learning content management system must be used. Singh (2003) affirmed that there are eight categories (Octagonal Framework) of interrelated and interdependent issues that need to be addressed in order to create meaningfully distributed learning environments in blended learning. These categories are institutional, pedagogical, technological, interface design, valuation, management, resource support, and ethical.Ghirardini (2011), analysing the cost-benefits of blended learning, highlighted that the design and development of structured materials of blended-learning must be self-contained and able to be used multiple times without making enduring

Programme material can be customized to meet the local audience needs involving culturally relevant features.

13

adjustments. “Well-developed e-learning courses can be delivered many times to different learners using the same materials”. On the other hand, Graham (2006) alerted to the role of cultural aspects involved in learning processes. “One strength of e-learning is the ability to rapidly distribute uniform learning materials. Yet, there is often a need for customizing the materials to the local audience to make them culturally relevant”. However, as Pina (2004) adverted, a key idea to guarantee the efficient of a bended learning program is the selection of appropriate means for design a programme for each educational need and target group.

14

 

References:Blended Learning: Resource Roundup (2014). Retrieved from http://www.edutopia.org/blended-learning-resources#graph1Bonk, C. J., & Graham, C. R. (2012). How do we design the blended? InThe handbook of blended learning: Global perspectives, local designs. John Wiley & Sons.Bremer, C. (2010, July). Efficient e-learning course design and media production. In Edulearn 2010 Proceedings, 2nd International Conference on Education and New Learning Technologies (pp. 5-7).Ehlers, U. D. (2004). Quality in e-learning from a learner’s perspective. European Journal of Open, Distance and E-learning, 7(1).Ehlers, U. (2010) ECBCheck. Handbook. InWent International Weiterbildung und Entwicklung GmbH.Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. The internet and higher education, 7(2), 95-105.Ginns, P.,& Ellis, R. (2007). Quality in blended learning: Exploring the relationships between on-line and face-to-face teaching and learning. The Internet and Higher Education, 10(1), 53-64.Ghirardini, B. (2011). E-learning methodologies: A guide for designing and developing e-learning courses. Rome: Food and Agriculture Organization of the United Nations.Graham, C. R. (2006). Blended learning systems. CJ Bonk & CR Graham, The handbook of blended learning: Global perspectives, local designs. Pfeiffer.Lim, D.H.,& Morris ,M.L. (2009). Learner and Instructional factors Influencing Learning Outcomes within a Blended Learning Environment. Educational Technology & Society, 12 (4), 282-293.Peres, P., & Pimenta, P. (2009). MIPO Model: A Framework to Help the Integration of. In Adult Learning in the Digital Age: Perspectives on Online Technologies and Outcomes: Perspectives on Online Technologies and Outcomes, 89.Pina, A. R. B. (2004). Blended learning: conceptos básicos. Pixel-Bit: Revista de medios y educación, (23), 7-20.Singh, Harvey (2003). Building Effective Blended Learning Programs. Issue of Educational Technology, 43(6),51-54.Tucker, C. (n.d.). The Basics of Blended Instruction. Retrieved from http://www.ascd.org

3. Quality of the digital blended learning training programsPrepared by: Anton (Sippyan), Hedda Vaassen, Saad Sulttan Hussein and Teklit Gebregiorgis

Summary and Objectives of the assignment:

The assignment group found the criteria in the annex to best check the aspects concerning COMMUNITY BUILDING and MOTIVATION for the described case and similar cases. Furthermore the assignment group proposed the extensions/completions as described in the annex.

The present group assignment will analyze which ECB CHECK Quality criteria can be applied to check the quality of the digital blended learning training programs/courses planned for implementation by an Alumni Community working in the field of peace and security in order to best follow up the training performance concerning the criteria of COMMUNITY BUILDING and MOTIVATION. The case to be analyzed is described under the subject “Target group / case to be studied” (point 2). At the same time the assignment will try to find out, whether the ECB CHECK Quality criteria are sufficient in order to cover all the aspects concerning COMMUNITY BUILDING AND MOTIVATION of the described case and will propose elements for extension.

3.A Definition of the Task and of the Instructions for the Assignment

(a) The participants have to develop together at least one new quality criterion for the chosen topic (here, the case under point 2) and a concept of how to apply these new criteria or(b) to work on the update and further development of the existing ECB Check Quality criteria applicable to the described case It is targeted to complete the first version of the assignment on or before the prescribed time (i.e., 13th March, 2016) and share it with the other participants in the Beta House under the title “docs”. 2 Case to be studied “Members of an alumni network on peace and security will train other network members by 16 sessions on selected modules of peace and security matters within a closed group on a virtual platform, and this by postings into a forum. This will be done starting from now within a PHASE A by 2 hours weekly posting sessions (every Wednesday) within the virtual group on the platform. During these sessions the lead trainer/instructor will give input and receive replies/suggestions by the participants (trainees).

The trainees will also interchange among themselves during this session. The lead trainer will invite the trainees every Monday to the Wednesday session and send out/post a summary of the session every Thursday. The training of PHASE A will consist of an introductory session, 14 module sessions and an evaluation session (in total 16 units). The trainees of PHASE A (if having successfully accomplished the training of PHASE A by active and regular participation in the virtual training sessions) will receive a PHASE A certificate entitling them to

16

become themselves in PHASE B virtual trainers, who will train via skype members of an NGO in a specific African country in selected modules during in a face-to-face workshop lasting about a week coordinated by the PHASE A lead trainer. The lead trainer/ instructor will supervise the skypetraining inputs. PHASE B will also end with an evaluation session and successful participation via skype trainings will entitle PHASES A trainees to get a certificate as an online trainer for the modules delivered.”

3.B Criteria and Evidence

CRITERION1:Category: D2 Motivation/ParticipationArea of the Criterion (Program related):Name of the Criterion: D.2.1 “The learning methodologies motivate

learners to actively participate in the learning process”

More detailed description of the criterion:

The learning methodology contemplates activities where learners are required to actively contribute,for example, reflection, peer feedback, group work, discussion boards.

What could be a guiding question to verify if the criterion is applicable?“Are the learning methodologies conceived to motivate learners to actively participate in the learning process? “

The following points (not comprehensive) can serve to verify whether strategies for motivation are applied:

Offer clear incentives Identify the destination, but not the path. Foster a team culture Recognize achievement Create a game Give prompt feedback Identify the consequences of failure Set clear milestones Demand improvement Communicate

(http://criteriaforsuccess.com/eleven-strategies-for-motivation/)

Update and further development of the existing ECB Check Quality criterion applicable to the described caseWhen exercises represent a challenge in themselves, but the user can always choose to be given the answer instead of being required to find it by himself / herself. This feature reinforces the control given to the user in his learning process, but at the cost of reduced challenge. However, the student can freely choose the exercises at the appropriate level of difficulty.

17

CRITERION 2:Category: D5 Collaborative LearningName of the Criterion: D.5.1 Collaborative learning is explicitly supportedArea of the Criterion (Program related):More detailed description of the criterion:

Collaborative learning is part of the pedagogical model. It includes, for example, peer review, group work, discussion boards, etc. Learners are provided with adequate environment and tools to take part in the collaborative activities and tasks. Tutors/facilitators provide guidance to learners while performing such activities/tasks and feedback once they are concluded.

What could be a guiding question to verify if the criterion is applicable?“By which indicators can you recognize that collaborative learning is explicitly supported?”

In valorizing to which extent collaborative learning is explicitly supported, you can analyze, whether building on and valuing the prior knowledge of the learners or encouraging confident sharing of ideas with a common purpose is ncouraged. The major characteristics for such an explicit support are:

Involvement Cooperation and teamwork Civic Responsibility

Update and further development of the existing ECB Check Quality criterion applicable to the described case

The instructor and/or facilitator can promote learning within the group first by getting to know the group members among each other via “getting-to-know” methods, which are merely intended to create trust and confidence among the learners and to create a group feeling, as with a group feeling, the participants will experience a certain “social responsibility” towards the other group members and will act with responsibility. It must also be paid attention to a certain tone within the group, so that interaction is done in a respectful andculturally sensitive way. It is also important that he hierarchy is kept low and that answers are also admitted from group members with a less hierarchical status. This should be promoted by the facilitator/instructor to foster collaboration. Collaboration on tasks must also be meaningful to the learners and make sense, otherwise they won’t be willing to contribute. So either, the learners all see the sense of collaboration or they should contribute to create this common sense by asking them beforehand how they would like to construct their way to achieve a common goal, otherwise they won’t support the goal.The facilitator/instructor should also stimulate collaboration by encouraging links between persons that have not yet collaborated. In some cultures, collaboration is not applied in learning groups, nor exchange of knowledge, so

18

every exchange of knowledge towards a clearly defined mutual goal should be encouraged by the facilitator/instructor. The learners differences should be taken advantage of – one is for example theoretical, the other more visionary, the other more systematic, the other more creative – the instructor/facilitator should combine all these different learning types within a group and encourage them to profit from each other’s diversity than by cheating each other when not performing in one’s own way.The facilitator/instructor should very much pay attention to critical voices – the critics should be given in an objective and not blessing way. Rules for criticism / comment giving should be established in the beginning of group work.The instructor/facilitator can teach the learners – while teaching a subject – also the methodology to rely on each other as resources but should also tell the learners explicitly that this form of collaboration is wanted.(http://study.com/academy/lesson/what-is-collaborative-learning-benefits-theory- definition.html)

CRITERION 3:Category: D5 Collaborative LearningName of the Criterion: D.5.2 Search functions are available for forums,

discussion boards or blogs, whenever such toolsare utilized

Area of the Criterion (Program related):More detailed description of the criterion:

The learning environment provides search functions to the collaborative tools utilized in the program/course.

What could be a guiding question to verify if the criterion is applicable?“Does the learning environment provide search functions to the collaborative tools?”

Update and further development of the existing ECB Check Quality criterion applicable to the described case

The learners should be informed that they should use key words so that others can retrieve their contributions if needed in team/group work and should use a specified terminology together or work it out so that when searching all group members refer to the same meaning. So a terminology should be added to a working group working on a specific subject for a longer period.

19

CRITERION 4:Category: D4 E-TutoringName of the Criterion: D.4.1 “A tutor provides guidance and accompanies

the learners throughout the learning processArea of the Criterion (Program related):More detailed description of the criterion:

A tutor is available to guide learners through the learning process, helping them and providingassistance with program related questions/problems. Tutors can sometimes also provide technical support or this service can be offered by another individual.

What could be a guiding question to verify if the criterion is applicable?“How can you evaluate and measure in which way and to what extent learners experience guidance throughout the learning process?”

Update and further development of the existing ECB Check Quality criterion applicable to the described case

Guidance is not only a technical guidance or on program related questions/problems. Guidance can also be felt by the tone and the quickness of a reply, the depth or the willingness to get further information, the attentiveness to the particular case of the learner, the insight in special circumstances, the interest in his/her personal situation, and perhaps the link that the tutor createsbetween the learner and the other learners. A quality of guidance can also evolve from the group dynamics to which the tutor contributes by his/her interference. As learners evaluate themselves also as beings in group surroundings, they must also feel well in the learners’ group they are in, so the tutor should also provide guidance to everyone to become a good group member in the whole context, which is dynamic. The demands on online tutors are much greater than those on face-to-face tutors in terms of roles, partly because the online tutor has a more intense focus on relationships, than in face- to face environments, where course administrators, and other staff, are more accessible, and partly because of the technology issue.(Ref: McKenzie-D 2000a: http://www.fredriley.org.uk/callhull/otis/t2-06.pdf ) (Ref, if needed: http://proceedings.informingscience.org/InSITE2007/IISITv4p201-210Sulc388.pdf )

20

CRITERION 5:Category: D4 E-TutoringName of the Criterion: D.4.2 “A tutor provides feedback to learners on

tasks/activities learners are required to perform.”Area of the Criterion (Program related):More detailed description of the criterion:

The feedback provided by the tutor is timely, accurate, helpful for future learning and targeted

What could be a guiding question to verify if the criterion is applicable?“How do you measure whether the feedback by the tutor is timely, accurate, helpful and targeted?”

Update and further development of the existing ECB Check Quality criterion applicable to the described case

It is widely recognized that feedback is an important part of the learning cycle, but both students and teachers frequently express disappointment and frustration in relation to the conduct of the feedback process. Therefore it should be made clear from the beginning in the learning process, which kind of feedbackthe learners can expect (certain feed-back rules should be set up from the beginning) and whether they can also give feed-back whether the feed-back is convenient / sufficient to them. The learners should know in advance what kind of feed-back they can expect and in which time limit. The type and frequency of feed-back should be foreseeable and known in advance to create a relation built on trust between the learner and the feed-back-giver. Generally, feedback has to be given as soon as possible after the completion of the learning task. The comments on learners’ tasks are frequently written in a language or style that demotivates the student. For example, in the study by Duncan (2007) he refers to the injunction of a lecturer to a student to “use a more academic style”, a comment, which lecturers obviously understood, but which students in the study reported as difficult to interpret. Therefore not only the contents but also the feed-back style to the learner should be taken into consideration as well as the learners’ needs – some are more academically orientated, while others are motivated by other learning goals. The tutor should take into consideration these aspects, if possible, in order to have a more positive impact and create a personal relation with the learner – which will encourage the learner more.(Ref: http://www.waikato.ac.nz/tdu/pdf/booklets/6_AssessmentFeedback.pdf )The graphic above shows that feed-back may be often very formal and object oriented, while feedback taking into consideration the individual behavior of the learner may contribute to even better learning results.

21

CRITERION 6:Category: D4 E-TutoringName of the Criterion: D.4.3 Tutors have specific knowledge and

competences to facilitate online courses/programs”

Area of the Criterion (Program related):More detailed description of the criterion:

Tutors should be specifically trained on online facilitation skills in order to adequately support learners in an online environment

What could be guiding questions to verify if the criterion is applicable?“In which online tools are the tutors trained? In which tools have they gained their experiences? How long? Worldwide or in specific cultures/ for specific regions? For which subjects are they competent to facilitate?”

Update and further development of the existing ECB Check Quality criterion applicable to the described case

Knowledge and communicational competencies are very important to support the learning process. These competencies are more or less those required to be a facilitator, trainer or a teacher. Nevertheless, to interact at a distance is different than in face-to-face (teacher’s and learners pace, use of electronic tools, …). That’s why the e-tutor has not only to animate and manage groups of learners, but also has to advise them to use the didactical resources, chats and forums or other e-forms. But technical competencies and subject related knowledge is not sufficient: Cultural competencies have also to be included and especially when dealing with sensitive subjects such as peace and security/human rights issues, these abilities are predominant and high trust-worthiness and personal confidentiality are a must.

CRITERION 7Category: D4 E-TutoringName of the Criterion: D.4.4 “A set of predefined tutoring skills are being

used as a standard within the program”.Area of the Criterion (Program related):More detailed description of the criterion:

Tutors are equipped with suitable skills for tutoring which are documented as a skill standard for tutoring

What could be guiding questions to verify if the criterion is applicable?“How is the skill standard for tutoring defined and is a certain tutor equipped with these skill standards? Where are the tutors supposed to tutor – in an online

22

course, in an online chat, in a MOOC, in a webinar, in a virtual space? (For each medium, different skills are necessary).”

Update and further development of the existing ECB Check Quality criterion applicable to the described caseThe essential tutoring skills are, finding the problem, using course materials to overcome difficulties and using direct instruction to overcome difficulties.The three roles for the online tutor are:

- the tutor as subject expert (Presenting, Demonstrating, Referring and Contributing),

- the tutor as coach (Questioning, Listening, Feeding back, Encouraging, Motivating and Controlling) and

- the tutor as assessor (email attachments, chat sessions, practical assignments, practical exercises carried out using audio and video, etc)

(Re: http://www.fastrak-consulting.co.uk/tactix/Features/tutoring/tutoring.htm )

In some cases – as in online discussions that serve for learning purposes – the tutor assists to animate the discussion and lead it towards the learning goal. Here facilitating capacities are also necessary and a good knowledge of the subject to be discussed about.

CRITERIA TO ADD TO ECB CHECK related community building in online trainings/communities for subjects on peace and security / human rights

Please consider the security aspect: All exchange must be technically safe, done in a closed virtual space, which needs to be highly protected; members should be selected on a very specific basis to be sure who is admitted. Tutors and trainers should also merit high confidence, so that the online training members can exchange among each other on a very high level of trust, otherwise no learningwill be possible. The trainers should also be culturally and politically highly sensitive and well informed, as peace and security matters include a high potential of conflict, if dealt without these qualities. In virtual working groups (especially during chat sessions) there should be the possibility of bilateralexchanges (“whispering”) between participants (especially tutors/trainers) to discuss problematic issues in the background while a chat is ongoing. In addition, an independent exchange space for the Steering Team should exist. The persons participating in an online training on peace and security issues should be authorized by their superiors to participate, as what is exchanged in online trainings is often not as much documented to the outside world as workshops with print material, so superiors are often reluctant to let their staff participate in such virtual exchanges as they feel they can’t control the exchange, while on the other side they see it as their duty to control all domains that are related to State security. Here is a conflict that should be paid attention to.

23

References

Aimard, V. & Mc Cullough, C. (2006). E-Learning in Europe: How do trainers, teachers and learners rate e-learning? Retrieved 20. 4. 2006 from: http://cms.eun.org/shared/data/pdf/report_survey_teachers_and_learners_and_e-learning_final.pdfBork, A. (2000). Teaching and learning are often mentioned together. Retrieved 2. 11. 2006 from:http://www.educause.edu/ir/library/pdf/ERM0010.pdfChang, S. (2004). The roles of mentors in electronic learning environments. AACE Journal, 12(3), 331–342.Dagger, D. & Wade, V. P. (2004). Evaluation of Adaptive Course Construction Toolkit (ACCT).Retrieved 31. 8.2006 from: http://wwwis.win.tue.nl/~acristea/AAAEH05/papers/6-a3eh_daggerd_IOS_format_v1.1.pdfDenis, B., Watland, P., Pirotte, S. & Verday, N. (2004). Roles and competencies of the etutor. Retrieved 21.4.2006 from:http://www.shef.ac.uk/nlc2004/Proceedings/Symposia/Symposium6/Denis_et_al.htmDiaz, D. P. (2002). Online drop rates revisited. The Technology Source Archives. May/June 2002. Retrieved 31.8.2006 from: http://technologysource.org/article/online_drop_rates_revisited/Flood, J. (2002). Read all about it: Online learning facing 80% attrition rates. V TOJDE, 3(2). Retrieved 31.8.2006 from: http://tojde.anadolu.edu.tr/tojde6/articles/jim2.htmHolmberg, B. (1995). The evolution of the character and practice of distance education. Open Learning, 10(2), 47-53.MacDonald, C. J., Stodel, E. J., Farres, L. G., & Breithaupt, K. (2001). In M. A. Gabriel, The demand driven learning model: A framework for web-based learning. The Internet and Higher Education,1(4), 9–30.Alexander, W, Higgison, C and Mogey, N (1999) Videoconferencing in Teaching and Learning: Case Studies, TALiSMAN/LTDI. Available online at http://www.icbl.hw.ac.uk/ltdi/ltdipub.htm#VCStudies(accessed 12 Feb 2001Collins, M and Berge, Z L (1996) Facilitating Interaction in Computer Mediated Online Courses. Background paper for our presentation at theFSU/AECT Distance Education Conference, Tallahasee FL, June, 1996. Online document available athttp://star.ucc.nau.edu/~mauri/moderate/flcc.html(accessed 12 Feb 2001).ANNEX (point 3 Criteria and Evidence)

24

4. Quality Assessment criteria for Blended LearningPrepared by: Jan, Maggy Beukes-Amiss, Albrecht Tiedemann, Obed Sindy, Mel Tan, Ana Riza Mendoza, Katrin Backhaus, Andrew Mina, Bahran Atabeyli, Olimpius Istrate, Anton, Nigel, Gino Schiavone, Katharina Rohde, Nelly Sargsyan, Tsetseg-Ulzii, Fabrizio Mellado, Alkautsar AL, Carlos Würschmidt

 Short summary:

4.A. The Task

In the framework of the MOOC “Quality in Digital Learning” participants formed working groups. The groups got the task to develop a set of 3 to max. 10 core quality criteria and a description on how the criteria might be applied in an overall quality assurance scheme such as the Open ECB-Check.

4.B. Chosen topic “QA criteria for Blended Learning”

The group use the following definition of blended learningBlended learning is an approach of facilitation a course / module / programme in which 20% - 70% of the content is facilitated face to face with learners away from their homes in a ‘facilitation room’, the other part of the course is facilitated at a distance with learners having control over time, place, pace, path of learning the content presented print based or digital

The group developed core criteria for the sub-standards:

1) Target group and learning -outcomes,2) Facilitation methods used in the BL course3) Assessment standards

For each standard a few criteria were developed, more are needed to cover the substandard. For every criterion we have defined one or more sets of indicators and evidence.

25

1 Target group and learning outcomesCriterion Indicators Evidence

1.1 A learners’ profile for the target group is defined

1. The learners’ profile covers, but not restricted to, information on age, gender, occupation, education level, working experiences, cultural background, distance from institute / geographical distribution, family circumstances, number of learners in course, time available for study, relevant practical skills, relevant attitudes, unhelpful knowledge / skills / attitudes, relevant experiences, relevant interests,

Complete description of profile of the target group approved by the institutionSurvey report on needs and expectation of target audience for the course

1.2 Learning outcomes are specific, well-defined and measurable

1. Learning outcomes for the course, module, and session are formulated in Mager’s format2. Learning outcomes are SMART: specific, measurable, achievable, realistic, time-based3. Learning outcomes cover knowledge, skills, attitudes, including coverage of general skills (see 2.3)

Documents attesting internal and external verification and approval of the stated learning outcomes in Mager’s format and being SMART

1.3 Evidence to be produced by learners to demonstrate achievement of the LO(s) is unambiguously described

1. Performance criteria – what learners have to do, to demonstrate, to produce are unambiguously formulated for each learning outcome.2. The range / scope of each learning outcome is unambiguously stated3. Evidence to be produced includes, but not restricted to, documentation of the process to attain competence (multiple versions of a document / 1st assessments instruments & reassessments), observation instruments, end products

Documents attesting internal and external verification and approval of the stated PCs and range

1.4 Implications of learners profile for methods of facilitation, type of media and technologies used are worked out in the planning stage of the course

1. Document outlining the implications for facilitation the BL course, the type of media to used, technology needed is developed

 

Completed and approved document on implications of the learners’ profile for the course design and facilitation

To continue

2 Facilitation methods used in the BL courseCriterion Indicators Evidence2.1 Facilitation methods (f2f or online) are appropriate, varied and effective to enhance active learning

1. Facilitation methods are aligned to the learning outcomes, the learners profile and the learning activities2. A variety of fit for purpose (online) tools for communication, collaboration, presentation of content, assessment, evaluation are used, including, but not restricted to trainer guided learning, experiential learning, cooperative learning, individualised learning, online learning, role plays, group work, interactive worksheets, presentations, demonstrations, web search, case studies, discussion fora, gamification…{ADD}3. A range of learner (online) support features are available (e.g glossary, help desk for technical issues (troubleshooting), diagnostic test, remedial tasks, extension tasks, ..4. (Variety of) modes of communication with facilitator /

A completed learning assessment plan for the course detailing for each planned session, time, the learning outcomes, the facilitators, activities, the learners’ activities, the assessment method used, the evidence produced by learners, the resources needed

27

administration / peers are used5.Distribution of learning activities between f2f and online is optimal in terms efficient use of resources (financial, human, environmental impact)

2.2 Facilitation methods / techniques used are, if required, supported by secure, reliable well-functioning and maintained technical infrastructure

1. The technological infrastructure supports stable access to digital resources by facilitators / learners2. The technological infrastructure supports f2f and online communication / interaction between leaners, between learners and facilitator, between learner and content

Checklist completed by external verifier indicating that technical infrastructure supports facilitation effectively

2.3 Facilitation ensures coverage of transferable / general skills

1. Facilitation has embedded general skills such as, but not restricted to, numeracy literacy, digital technology literacy, communication skills, media literacy, personal & interpersonal skills, environmental / global literacy, problem solving & decision taking skills, management & entrepreneurial skills, planning & organising skills2. Facilitation includes sufficient opportunities for learners to practice and reflect on the use of general skills

Checklist completed by external verifier having checked the learning / assessment plan for adequate coverage and assessment of (embedded) general skills

2.4 Facilitation is provided by qualified persons

1. BL course facilitator is well qualified in (i) subject content (ii) pedagogical aspects of BL facilitation

CV of facilitator includes traceable evidence of subject and pedagogical qualifications

To continue3 Assessment standards

Criterion Indicators Evidence3.1 Assessments are well planned throughout the course

1. Course assessment plan specifying briefing date/time, duration, date of assessment, content of assessment (specification of Learning outcomes to be assessed and the conditions to receive a certificate)

Course assessment plan available to learners at start of course

28

2. Assessment criteria/ rubrics are clarified with the learners at the start of the course and prior to assessment sessions3. Assessment plan includes assessment of general skills (see 2.3)4. Assessment plan includes opportunities for reassessment of learners (those not yet competent on first assessment)5. Assessment plan includes variety of assessment methods for self-assessment, peer assessment, formative assessment and summative assessment (see 3.5)

3.2 Assessment is inclusive1. Assessment is designed such that it assesses the learning outcomes not the speed, manual dexterity, vision, hearing, or physical endurance of the learner.

Assistive technologies are in place to assess learners with special needs / disabilities.

3.3 Assessment instruments are valid

1. Aligned to the learning outcome2. Covering the LO range fully / comprehensively3. Not asking more nor less than stated in the LO’s range - covered by aligned to LO4. Assessment instruments for learners include clear guidance on what they are to do / to demonstrate5. Assessment instrument are accompanied by detailed instructions for assessor how to conduct the assessment and how to judge the evidence submitted by learners

Completed validity checklist by assessor and approved by internal verifier

3.4 Assessment instruments are effectively designed

Errors in assessment are avoided (logical error, Pygmalion effect etc.)Guidelines / instruction for the assessment task are clearly stated in detailTask / Questions are clearly phrased, using language at

Completed assessment instrument checklist by assessor and approved by internal verifier

29

level of learners and technologies needed by learners with special needs/disabilitiesSimple and appropriate layout, without visual distractors

3.5 The assessment method is appropriately chosen, considering the level of LO, competencies/ attitudes/ behaviour targeted, type and extent of content in focus

1. Assessment plan includes, but not restricted to, self-assessment, peer assessment, formative assessment, summative/outcome based assessment in order to assess knowledge, skills and attitudes2. Assessment plan includes variety of assessment instruments such as, but not restricted to, multiple choice questions, short answer questions, matching activities, essay type questions, reports, educational games

Course assessment plan available to learners at start of course stating nature of assessment and format

3.6 Assessors and internal verifiers are competent

1. Assessors and internal verifiers have been trained, certified, and have a good track record of experience on the subject matter

Valid certificates or a compilation/record of experiences (with corresponding certificates) of assessor / internal verifierAssessors have a profile on website with their experience/education outlined

3.7 Assessment activities are structured such as to minimise plagiarism

1. Number of cases of plagiarism identified in the course assignments / assessments2. A plagiarism detection programme (e.g. Turnitin) is used3. An extensive question bank is available to create (automatically) equivalent but different tasks / assessment questions4. A programme to raise awareness on what plagiarism is (training in citing resources used) and its implication is compulsory for all staff and learners, including, but not restricted to, ethical and responsible behaviour in a digital world

Report on number of cases of plagiarism identified in the course assignments / assessmentsLogs where anti-plagiarism software was usedReport on sanctions applied as aligned in plagiarism policyExternal verifier report that listed indicator are in place, monitored and implemented 

30

5. An academic integrity policy and procedures is made available to staff and learners6. Learner are to sign ‘honesty declaration’ before start of assignment / assessment task7. In online assessments navigation to other webpages is disabled8. During assessments it is ensured that learners have no access to additional / prohibited digital devices (iPad, smart phone) but only to the (electronic) device used for the assessment

3.8 The e-testing location and equipment meets the required assessment conditions and specific requirements of learners

1. Learners have the appropriate equipment for the e-test as well as any specific assistive technology

External verifier completed checklist 

3.9 The security arrangements and procedures related to the layout of the room, e-test content and candidate details are implemented

1. Positioning of workstations and invigilator’s desk allows detection of any unauthorised activity2. Learners are made aware of any security arrangements and procedures that apply to them

External verifier completed checklist, having check that the security arrangements and procedures related to the layout of the room, e-test content and candidate details are implementedCopy of briefing information on security arrangements and procedures given to learners

3.10 The institution has assessment quality assurance policy, strategy and procedures

1. Institutional policy, strategies and procedures are aligned with the accrediting body’s regulations 

Evidence of accreditation by accrediting body as official assessment centre

to continue

31

32

4.C. How the criteria might be applied in an overall quality assurance scheme such as the Open ECB-Check.

The Open ECBCheck is a quality improvement scheme for E-Learning programmes. The group identified a few criteria for a blended learning programme, hence these criteria not only look at the e-learning part of the blended programme but also to the face to face aspect. But so does the Open ECBCheck, which is not restricted to E-learning only (see for example D1.1 mentioning “blending of learning method” and specifically includes in the list of methods “face-to-face”). As such the identified criteria are not NEW (as given brief wants to be developed), but adapting / rephrasing / using as is resources mentioned in the reference section. As shown below the criteria / indicators / evidence listed by the group fit completely into the ECBCheck framework.Below we look at the criteria, indicators and evidence listed in the document and how the relate to the ECB Check framework.

1) Target group and learning outcomeCriteria  1.1 A learners’ profile for the target group is defined

Criterion is implicitly covered in B1

1.2 Learning outcomes are defined Criterion is covered by A1.21.3 Evidence to be produced by learners to demonstrate achievement of the LO(s) is unambiguously described

Criterion is not included in the ECBCheck and could be made a descriptor in section A of the ECBCheck document

1.4 Implications of learners profile for methods of facilitation, type of media and technologies used are worked out in the planning stage of the course

Criterion is implicitly covered by B1

2) Facilitation methods used in the BL course

Criteria  

2.1 Facilitation methods (f2f or online) are appropriate, varied and effective to enhance active learning

Criterion is basically covered in D1.1 / D1.3 / D2.1 / D3.3 / D5.1 / D5.2. Our suggested indicators could improve / extend some of the descriptors. The subdivision used in the ECBCheck document is more user friendly than the way we did it by packing in indicators what are in the ECBCheck separate criteria

2.2 Facilitation methods / techniques used are, if required, supported by secure, reliable well-

Criterion is covered in F4

functioning and maintained technical infrastructure

2.3 Facilitation ensures coverage of transferable / general skills

Criterion is not covered could be incorporated is an appropriate descriptor of content coverage

3) Assessment standards

Criteria  3.1 Assessments are well planned throughout the course

The criteria listed are related to D6 Assignments and learning progress. This section in the ECBCheck could be revisited and strengthened in the light of the above criteria. The ECBCheck appears to ‘restrict’ to knowledge assessment, however we feel that assessment of skills / attitudes (behaviours) should be (and can be) included in (e-)assessment. Also the importance of internal and external verification and security issues should be covered in the ECBCheck document

3.2 Assessment is inclusive3.3 Assessment instruments are valid3.4 Assessment instruments are effectively designed3.5 The assessment method is appropriately chosen, considering the level of LO, competencies/ attitudes/ behaviour targeted, type and extent of content in focus3.6 Assessors and internal verifiers are competent3.7 Assessment activities are structured such as to minimise plagiarism3.8 The e-testing location and equipment meets the required assessment conditions and specific requirements of learners 3.9 The security arrangements and procedures related to the layout of the room, e-test content and candidate details are implemented

In conclusion:

The suggested criteria and indicators are too a large extend covered in the Open ECB-Check document, but some elements mention in the indicators in this document could be used to enhance the descriptors in the Open ECB-Check document. This document can be used to improve on the Open ECB-Check document.

34

References Elements from the following documents were taken as they appeared in the documents, or rephrased and placed in the format as seen in the above document.

ECB√Check (2012) OpenECBCheck Quality criteria 2012 [online] Available from http://www.ecb-check.net/downloads/ [9 March 2016]European Association of Distance Teaching Universities (EADTU) (2012) The European quality benchmark for online, open and flexible education. Quality Assessment for E-learning: a Benchmarking Approach. 2nd ed. [online] Available from http://e-xcellencelabel.eadtu.eu/tools/manual [16 March 2016]

iNACOL (2011) National Standards for quality online teaching. [online] Available from http://www.inacol.org/wp-content/uploads/2015/02/national-standards-for-quality-online-teaching-v2.pdf [9 March 2016]SQA (2015) Quality Assurance Criteria 2015-18 [online] Available from www.sqa.org.uk/files_ccc/All_Quality_Assurance_Criteria_2015_18.pdf [9 March 2016]Wright, C.R. (2011) Developing and reviewing online courses: items for consideration. [{online]} Available from https://altc.alt.ac.uk/blog/2011/11/developing-and-reviewing-online-courses-items-for-consideration/ [9 March 2016][placed in alphabetical order]

35

5. Application and Update of existing ECB-Check quality criteriaPrepared by: Uranchimeg Tudevdagva, Kerynn Dahl, Angela Falkensteiner and Eleonora Deschamps

Short summary

IntroductionThe authors of this working group focus on the update of pre-selected quality criteria of the ECB-Check and placed emphasis on Blended-Learning scenarios when doing so. To understand the groups workflow, it seems important to shortly point out the group’s understanding of Blended-Learning:Concerning the traditional definition of Blended-Learning, this learning format is a mixture of online learning and face2face-sessions. However, we think that nowadays there are a lot of new possibilities (e.g. webinars) to extend/improve such face2face-sessions and make them accessible for people all over the world. So for this working group, Blended-Learning also includes new ways of communication. In consideration of this definition, the group placed emphasis on eTutoring as well.

We would now like to list/explain our steps that lead to this final document as well as necessary following steps.

Step 1: We agreed on three core criteria: Learning Objectives, Usability and Motivation.

Step 2: We then chose some quality criteria of the ECB-Check that fit our three main criteria best. To get some inspiration for developing new criteria, we applied the existing criteria on this MOOC.

Step 3: We then collected some additional descriptors for the existing core quality criteria that seem to be considered in a quality check as well.

Please note: The ECB-Check of course might already contain several of these collected criteria but some of them don’t seem to be listed together below one core criteria. Therefore we would have liked to create a new list with a collection of quality criteria that seem to be very important to our group.

Additional steps after peer review: Step 4: The group collected all feedback and discussed about improving

our document. Step 5: We then created the final version of our document + a short

presentation.

Further steps might be:

Expansion of our chosen core quality criteria (e.g. didactic design etc.).

36

Determination of different target audiences/stakeholders (e.g. students, professors, etc.) and consider/expand our selected quality criteria according to their possible point of view.

Step 1: Core Quality Criteria (and sub-criteria from the ECB-Check)

1. Learning ObjectivesEach organization needs to determine its own, individual learning objectives for their learners as policies, stakeholders etc. differ. In addition, each organization needs to focus on different objectives. It is important to remember: Learning objectives must be very precise and formulated understandably. Of course there exist a lot of learning objectives and it would take some time to focus on each example. Therefore, the working group agreed on just one quality criterion concerning learning objectives. We think that the following criterion is very general and can therefore be applied by almost every organization.

A.1.1 All information potential learners need to orient their decision about taking the course is available.

2. UsabilityQuality criteria (standards) of this core criteria focus on the usability of learning environments. That might be e.g. the technical platform of a MOOC or a learning management system (LMS). In this case (concerning our final document), we used this MOOC-platform as example for the application of existing ECBCheck usability-quality criteria.

E.2: Usability standards are met.Usability focuses on the easiness of the learning environment and its components, including the way materials and content are presented.

E.3: The navigation (through the mandatory learning materials) allows learners to know about their progress and position in relation to the overall content.

3. MotivationIt seems important to differ two different aspects/purposes of this core criterion. On one hand the (technical, didactical) environment of a Blended Learning scenario should motive participants to participate in the learning process:

D.2.1 The learning methodologies motivate learners to actively participate in the learning process.

D.2.2 The gamification elements in the course motivate learners to keep their learning active and interest in course during learning process.

On the other hand, within Blended Learning it is also important to take care of the participants and guide them through the learning process. Participants

37

should feel welcome and should know that support is available = evaluation of facilitation:

D.4.1 A tutor provides guidance and accompanies the learners throughout the learning process.

Step 2: Assessment of the Quality in Digital Learning MOOC by applying the above core quality criteria

1. Learning ObjectivesECB Check Criteria A.1.1 All information potential learners need to orient their decision about taking the course is available.

Application to the MOOC Quality in Digital Learning: The learning objectives are clearly communicated to participants through various communication channels including but not limited to, the newsletter, the home page news which provides clear information stipulating which stage the MOOC is at and what objectives are desirable during that stage, while the Step by Step Guide and Schedule also clearly delineate what is expected.

2. UsabilityECB Check Criteria E.2: Usability standards are met.Usability focuses on the easiness of the learning environment and its components, including the way materials and content are presented. E.3: The navigation (through the mandatory learning materials) allows learners to know about their progress and position in relation to the overall content.

Application to the MOOC Quality in Digital Learning: The MOOC is clearly designed providing separate areas for information (About, Open House and Webinars) and the exchange of ideas among participants (Beta House and Community). The most relevant information at the time is highlighted under ‘most liked this week’. The Calendar provides an excellent overview of upcoming events allowing participants to plan their activities accordingly. The interface is easy to use and well structured.

3. MotivationECB Check Criterion D.4.1 A tutor provides guidance and accompanies the learners throughout the learning process.ECB Check Criterion D.2.1 The learning methodologies motivate learners to actively participate in the learning process. ECB Check Criterion D.2.2 The gamification elements in the course motivate learners to keep their learning active and interest in course during learning process.

Application to the MOOC Quality in Digital Learning: The MOOC quiz helps to create a sense of competition and adds a fun game element to the MOOC promoting learner interest and motivation. The ‘Do you Know’ tool also piques learners interest and curiosity encouraging further involvement and

38

participation. Further game element is collection of budges. First it starting not so difficult but later it will lead learner to keep their activity in MOOC. This kind of game is very good supporting learner’s interest during digital learning. The Webinars provide a theoretical overview on subject specific topics as well as the opportunity to attain further expert knowledge by engaging in the online discussion and being able to chat to the expert presenters directly. Moderator Sybille Tuschter stimulates activity and participant involvement by making comments and suggestions and providing overall encouragement to participants. The incentive of certification at the end of the MOOC by completing the assignment encourages learners to actively participate rather than simply passively. The assignment itself also ensures that learners really engage with the topic at hand, a two-way exchange of information is enabled and dialogue and discussion are promoted.

Step 3: Additional descriptors concerning our chosen core quality criteriaIn step 3, we focused on the agreed core quality criteria (that are also part of the ECBCheck) and tried to find some additional descriptors for quality measurement.

1. Use of gamification (Motivation)Methods such as gamification encourage interaction and participation. By creating a sense of competition among participants, learners also feel more motivated to achieve the learning objectives. Learners should progress through content, motivation actions, influence behavior and drive innovation.The use of incentives is used as a strategy to increase motivation at work. Gamers are rewarded with badges, leaderboarders, levels-up and experience points. Also for performing desired behaviors or actions, passing a quiz, turning in a project, or completing a self-directed study.

2. Use of incentives (Motivation)Incentives such as rewards for example, badges, keep students motivated.Activities are tracked so that students can assess their progress and adjust their participation accordingly.

3. Definition and use of new terms within a Blended-Learning course (Usability)

Courses should use common terms for navigation so it can be ensured, that all participants can get along easily in the course. If new terms are used (e.g. concerning this MOOC “Open House”, “Beta House” etc., their meaning has to be explained within the course. An indication of these explanations must be presented to the participants at the very beginning.

4. Establishing contact with the tutors (Usability)Within a Blended-Learning course, it should be clearly stated who is/are the tutor/s, for which tasks he/she/they is/are responsible for and how participants can establish contact (contact information is given in the course).

5. Information about previous knowledge (Learning Objective)

39

If participants need previous knowledge in order to take part in a Blended-Learning course, this should be communicated at the very beginning of the course.

6. Diversity of offered learning material (Motivation)Different kinds of learning material should be offered in a Blended-Learning course (e.g. Links, Videos, Pictures/animated Pictures etc.).

7. Accessibility of Blended Learning course/platform (Usability but also technology)

The Blended Learning course/platform can be accessed from anywhere by learners if they have internet; in the same vein, the course content take in account different place of the learners by offering the possibility to access different documents, video, website depending of the area speed of connection.

References: ?

40

6 Quality Criteria for Social Media in Blended LearningPrepared by: Mohamed Moustafa, Hilkka, Shikongo Nathanael, Carlos Würschmidt

Short summary

IntroductionSocial networking technologies have been viewed as tools that enable the use of participatory pedagogies able to address the problems that have traditionally plagued distance education: creating a sense of presence, community-building, and learner participation in interactive discussions.

Social media or Web based applications include online chat forums, wikis, blogs, social networking sites make knowledge sharing easy and unobtrusive for the individual. This type of tools facilitates communication, sharing information and online socialization. Using Web 2.0, users may easily express or share their opinions, ‘think by writing’, seek others’ opinions and feedback and be connected with the others.

Social media ( Facebook,Twitter,Telegram, WhatsApp,YouTube & google+ ) features allow the participants to improve collaborative learning, e.g. the participants get information or feedback, work together in assignments.

Helping tool: A short online survey about social media in blended learning, consisting of 10 questions, was conducted. The target group; educators, learners and students in different fields (8 people from MOOC participated, the final results organized question by question are in a document in the Beta House). The results of the survey were included in the assignment.

Suggested Criteria

Criterion 1: Collaborative learning is strengthened through social media (Facebook,Twitter,YouTube,WhatsApp, Google+ )

Area of the Criterion: technical requirements

Guiding questions to verify whether the criterion is applicable:

Are the learning environments enabled with social media features which strengthen collaborative learning between the participants?

Does the learning environment offer/integrate Social Media features or the possibility to connect to Social Media platforms for collaborative learning?

How are these features integrated? Can learners find and use them easily? Do the offered Social Media features (within a learning environment) help

to strengthen collaborative learning between the participants?

41

Does the program contains specifically designed group assignments, which require collaboration among learners?

Does the learning environment offer collaboration tools such as forums, chats, blogs, to support collaborative tasks?

Is the learning environment supportive to collaborative activities/tasks? Are collaborative tools such as forum, chat, collaborative working spaces,

document sharing available to learners?

Description:Collaborative learning is part of the pedagogical model. It includes, for example, peer review, group work, discussions boards, etc. Learners are provided with adequate environment and tools to take part in the collaborative activities and tasks. Tutors/facilitators provide guidance to learners while performing such activities/tasks and feedback once they are concluded.

Criterion 2: Teachers and students engagement, their interaction through social media tools are key elements of learning process

Area of the Criterion: teaching methodologies

Guiding questions to verify whether the criterion is applicable:

Are students and teachers engaged? Do students and teachers communicate e.g. via WhatsUp? This could be one on one sessions with individual students , twice a month as a way of validating if the students are really interested

Are their needs and building relationships covered? Do students send questions to the teacher and receive feedback from their teachers? E.g. on a weekly basis, this question and answer sessions are held regularly

Are they sharing experience? Are the number of communications being recorded and are used as a survey to look at the frequency and interest of students in the learning process, are the records are kept?

Description:Students and teachers communicate, discuss and share the ideas and thoughts using social media. Twitter could an excellent tool to facilitate discussions, but the users are limited theirs with only few words which means you cannot go for more details about the topics. The strengths of each social media feature must be considered. An active ”Facebook page” can become an active platform for discussion with active members.

Criterion 3: The social media tools used are appropriate/ suitable for supporting the learning outcome.

Area of the Criterion: content design

Guiding questions to verify whether the criteria is applicable:

42

Does the content lead the learner to remember, understand or apply? Is the content provided in a way that focuses on developing the social

media competences? Are the students connecting the dots between what the content of the

study materials and the content of materials found on social media? If the students are able to link and connect what they are learning from the course content with other materials posted on social media, than we can say they fullfill the quiding questions .

Can the students carry out a practical activities, using a combination of social media tools to achieve a practical goa? (if yes) , than we can say they understand ,remember and apply .

Are learners able to voluntarily share the information that they have found by themselves on social media sites without being guided by the teacher?

Description:The content is provided in a manner that allows the learner develop competences taking into account the advantages of social media, self-paced learning and online discussion. The content allows the learner to remember and recall information, understand concepts and apply by accomplishing tasks and also to use visuals to expand on the subject such as videos.

References: ?

43