Multichannel Learning System (MLS) Project Plan Documents/Draft... · Web viewThe Department of State (DoS), Department of Defense (DOD) and other US Government organizations use

Draft Multichannel Learning System (MLS) Project Plan

Draft

Multichannel Learning System (MLS)Project Plan

Prepared for:Ms. Janie Glover

U.S. Naval Education and Training Security Assistance Field Activity (NETSAFA) Technical Director

Project Manager

-- August 2013

1


Table of Contents

Introduction..............................................................................................................................................3Background..........................................................................................................................................4Project Challenge.................................................................................................................................6Current State of Learning Environments.............................................................................................7Multichannel frameworks and MLS....................................................................................................7Document Revision History................................................................................................................8

Multichannel Learning System (MLS) Project........................................................................................9Overview.............................................................................................................................................9Objectives............................................................................................................................................9Stage-Gate Approach...........................................................................................................................9Concept..............................................................................................................................................10Concept Evaluation...........................................................................................................................10

Management Approach..........................................................................................................................11Project Management..........................................................................................................................11MLS Project Team............................................................................................................................12Supporting Teams..............................................................................................................................12Roles & Responsibilities...................................................................................................................12US/International Organizations........................................................................................................12

Project Working Groups........................................................................................................................16Learning Content Working Group.....................................................................................................16Testing and Evaluation Working Group............................................................................................16

Human Research Protection Program....................................................................................................17MLS Access Portal................................................................................................................................18Appendix................................................................................................................................................20

Appendix A: Participating Organizations.........................................................................................21Appendix B: Acronym Index............................................................................................................22

Index......................................................................................................................................................23

Figures

Figure 1 - e-Learning Research Roadmap...............................................................................................5Figure 2 - (Draft) Stage-Gate Approach..................................................................................................9Figure 3 - Concept Evaluation (CE) Overview.....................................................................................11Figure 4 - Organizational Chart.............................................................................................................12

Tables

Table 1: Document Revision History......................................................................................................7

ii


Introduction

The Department of State (DoS), Department of Defense (DOD) and other US Government organizations use training and education as a prime tool for achieving strategic objectives, establishing rapport, opening communication, building capacity and maturing relationships with Coalition and international partners. Training is the critical component in the International Security Cooperation Program managed by the Naval Education & Training Security Assistance Field Activity (NETSAFA), United States Marine Corps Security Cooperation Group (MCSCG), Air Force Security Assistance Training Squadron (AFSAT), United States Coast Guard International Affairs and Foreign Policy (USCG/IA), and the U.S. Army Security Assistance Training Field Activity (SAFTA). Defense Security Cooperation Agency (DSCA) directs, administers and supervises the execution of the Security Cooperation Programs managed by NETSAFA, MCSGC, AFSAT, USCG/IA and SATFA and Defense Institute of Security Assistance Management (DISAM) provides professional education to those staffs as well as Security Cooperation Offices.

In 2011 - 2012, a Mobile Learning Environment (MoLE) Coalition Warfare Program (CWP) Project was initiated which focused on leveraging the global cellular network infrastructure, mobile technologies and emerging mobile devices in order to facilitate the sharing of training content. As a result of a two-year project that involved over 24 nations, the MoLE Project demonstrated the capability to support training and education delivery via mobile devices, and showed that any new content could be developed to support Department of Defense (DoD), Combatant Commanders (COCOM) and other stakeholders by articulating and refining the requirements for the use case for mobile learning.

The MoLE projects clearly showed that education and training conducted within mobile learning domain could enable future military and civilian leaders to assume greater responsibility in areas related to regional security cooperation, humanitarian assistance, peacekeeping, disaster relief, medical disaster, piracy/smuggling interdiction, etc. It was an assessment of using mobile technologies and showed that mobile applications, when developed appropriately (i.e., knowledge-, community-, and user-centered) can be an effective learning medium for on-demand training. The project, however; did not evaluate the effectiveness of learning material/content or the interoperability among US/international partners.

The Multichannel Learning System (MLS) project will evaluate the effectiveness of using multiple learning mediums (e.g., web-based learning, mobile applications, videos, e-publications, etc.) as well as improve the tracking and traceability of joint training among coalition partners. It will encompass a portfolio of multiple media systems, e.g., web-based, mobile learning applications, videos, etc., in order to provide a more individualized approaches for the delivery of learning courses/content to U.S. service members and our international partners. This will be accomplished through a series of technical trials to a demonstration of these capabilities, which includes an extensive proof-of-concept among all project participants.

The MLS Concept Evaluation (CE) Evaluation will include portions of the International Military Student Pre-Departure Briefing (IMDSPDB) that is provided to international students who receive training and education in the United States. As a result, these trials and the MLS CE should enable the DOD and military training communities to maximize investments in international training (i.e., an enhanced International Military Student Pre-Departure Briefing (IMDSPDB)) as well as evaluate the effectiveness of using multiple mediums in meeting course/content learning outcomes. These trials will also enable the DoD and military training communities to maximize investments in international training, evaluate coalition interoperability capabilities, information sharing and collaboration, as well as build new partnerships (e.g., Germany Ministry of Defense (MoD) and India).

3


The MLS Project has the endorsement of the Defense Security Cooperation Agency (DSCA), based on the project’s capability to ensure that coalition partners have global access to the Military Services’ state of the art online education and training programs. At the end of this project, the U.S. will have an enhanced operational training capability with participating partner nations, and these capabilities will be integrated into other Security Cooperation Education and Training Programs (SCETP) managed by DSCA. DSCA and the Joint Staff will participate in the “U.S. Only MLS SCETP Development and Integration Working Group,” as funding permits, to design and develop a Concept Evaluation or prototype for the integration of this effort to implement any additional changes to the Joint Knowledge Online (JKO), Security Assistance Network (SAN) or Defense Security Assistance Management System (DSAMS).

Background

In 2008, the Commander, U.S. Naval Forces Europe (CNE) set a requirement for a multinational, multilingual e-Learning environment that would support Allies, Coalition partners, North Atlantic Treaty Organization (NATO) nations, Partnership for Peace (PfP) and selected African nations. This requirement was based on meeting an urgent demand to provide training and education capabilities that would meet the Combatant Commander’s (CCMD) requirement to effectively operate in large areas of operations (AOR) where the most difficult challenge is the ability to train and communicate due to the vast nature of the AOR’s with limited connectivity and infrastructure. To meet this need, the Coalition Warfare Program (CWP) initiated a project, entitled the Multinational Virtual Learning Environment (MVLE)1.

MVLE was based on an emergent requirement to establish an advanced distance/distributed learning program, which would assist key regions of Europe and Africa in training future military and civilian leaders. Since this was the first step in providing an e-Learning capability within the global infrastructure, the MLVE Project focused on creating and strengthening partnerships in key nations in the Black Sea Region (BSR)/South Eastern Europe (SEE), addressing DoD International Security Cooperation objectives, using regional expertise in the African national, Baltic Sea, Caspian Sea, Black Sea and Mediterranean Sea to assist in developing an advanced e-Learning capability. This initial development and deployment effort involved ten countries2. The MLVE Project was an important step in establishing a coalition e-Learning environment with non-traditional partners. Through the use of a “stage gate approach,” the project showed that global partners could collaborate to develop a capability based on common goals and objectives.

MVLE showed that education and training conducted within the “learning domain” could enable future military and civilian leaders to assume greater responsibility in areas related to regional security cooperation, peacekeeping, disaster relief, and piracy/smuggling interdiction. These efforts were conducted in conjunction with the U.S. Department of Defense, U.S. European Command (USEUCOM) and particularly the U.S. Navy in key regions of Europe. Although the project was quite successful, it also identified several outstanding requirements that were needed in order to fully meet CCMD Training Requirements and meet the needs of the individual user These requirements include: (1) more functionality, (2) ease of use, (3) up-to-date learning content, (4) better, faster connectivity and (5) more advance navigation.

1 Multinational Virtual Learning Environment (MLVE) Concept Evaluation Operational Evaluation Report; 1 October 20082 Azerbaijan, Bulgaria, Georgia, Germany, Moldova, Norway, Romania, Tadzhikistan, United Kingdom, United States

4


2013C

olla

bora

tion

Pr

ojec

ts

20122008m

-Lea

rnin

gSp

eech

en

able

d

2009 2010 2011G

ener

al T

opic

s

FY11-FY12 Coalition Warfare Program (CWP) (Mobile Learning Environment (MoLE))

Multimodal Enabled Advanced Distributed Learning (N08-14)

Understanding and Assessing Cultural Differences (N09-05)

S&T Training Needs Analysis (N08-36)

ADL Structuration and Complex Adaptive Systems (N09-36)

Understanding Cultural Differences When Deploying Speech-Enabled Capabilityes (N10-23)

S-1000D API Interface (N08-35)

MVLE CCIF/CWP Project

S-1000D API Interface Follow-on Project (N09-24)

Approaches in Azerbaijian to Regional Crisis Mgmt (N09-37)

Coalition Chat Line (CCL)

What Do Learners Within An ADL Environment Need? (N09-29)

m-SCORM Challenges (N09-35)

Bi-Lingual m-Learning: A Use Case (N09-39)

JFCOM Atlas Pro Mobile

Learning (m-Learning) and

Speech-enabled

Capability

Coalition Chat Requirements (Proposed)

Linguistic/Medical Terminology Assistance (N11-02)

Linguistics Requirements (Proposed)

m-Learning S&T Requirements (N11-03)

Mobile Medical Development Project (N11-01)

Mobile Learning Contents for MSO (N11-37)

Mobile App Infrastructure for Cross Platform Deployment (N11-38)

Swiss ADL Research & Protype Development (N12-01)

What do learners in an m-Learning Environment Need? (N09-29)

Figure 1 - e-Learning Research Roadmap

After the MLVE Project ended, the Department of Defense (DoD) members3 of the MVLE Working Group, identified several approaches to addressing the shortfalls identified post-MVLE Project. As Figure 1 shows, these included additional science and technology research projects that were focused on both the short-term requirements, which focused on providing a capability based on the “low hanging fruit” of technology and on the long-term requirements, which focused on capabilities needed by the DoD but had not yet matured (e.g., automatic language translation, speech-enabled capabilities, etc.).

In working with the DOD Community, approximately ten e-Learning/advanced distributed learning research efforts were initiated. The Office of Naval Research (ONR) and Office of Naval Research Global (ONRG), focused more on the long-term goal, and investigated and evaluated the effectiveness of a speech-enabled capability. As a result, both organizations funded a two-year research project, entitled Multimodal Enabled Advanced Distributed Learning (ME-ADL), to assess the use of speech as an enabled capability. The results show that using speech as an enabled capability, by providing ‘snippets’ of learning content, is an effective way of ensuring that core learning goals and objectives are fulfilled. It also showed that speech, as a learning medium, would provide more interaction and engagement with multicultural environments.

In 2011, the Mobile Learning Environment (MoLE) was accepted as a Coalition Warfare Program. Utilizing state-of-the-art in mobile technology, MoLE advanced e-Learning to a new level and helped satisfy current training demands since most countries are using cellular networks, mobile devices and emerging mobile application/service models to deliver a range of training experiences. This capability, referred to as mobile learning or more commonly “m-Learning,” is through mobile devices to wirelessly link remote and highly mobile users directly with resources to obtain training and education.

The MoLE Project helped to address several of the shortfalls identified in the MVLE Project. It leveraged the global cellular network infrastructure, mobile technologies, and emerging mobile 3 USEUCOM, Office of the Secretary of Defence for Personnel & Readiness (OSD P&R), Deputy Director for Joint Staff (J7) Joint and Coalition Warfighting (DD J7 JCW) Joint Knowledge Online (JKO), Commander Naval Forces Europe (CNE), U.S. Space & Naval Warfare Systems Center Pacific (SSC-Pacific), SSC-Europe, SSC-Charleston, Office of Naval Research (ONR), Office of Naval Research Global (ONRG), ONR Program 38 S&T

5


application/service models, which have resulted in building a mobile learning (m-Learning) capability that integrated into the Deputy Director, Joint Staff (J-7) for Joint and Coalition Warfighting (DD J7 JCW) Joint Knowledge Online (JKO) portal. This facilitated the sharing of educational content between US and multinational partners. .

Since the first e-Learning project, the Deputy Director for Joint Staff, Joint Coalition and Warfighting (DD J7 JCW) Joint Knowledge Online (JKO) has made significant improvements to support DoD/CCMD requirements (e.g., Language and Cultural Training, mobile learning, etc.). However, there are still a few emergent requirements that are required to meet the full spectrum of requirements. Since the conception of the MVLE Project, learning technologies have developed to a point where the technology can personalize education, rapidly assess student learning, support social learning, diminish learning boundaries (e.g., language training, cultural awareness, etc.), serous games, intelligent tutoring, and provide alternative teaching methods and approaches to lifelong learning.

In meeting the goal for a technology-enabled capability, the Multichannel Learning System (MLS) Project was identified as a project to provide a more effective learning capability that would meet the needs of U.S./international partners (e.g., regional engagement, exercise/scenarios, Security Cooperation, Global Health Awareness, etc.). The goal of the MLS Project is to incorporate various training modalities (e.g., audio, e-course, mobile apps, etc.) to advance the state of education and training across DOD and the Federal Government.

Project Challenge

Training is critical to the success of a broad spectrum of deployed military operations. Most of the current training methodologies and technologies are not well suited for addressing emergent training requirements for the generation of learners (i.e., Generation X, Y and Z). This gap limits the ability to meet the global training and education needs of today’s demographics as well as provide an instructional medium that supports a personal approach.

Commanders, and other US defense-related organizations, that support international training, are finding that live instructor-led training is oftentimes unsustainable due to a high-pace operational environment, limited numbers of instructors and other budgetary constraints. To help bridge the gap, the DOD has made significant investments in e-learning platforms. While traditional e-learning systems are effective at delivering curriculum aimed at professional growth and development, they are not ideally suited for portability, broad accessibility and/or real-time collaborative learning often required during a plethora of military operations. As a result, technology with greater mobility is required, not to replace existing technologies but to supplement them in order to extend the continuum of training and education into environments not being sufficiently served.

It has been realized that the use of technology (e.g., social media, mobile apps, e-readers, audio books, etc.) is so widespread that it must become an integral part of people’s lives and learning. However, in many ways the technology is detached, and even isolated, from the human user. The user, who the technology is supposed to serve, is subservient to the technology rather than the other way around.

Technology needs to be ‘brain friendly’ in order to complement the cognitive process of the human user. This is a critical requirement if the technology is going to be used, and if it is providing an efficient and effective approach that is beneficial to the user and maximizes learning. This means that the learning needs – course objectives – is the focus of the learning and not the, oftentimes complex, navigation that is cognitively demanding.

Current State of Learning Environments

6


Governments, enterprises, and commercial businesses are migrating towards the omnipresent adoption of computational systems which adapt and personalize learning to individual needs. Advanced computational technologies and algorithms are advancing in parallel with discoveries in understanding human cognitive processes. The implications of these advances will dramatically form the framework in which we develop instructional and learning strategies for the future.

The Military Services and the Educational Communities have continuously expressed a commitment to improving each Service’s learning models by leveraging technology without sacrificing standards and the quality of training while simultaneously complying with existing laws and regulations. In looking at the learning continuum, the focus should be on providing dynamic environments, blending physical and virtual learning as well as the social learning aspects to provide an environment that focuses on meeting the needs of the individual learner and the learning outcomes. This requires a re-look at the learning environment since the arrival of a new generation of recruits4 come from a technology-dense environment where most educational learning and experiences comes from non-traditional learning approaches.

According to the ‘On Learning: Future of the Air Force Education and Training’ writings, in addition to the widely accepted education model, Bloom’s Taxonomy of affective, psychomotor and cognitive learning, learning must account for each of these domains and ensure education and training also accounts for the emotional, physical, and mental aspects of learning. Therefore, the nature of education and training dictates the right mix of training delivery in order to optimize and leverage the use of new technologies to supplement, and where appropriate, replace “hands-on” training. Therefore, the future of learning will be affected by the shifting paradigms in four areas – learners, content, technology, and the geopolitical environment.

In just a few years, since the guidance document was written, the use of technology has become so widespread that it has become an integral part of people’s lives and learning. However, in many ways the technology is detached and even isolated from the human user. The user, who the technology is supposed to serve, is subservient to the technology rather than the other way around. Technology needs to be ‘brain friendly’ so it fits and complements the cognitive processes. This is critical if technology is going to be used, and it the learning experience is to be efficient and effective. If technology is not brain friendly it may not be used, or its use may be detrimental, or at least cumbersome, rather than enhance learning.

Multichannel frameworks and MLS

In the last decade, the delivery of learning, education and training (LET) materials have been revolutionized and new approaches to provide education and training content are available. E-learning has become a collective term used to describe training by any electronic means (e.g., web-based, mobile technologies, serious games, etc.) anytime and anywhere. This has resulted in the call for an advanced state of LET referred to as “Personal Assistant for Learning (PAL)”. One of the key attributes of PAL is tracking the experiences associated with learning or training performances across different devices and formats while adapting to the learning styles of the individual – this is one of the goals of the Multichannel Learning System (MLS) project.

In order to drive successful adoption of technology as a learning medium, we need to identify the frameworks which will drive usability and efficacy. In “A Roadmap for Education Technology,” the Computing Research Association identifies several key challenges which must be addressed in order to successfully leverage the future computing technologies of tomorrow:

1. Personalizing Education a. Develop computational tools which systematically mimic a human tutor by

understanding an individual’s strengths, weaknesses, and motivations.

4 On Learning: The Future of Air Force Education and Learning; Air Education and Training Command, 30 January 2008

7


b. Understand learning models that are representative of what learners know and can do by identifying pedagogy unique to each individual learner

c. Developing data models which explain themselves to the user by addressing interface challenges between user and software

2. Assessing Student Learning a. Evolve beyond the traditional “Teach/Stop/Test” learning model to a model which

assesses learners understanding seamless without formal test boundaries.b. Understanding the characteristics and attributes that are truly representative of learning

competencies

3. Diminishing Boundaries a. Understanding how to best support learning which occurs outside of traditional academic

settingsb. Understanding how to support the transition between formal and informal learning

environments without compromising learning effectiveness

4. Developing Alternative Teaching Modes a. Understanding how software can support both social learning collaboration and content

coachingb. Capturing the key processes by which learners generate and evaluate knowledge.

Technology-enhanced learning, which involved the use of multiple learning modalities like audio books, avatars, e-readers, mobile apps, speech, tutoring, etc., must be built integrated to maximize learning and meet the needs of the individual using the above frameworks are key considerations. This also means that “learning needs” drive the technology and makes learning easier since the learner to focus on the actual learning and not how to locate the learning material. Under this approach, the user focuses on the learning objectives instead of how to acquire the information. For example, many e-learning modules make navigation complex and demanding. This taxes the brain resources for navigation through the learning module, rather than on the learning content. Through the use of speech as an enabler, the user will be able to rapidly gain access to the learning instead of focusing on the ‘next button’ so they can get to the screen that contains the course.

Therefore, a technology-enhanced capability needs to be flexible by using different approaches to navigation through the learning journey, and one that strives to minimize the cognitive load on the learner so the user can focus on the learning and not on figuring out how to navigate. By setting up expectations via consistency, the technology can guide and help the users’ brain intuitively process information; hence, reducing cognitive load. This means that learning should be engaging, interesting and interactive and suited to meet the needs to the learners’ preference.

Document Revision History

The MLS Management Plan is a living document. As conditions change and lessons are learned, this plan will be updated and coordinated with the critical stakeholders.

Versions Date Description Status0.1 1 December 2012 Original Draft Version0.2 8 April 2013 Rev 1 Draft Version0.3 1 August 2013 Rev 2 Draft Version

Table 1: Document Revision History

Multichannel Learning System (MLS) Project

Overview

8


The MLS Project Activities will be separated into phases, referred to as ‘stage-gates,’ which aligns to fiscal years (i.e., Phase I (FY 2014) and Phase II (FY 2015)). To assure MLS Project Goals and Objectives are met, a stage-gate approach will be finalized as a product of the initial MLS Kick-Off Review with participation from the international Participates who have volunteered to participate based on letters of commitment toward the development, testing and deployment of a sustained capability.

The MLS Project Management Plan will be finalized and all project task, deliverables, and actions will be identified according to each project phase and incorporated into the updated MLS Management Plan.

Objectives

The MLS Project has both a short-term and long-term strategic goal. The short-term goal is to evaluate a capability that enables the MLS international partners to evaluate the effectiveness of using a multichannel capability that will be used by more than 140 countries that participate in the Security Cooperation Education and Training Program (SCETP).

The long-term goal is to assess this capability by using courses that were provided by the Security Cooperation Programs, and other DoD Organizations, via the Joint Knowledge Online (JKO) where long-term sustainability and interoperability is assured. This objective will permit the Security Cooperation Education and Training Program (SCETP) Process to automatically track and report the successful completion of multichannel-related training to Congress; and permit the SCET Working Group to design and deliver an integrated MLS capability into the DD J7 JKO LMCS, Security Assistance Network (SAN) and Defense Security Assistance Management System (DSAMS).

Stage-Gate Approach

The Multichannel Learning System (MLS) Project will commence in October 2013 based on decisions by The Defense Institute for Security Assistance Management (DISAM) Deputy Commandant, and the Naval Education and Training Security Assistance Field Activity (NETSAFA) Technical Director based on Figure 2. The full timeline, milestones and deliverables will be confirmed after the MLS Kick-Off Meeting.

Figure 2 - (Draft) Stage-Gate ApproachAfter the MLS Kick-Off Project Meeting, Figure 2 will be modified to reflect international

participant research and development activities.

9


Concept

The Multichannel Learning System (MLS) Project provides a unique opportunity for the Security Cooperation Education and Training (SCETP) Programs, led by the Defense Institute of Security Assistance Management (DISAM) and the Naval Education and Training Security Assistance Field Activity (NETSAFA), to significantly improved the International Military Student Predeparture Brief (IMDSPDB), which is provided to both officers and enlisted personnel from more than 140 countries. In the current format, which is more than ten years old, potential students are provided access to the material via the DISAM Website, on a CD or provided via their respective country’s U.S. American Embassy.

The MLS Project is seen as an opportunity to update the content, as well as the manner in which it is made available, so foreign students traveling to the United States have an enjoyable experience. For example, many of the students come from a wide variety of national backgrounds (e.g., Europe, Asian Subcontinent, Pacific Rim, etc.) and experience both cultural and military shock. This unfamiliarity of the US environment can cause problems in lieu of the demanding classroom environment which they may face. In addition to the cultural and military shock, there may be technological challenges (e.g., personal technology, limited access to the internet, etc.) encountered in preparation for their US-based training. Therefore, the MLS Project will look at the importance of contemporary pedagogical issues related to using various technologies (i.e., web-based learning, mobile applications, video, e-publications, etc.); and conduct an in-depth analysis of the International Military Student Pre-Departure Briefing (IMDSPDB) Indoctrination Training Program to obtain a understanding of where the content aligns with the ‘learner journey’(e.g., pre-departure, in-transit and post-arrival indoctrination requirements).

The project will consist of a multi-phased approach toward the development of an enhanced IMDSPDB Indoctrination Training Program. Phase I will include the selection of a small subset of the Indoctrination material that will be used as a test case. Based on the aforementioned objectives (i.e., pedagogical issues related to using various technologies, and understanding of where the content aligns with the ‘learner journey’), the MLS Project Team will test the effectiveness of the content to meeting ‘indoctrination material requirements’. The results of this test will be documented and recommendations will be made regarding enhancements for the entire Indoctrination Training Program.

Phase II, which will occur after mid-2014, will be a decision by DISAM on how the final IMDSPBD Product will be upgraded to address the aforementioned objectives as well as develop the content to ensure that it is optimum for each learner generation. Once the upgraded capability has been developed, interested MLS participants are encouraged to participate in an evaluation of the enhanced capability before it is fielded.

The MLS Project also has a long-term objective of developing a capability that permits international participants access to courses other than the IMDSPDB Courses (e.g., Security Cooperation, Trafficking in Persons, etc.). In addition to accessing additional courses, international participants will have the course results pushed to the respective Reporting Databases. For example, for the Security Cooperation Courses, the results will be pushed to the ‘Security Cooperation Reporting Databases’ to meet DOD and other US Agency Requirements. For courses like the Trafficking in Persons, which is a required course for all DOD personnel, the respective Service/DoD Personnel System will be able to pull the results from JKO for entry into each person’s training record.

Concept Evaluation

The MLS Project has two goals; specifically, (1) Identify what is the best method to provide distance education to prepare international military students for resident training in the United States;

10

http://www.disam.dsca.mil/


and (2) Identify a ‘Joint’ solution for providing controlled access to online courses to support other Security Cooperation Training Organizations via the Joint Knowledge Online (JKO) capability.

Figure 3 – Concept Evaluation (CE) Overview

In order to “Identify what is the best method to provide distance education to prepare international military students for resident training in the United States,” Figure 3 (Concept Evaluation (CE) Overview) applies. Each of the CE volunteers will receive an email informing them of the link to the MLS CE Portal that will be used to obtain their acceptance to the MLS CE Informed Consent as well as the collection of demographic data. Once this has been accomplished each volunteer, based on the CE strategy developed by the Testing and Evaluation Working Group, will take one or more courses identified by the Learning Content Working Group.

For the data collection component, each of the courses will consist of a pre-test, the content being tested, a post-test, and a qualitative and quantitative comments questionnaire. The data collection approach will permit the MLS Testing and Evaluation Team to analyze the data to see which methods provided the optimum return-on-investment for the respective demographics. In addition to the courses, additional content may be added to allow users to evaluate the effectiveness of additional tools to support learning. For example, there are components of the International Military Student Pre-Departure Briefing (IMDSPDB)) content which includes a checklist to help potential students prepare for their US training. A checklist will be provided to evaluate it effectiveness in checking and/or adding additional items.

In reference to “Can the MLS Project provide a ‘Joint’ solution for providing controlled access to online courses to support other Security Cooperation Training Organizations via the Joint Knowledge Online (JKO) capability”, this component will be addressed separately by the Integration and Development Working Group, a US-only working group, that will look developing a Strategic Roadmap for developing and integrating future Security Cooperation Training Reporting Requirements into a repository system.

Management Approach

Project Management

The Multichannel Learning System (MLS) Project will consist of a Management Team and three working groups (i.e. Learning Courses, Testing & Evaluation, and the US-only Integration and Development); US Department of Defense (DOD) Advisors, and the prime subcontractors.

11


Figure 4 - Organizational Chart

MLS Project Team

Project Manager – NETSAFA Technical DirectorProject Management Support – Lockheed Martin International Training Team (LMITT), Q&P LtdLearning Working Group – DISAM Testing & Evaluation Working Group – 711th HWP/RHA, ONR RC

Supporting Teams

Learning Content Working Group – NETSAFA, DISAM, LMITT and Country Team Delegates will participate to ensure the development of the learning content meets core learning informational objectives.

Testing & Evaluation Working Group - 711th HPW/RHA, ONR RC, Defence Academy United Kingdom (DAUK), Q&P Ltd, and a representative from each country will ensure the testing and evaluation process meets the goals and objectives of the MLS Project as well as DoD Human Research Protection Program (HRPP) Requirements.

Integration and Development Working Group – NETSAFA, LMITT, DD J7 JKO, AFSAT, SATFA, MCSGC, USCG, DISAM, Security Assistance Network (SAN) Representative, and Defense Security Assistance Management System (DSAMS) Representative. This working group will develop a Strategic Roadmap for developing and integrating future Security Cooperation Training Reporting Requirements into a repository system so data is available to Army Training Requirements and Resources System (ATRRS) and DSAMS.

Roles & Responsibilities

To achieve both horizontal and vertical communications and integration among the US/International Organizations, the following MLS Organizational Roles and Responsibilities were defined.

US/International Organizations

12


a. U.S. Naval Education and Training Security Assistance Field Activity (NETSAFA)

- Project Leader for the Multichannel Learning System (MLS) Project- Facilitates MLS Management Meetings, Teleconferences, and related discussions- Participate in the MLS Collaboration Portal- Liaise with Stakeholders and other key personnel to ensure project goals and

objectives are addressed- Adhere to the US Department of Defense (DOD) the Human Research Protection

Program (HRPP)/Research Ethics Requirement as discussed in Research Ethics

b. Deputy Director, Joint Staff (J-7) for Joint Knowledge Online (DD J7 JKO)

- Participate as a member of the Integration and Development Working Group- Adhere to the US Department of Defense (DOD) the Human Research Protection


c. Defense Institute of Security Assistance Management (DISAM)

- Lead for the Learning Courses Working Group- Participate in all MLS discussions & reviews- Learning Courses MLS Collaboration Portal Lead - Adhere to the US Department of Defense (DOD) the Human Research Protection


d. U.S. Air Force Research Laboratory, Warfighter Readiness Research Division (711th HPW/RHA)

- Leads the Testing & Evaluation Working Group- Participate in all MLS discussions & reviews- Participate in the MLS Collaboration Portal- Participate in the MLS Concept Evaluation (CE) - Adhere to the US Department of Defense (DOD) the Human Research Protection


e. Office of Naval Research (ONR) Reserve Component (RC) Test & Evaluation (T&E) Team

- Co-Lead for the Testing & Evaluation Working Group - Participate in all MLS discussions & reviews

- Participate in the MLS Collaboration Portal - Participate in the MLS CE - Adhere to the US Department of Defense (DOD) the Human Research Protection


f. Office of Naval Research Global (ONRG)

- Participate in all MLS discussions & reviews- Provide co-funding to support MLS-related Science & Technology (S&T) and

Project Meetings

g. U.S. Naval Space & Warfare System Center Pacific (SPAWAR Pacific)

13


- DoN Approval Authority for the Human Research Protection Program (HRPP)/Research Ethics Protocol Documents

- Participate in all MLS discussions & reviews- Participate in the MLS Collaboration Portal - Participate in the MLS CE

h. Advanced Distributed Learning (ADL) Initiative

- Participate in all MLS discussions & reviews- Provide co-funding to support MLS-related Science & Technology (S&T) and

Project Meetings- Participate in the MLS CE - Adhere to the US Department of Defense (DOD) the Human Research Protection


i. Lockheed Martin International Training Team (LMITT)

- Learning Courses Advisor to the Learning Courses Working Group Lead- Participate in MLS discussions & reviews- Provides support to the Integration and Development Working Group- Participate in the MLS Collaboration Portal - Participate in the MLS CE - Adhere to the US Department of Defense (DOD) the Human Research Protection


j. Canada: Canadian Defense Academies

- Participate in the MLS Project Reviews- Canadian Lead for the resolution of Human Research Protection Program

(HRPP)/Research Ethics Requirement as discussed in Research Ethics - Provide participants for the MLS CE

k. Germany: Centre for eLearning Technology (CalTech) of the Saarland University, German Research Centre for Artificial Intelligence and Higher School of Technology and Economics

- Participate in the MLS Project Reviews- German support for the resolution of Human Research Protection Program


l. Germany: Fraunhofer-Institute for Communication, Information Processing, and Ergonomics (FKIE)

- Participate in the MLS Project Reviews- German support for the resolution of Human Research Protection Program

(HRPP)/Research Ethics Requirement as discussed in Research Ethics- Provide participants for the MLS CE

m. Germany: GIRAF PM Services GmbH

- Participate in the MLS Project Reviews

14


- German Lead for the resolution of Human Research Protection Program (HRPP)/Research Ethics Requirement as discussed in Research Ethics

- Provide participants for the MLS CE

n. Georgia: Georgian Research and Educational Networking Association (GRENA)

- Participate in the MLS Project Reviews- Georgian Lead for the resolution of Human Research Protection Program


o. Georgia: International Center for Advancement of Research, Technology & Innovation

- Participate in the MLS Project Reviews- Provide participants for the MLS CE

p. India: Hindustan University Center for Defence Technology Studies (CDTS)

- Participate in the MLS Project Reviews- Indian Lead for the resolution of Human Research Protection Program


q. Jordan: King Hussein Medical Center

- Participate in the MLS Project Reviews- Jordan Lead for the resolution of Human Research Protection Program


r. Serbia: Center for Simulations and Distance Learning of the Military Academy in Belgrade/Jefferson Institute

- Participate in the MLS Project Reviews- Serbian Lead for the resolution of Human Research Protection Program


s. United Kingdom: Q&P, Limited

- Provide support to the MLS Project Manager, DISAM Online Director, and the Testing and Evaluation Leads

- MLS Project Lead for Human Research Protection Program (HRPP)/Research Ethics Requirements

- Coordinates and participate in MLS discussions and reviews

t. United Kingdom: Defence Academy of the United Kingdom (DAUK) at Cranfield University

- Participate in the MLS Project Reviews

15


- DAUK and European Union (EU) Representative for the resolution of Human Research Protection Program (HRPP)/Research Ethics Requirement as discussed in Research Ethics

- Participate as an active member of the Testing & Evaluation Working Group- Provide participants for the MLS CE

u. United Kingdom: Tribal Education Ltd

- Provide Technical Support to the Learning Content Working Group- Participates in all MLS discussions and reviews- Develops technical architecture to support the MLS CE- Ensure Testing & Evaluation Plan is aligned with the MLS CE Informed Consent

v. United Kingdom: Sutton College of Learning Adults (SCOLA)

- Participate in the MLS Project Reviews- Lead for ensuring the Human Research Protection Program (HRPP)/Research

Ethics Requirement as discussed in Research Ethics meet their academic requirements

- Provide participants for the MLS CE

Project Working Groups

The MLS Project will consist of two working groups that will ensure the project goals and objectives are fulfilled, which will form the foundation of the project (i.e., Learning Content, Technology & Transition, and Testing & Evaluation). The third working group, the Integration and Development, will be a US-only team focused on meeting US-specific requirements and will not be referenced any further in this project plan.

Learning Content Working Group

The Learning Courses Working Group will ensure the learning courses provided by the US and the International Participates are segmented, as appropriate, to ensure they are developed according to the optimum format (e.g., audio course, e-course, apps, etc.) to meet the learning objectives. Initially, the goals and objectives of this working group include:

(1) Identify learning courses that are being provided by US/International Participates to ensure course-specific learning objectives are met. Specifically,

- Ensure courses being provided by US/International Participates can be provided in multiple formats

- International partners assist in adapting content to meet national requirements

- Civil military cooperation content may also be included as appropriate

(2) Identify the delivery formats that will support just-in-time learning through the use of various formats in order to ensure a technology-enabled capability is provided that complements traditional training and education systems as well as more of a “personal assistance for learning” approach

(3) Identify assessment tools and develop instructional measures of effectiveness based on research being conducted by US/International partners

16


Testing and Evaluation Working Group

The Testing and Evaluation (T&E) Team will focus on establishing the testing and evaluating process that will show that the MLS Concept Evaluation meets the project goals and objectives. Initially, the goals and objectives of this working group include:

(1) Conduct an operational evaluation of technology-enabled capability (e.g., audio books, e-readers, mobile apps, speech interface, tutoring, etc.), utility, usability and suitability in a multinational environment; specifically:

- Coordinate with the Learning Courses Working Group and the Technology & Transition Working Group to identify critical assessment issues

- Develop operational measures of effectiveness and performance

- Develop data collection and analysis plans to ensure these plans are sufficient to fully evaluate the project’s technical components

- Coordinate the data collection and analysis plan, with the MLS Project Research Protocol Administrator, to ensure it complies with the Human Research Protection Program (HRPP)/Research Ethics requirements

(2) Provide recommendations on how the project’s lessons learned may be incorporated into future learning initiatives to ensure it meets the goals and objects of the Department of Defense (DOD) and other US agency international training requirements.

Human Research Protection Program

The Multichannel Learning System (MLS) Project is categorized5 as a research project since it is a “systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge” and data collection will be through intervention or interaction with individuals. This data collection will occur using a number of sources (e.g., surveys, data analysis, apps, etc.), and interactions will be developed in accordance with the Testing and Evaluation Plan and disseminated in the MLS Concept Evaluation (CE) Informed Consent.

Although the data collection effort will not impinge on the privacy, confidentiality of the participates, nor pose any risk to the individuals involved, the Informed Consent document will be developed and utilized to ensure that each volunteer is aware of the purpose, duration of the Concept Evaluation, procedures and processes, Testing & Evaluation Process, risk and discomforts, potential benefit, voluntary Participate and withdrawal and confidentiality. In addition to these issues, for that volunteers that are not proficient in English, the Informed Consent will be provided in their native language.

During the first year of the project, each country will have a representative responsible for ensuring all Human Research Protection Program (HRPP)/Research Ethics issues meet their Country’s Requirements. Each will be required to have read the Belmont Report6, the Federal Policy for the Protection of Human Subjects7, and provide documentation to ensure the research protocol strategy, required by their respective country, is aligned with the MLS Informed Consent process.

5 Title 45 CFR 466 http://ohsr.od.nih.gov/guidelines/belmont.html 7 Title 45 CFR 46, Subpart A

17

http://ohsr.od.nih.gov/guidelines/belmont.html


From a US-perspective, the MLS Project is exempt from review since the project is ‘research conducted on the effectiveness of or the comparison among instructional techniques, curricula, or classroom management methods’.8 However, since the MLS Project is a global project that involves international participation, it will have an Institutional Review Board (IRB) to ensure that the testing and evaluation process meets all requirements identified by the United States9, European Union10, and each International Country participating. This process will be identified as a milestone that must be completed during the first ten months of the project.

This is an important issue since every country participating in the project may, or may not, have the same requirements as the US. For example, the European Union is more restrictive with regards to the Data Protection Requirements and the Rights of Individuals. Accordingly, the EU Data Protection Requirements state that ‘all study Participates present in a research project need to be informed about the planned research use of the collected data independently of the type of data collected’ – ‘Informed Consent is not just about Patients.11 This requirement means that individuals need to be aware of the (a) Methods used for handling personal data, (b) Justification for requesting or obtaining their data, (c) Duration of data use and storage, and a (d) Guarantee concerning the rightful use of the data.

Therefore, to ensure that all Human Research Protection Guidelines are followed, each International Country Representative involved in the MLS Project will be required to have one representative that will ensure their respective country’s guidelines and requirements are fulfilled. This individual will be responsible for coordinating with the appropriate Country Representative and for obtaining written permission before they can participate in the MLS Concept Evaluation. Once these letters have been obtained, the MLS Project will have both an Independent IRB and US Government IRB to ensure no actions are outstanding.

MLS Access Portal

Access to MLS Project Collaboration is located at https://wss.apan.org/1539/JKO/mls. This Community of Interest (CoI) provides a repository of project materials in order to ensure each stakeholder and participant has access to relevant materials/briefings. To ensure information is provided across the project, the following libraries are available:

- On the front page, participants will see a brief description of the MLS Project and the Project Calendar. On the left-hand side of the portal, which is referred to as the “Quick Launch”, participants have access to the Contacts, MLS Shared Documents, Learning Content Library, Mobile Learning Library, Research Protocol Library, and Testing & Evaluation Library. The Management Team also has access to the Management Library and the Integration Library, which is related to the US-only Working Group (Integration and Development Working Group).

- Two “working group” libraries (i.e., Learning Courses, Testing & Evaluation) contain relevant documents relevant to each working group. Each contains the title to the item as well as a brief description.

- The Management library, which is only viewable by the Management Team, contains key project documents and presentations. Adobe PDF documents of each can be viewed in the “MLS Shared Documents” section.

8 32 CDR 219 (Sec 219.101)9 DoD Directive 3210.7; DoD Directive 3216.2; DoD Directive 5400.11; SECNAVINST 3900.39D and 32 CFR 219; U.S. Air Force Research Laboratory (AFRL) Instruction 40-40210 EU Directive 95/46/EC; Ethical Review in SEVENTH Framework Programme (FP7); Guidance for Informed Consent11 Ethical Review in SEVENTH Framework Programme (FP7); Guidance for Informed Consent

18

https://wss.apan.org/1539/JKO/mls


- The Mobile Learning library contains presentations/documents related to developing mobile capabilities as well as collaborations with the Joint Knowledge Online (JKO).

- The Research Protocol Library contains references/guidance instructions related to Human Research Protocol Requirements. Once the MLS Research Protocol Documents and the CE Informed Consent has been approved by any International Institutional Review Board (IRB), these documents will also be added to this library.

19


Appendix

20


Appendix A: Participating Organizations

Canada Canadian Defence AcademiesGeorgia Georgian Research and Educational Networking Association (GRENA)Georgia International Center for the Advancement of Research, Technology & InnovationGermany Centre for eLearning Technologies (CalTech) of the Saarland University, German

Research Centre for Artificial Intelligence and Higher School of Technology and Ergonomics

Germany Fraunhofer-Institute for Communications, Information Processing, and Ergonomics (FKIE)

Germany GIRAF PM Services GmbHIndia Hindustan University Center for Defence Technologies Studies (CDTS)Jordan King Hussein Medical CenterSerbia Center for Simulations and Distance Learning of the Military Academy in BelgradeSerbia Jefferson InstituteUnited Kingdom Defence Academy of the United Kingdom (DAUK) at Cranfield UniversityUnited Kingdom Q&P, LTDUnited Kingdom Sutton College of Learning AdultsUnited Kingdom Tribal Education LtdUnited States Advanced Distributed Learning (ADL) InitiativeUnited States Defense Institute of Security Assistance Management (DISAM)United States Joint Knowledge Online (JKO)United States Lockheed Martin International Training Team (LMITT)United States Naval Education and Training Security Assistance Field Activity (NETSAFA)United States Office of Naval Research Global (ONRG)United States Office of Naval Research Reserve Component (ONR RC)United States U.S. Air Force Research Laboratory (ARFL) Warfighter Readiness Research

Division (711 HPW/RHA)United States U.S. Naval Space & Warfare Systems Center Pacific (SPAWAR Pacific)Appendix B: Acronym Index

AFRL U.S. Air Force Research LaboratoryAFSAT U.S. Air Force Security Assistance Training SquadronAOR Area of ResponsibilityAPAN All Pacific Area NetworkATRRS Army Training Requirements and Resources SystemBSR Black Sea RegionCCMD Combatant Commanders’CDTS Center for Defence Technologies StudiesCalTech Centre for eLearning TechnologyCoI Community of InterestCWP Coalition Warfare ProgramDAUK Defence Academy of the United KingdomDISAM Defence Institute of Security Assistance ManagementDOD Department of DefenseDoS Department of StateDSAMS Defense Security Assistance Management SystemDSCA Defense Security Cooperation AgencyFKIE Fraunhofer-Institute for Communications, Information Processing, and ErgonomicsGRENA Georgian Research and Educational Networking AssociationHPW/RHA U.S. Air Force Research Laboratory, Warfighter Readiness DivisionHRPP Human Research Protection ProgramIMDSPDB International Student Pre-Departure BriefingIRB Institutional Review BoardJCW Joint and Coalition WarfightingJKO Joint Knowledge OnlineLET Learning, Education and Training

21


MCSCG Marine Corps Security Cooperation GroupMLS Multichannel Learning SystemMoD Ministry of DefenceMoLE Mobile Learning EnvironmentMVLE Multinational Virtual Learning EnvironmentNATO North American Treaty OrganizationNETSAFA Naval Education and Training Security Assistance Field ActivityONR Office of Naval ResearchONRG Office of Naval Research GlobalPAL Personal Assistant for LearningPfP Partnership for PeaceCE Concept EvaluationSAFTA U.S. Army Security Assistance Training Field ActivitySAN Security Assistance NetworkSCETP Security Cooperation Education and Training ProgramSCOLA Sutton College of Learning AdultsSEE South Eastern EuropeT&E Testing and EvaluationUSCG/IA U.S. Coast Guard International Affairs and Foreign PolicyUSEUCOM U.S. European Command

22


Index

Acronym Index......................................................................................................................................21Background..............................................................................................................................................4Concept..................................................................................................................................................10Current State of Learning Environments.................................................................................................7Document Revision History.....................................................................................................................8Human Research Protection Program....................................................................................................17Introduction..............................................................................................................................................3Learning Content Working Group.........................................................................................................16Management Approach..........................................................................................................................11MLS Access Portal................................................................................................................................18MLS Project Team.................................................................................................................................12Multichannel frameworks........................................................................................................................7Objectives................................................................................................................................................9Overview..................................................................................................................................................9Participating Organizations....................................................................................................................20Project Challenge.....................................................................................................................................6Project Management..............................................................................................................................11Project Working Groups........................................................................................................................16Concept Evaluation (CE) Framework....................................................................................................10Roles & Responsibilities........................................................................................................................12Stage-Gate Approach...............................................................................................................................9Supporting Teams..................................................................................................................................12Testing and Evaluation Working Group................................................................................................16US/International Organizations.............................................................................................................12

23