Mobile Learning Environment (MoLE) Coalition Warfare ... Documents/MoLE_Data_Analysis... · Learning courses until the JKO Mobile App is developed. ... 2.5 Deviations from the Test

Coalition Warfare Program (CWP) Mobile Learning Environment (MoLE) Proof of Concept (PoC) Test Plan

Mobile Learning Environment (MoLE)

Coalition Warfare Program (CWP)

Proof of Concept (PoC) Test Plan

Prepared By: Office of Naval Research (ONR) Naval Reserve (NR) Science & Technology

Program 38 Test & Evaluation Assessment Team and the

MoLE Testing & Evaluation Team


i

Table of Contents

1 Introduction ..............................................................................................................................1

1.1 Document Overview ....................................................................................................... 1

1.2 System Description ......................................................................................................... 1

2 Administrative Information .....................................................................................................2

2.1 General ............................................................................................................................ 2

2.2 Responsibilities ............................................................................................................... 3

2.2.1 MoLE Project Sponsor ............................................................................................ 3

2.3 Naval Reserve Science & Technology Program (Program 38) ...................................... 3

2.4 Defence Academy of the United Kingdom (DAUK) ..................................................... 3

2.4.1 U.S. Air Force European Office of Aerospace Research & Development (EOARD) ............................................................................................................................ 4

2.4.2 U.S. Air Force Research Lab (AFRL) Warfighter Readiness Research Division (711 HPW/RHA) ................................................................................................................ 4

2.4.3 Q&P, LTD............................................................................................................... 4

2.4.4 U.S. Army Technology & Advanced Telemedicine Research Center (TATRC) ... 4

2.4.5 Tribal Education Limited ........................................................................................ 5

2.4.6 Partner Nations........................................................................................................ 5

2.5 Deviations from the Test Plan......................................................................................... 5

3 Release of Information .............................................................................................................5

3.1 Assessment Data ............................................................................................................. 5

3.1.1 Research Ethics & Informed Consent ..................................................................... 6

3.1.2 Proprietary Data ...................................................................................................... 6

3.1.3 Release of Information to the Press or Commercial Vendors ................................. 6

4 Pre-Proof of Concept Evaluation Activities ............................................................................6

5 Proof of Concept Evaluation Objectives ..................................................................................7

5.1 Evaluation Criteria .......................................................................................................... 7

5.2 Scenario and Tasks ......................................................................................................... 8

5.3 Test limitations and Uncontrolled Variables ................................................................ 10

5.4 Data Analysis ................................................................................................................ 10

5.5 Proof of Concept – Timeline Order of Events List ....................................................... 18


ii

5.6 Data Collection and Analysis.......................................................................................... 1

6 Reports .....................................................................................................................................1

6.1 Final Assessment Reports ............................................................................................... 1

Figures

Figure 1-1.MoLE Application Interface Screens ............................................................................ 2

Figure 2 -MoLE Data Collection Website ...................................................................................... 1


1

1 Introduction

The Mobile Learning Environment (MoLE) Project is based on a requirement by the Commander, U.S. Naval Forces Europe (CNE)/Commander, Naval Forces Africa (CNA)/Commander, SIXTH Fleet (C6F) to effectively operate in the largest maritime area of operations (AOR) where the most difficult challenge is the ability to train and communicate.

MoLE, a Coalition Warfare Program (CWP), will help mitigate the long-standing challenge of delivering low-bandwidth on-demand training by using mobile devices where there is a limited internet connectivity and limited infrastructure. The project is sponsored by the U.S. European Command (EUCOM), and supported by other Combatant Commands (COCOM) and Department of Defense (DoD) Services.

The objective is to conduct a Proof of Concept (PoC) of the mobile learning technology capable of addressing the emerging requirement to provide training to the CNE-CAN-C6F; Deputy Director for J7 Joint and Coalition Warfighting (DD J7 JCW), NATO, Coalition, Partnership for Peace (PfP) and other International partners by using mobile devices as a training medium.

The MoLE Project will leverage the global cellular network infrastructure, mobile phones and emerging mobile application/service models to build a mobile learning (M-Learning) capability that integrates into the Joint Knowledge On-Line (JKO) Internet Public (IP) portal. The MoLE project will develop and evaluate a proof-of-concept m-learning capability that enables military personnel and Coalition partners to access and perform training, from any location, at any time, independent of bandwidth, infrastructure and geographical constraints.

1.1 Document Overview

This document addresses the evaluation approach and functional requirements for the MoLE framework and the plan to assess the use of mobile learning technology during the Proof of Concept (PoC) scheduled for 3QFY12.

This proof of concept will demonstrate an enhanced interoperability, and yield high benefits, to all Services, Combatant Commanders (COCOM), the International Health Division, Force Health Protection & Readiness organization, and others once deployed on the JKO IP by providing deployment readiness training and medical education and training to military and related civilian personnel of foreign countries.

1.2 System Description


2

The Mobile Learning Environment (MoLE) Project uses commercially available mobile technologies and the JKO IP Atlas Pro Learning Management System (LMS) to access m-Learning courses until the JKO Mobile App is developed.

MoLE will initially focus on using currently available mobile devices to assess which mobile platforms and solutions ‘best’ meet the operational needs and requirements since not every participant in this proof of concept has a state-of-the-art mobile device. This approach is very cost-effective since most mobile devices (e.g., GSM and 3G capabilities) are accessible worldwide. Therefore, the proof of concept will ensure time and financial resources are focused on identifying and developing an effective operational capability rather than the procurement of specific types of mobile technologies.

The MoLE Application is initially designed for use in supporting health sector field-related operations in noncombat environments, such as, an international or in-country disaster response mission or for foreign humanitarian assistance. The mobile applications being developed for the MoLE Project will use a state-of-the-art approach and use open architecture standards to support a wide range of medical content being integrated within the mainstream learning platform (VLE) for tracking. Figure 1-1 shows the high level view and various MoLE Application screens.

Figure 1-1.MoLE Application Interface Screens

2 Administrative Information

2.1 General

This section describes the MoLE Testing & Evaluation Team, the project sponsor, Program 38 personnel, Defence Academy of the United Kingdom (DAUK), Support Contractors and other US/International Government Personnel.


3

2.2 Responsibilities

2.2.1 MoLE Project Sponsor

The MoLE project is sponsored the Commander, U.S. Naval Forces Europe (CNE)/ Commander, U.S. Naval Forces Africa (CNA)/ Commander, SIXTH Fleet (C6F). The project co-sponsor, and responsible for the transition of the capability, is the Deputy Director Joint Staff J7, Joint and Coalition Warfighting (DD J7 JCW). There are a long list of US/International organizations involved in the project as providers of medical content, advisors to the Testing & Evaluation Working Group and active participants in the Testing & Evaluation activities.

2.3 Naval Reserve Science & Technology Program (Program 38)

The Naval Reserve Science & Technology Program 38 is responsible for developing and deploying the MoLE Testing & Evaluation Plan that will assess the ability of the MoLE Project to meet the goals and objectives.

The Program 38 Reservists will work across all MoLE Working Groups (e.g., Medical Content and Transition & Technology) is ensuring the Testing & Evaluation Strategy identified and collects the necessary data to ascertain MoLE’s ability to support the m-Learning environment. Program 38’s responsibilities include:

a) Promulgate, and make changes if necessary, to this Test Plan; b) Coordinate arrangements for services and other assessment requirements as needed; c) Provide test plan briefings for all US/International MoLE Participants during

appropriate meetings and workshops; d) Provide oversight on all data collection, analysis of the test results, and publish

appropriate reports in accordance with the Human Research Protection Program (HRPP) Guidelines and the MoLE Management Plan;

e) Provide advice to the MoLE Project Manager, including keeping him informed of any condition that may affect execution of this assessment; and

f) Collaborate with Support Contractors and other US/International Organizations providing support to the Testing & Evaluation Working Group;

g) Support the data recording requirements of this Test Plan; and h) Provide final analysis of the MoLE Proof of Concept for inclusion in the MoLE Final

Report.

2.4 Defence Academy of the United Kingdom (DAUK)

The DAUK will provide support to Program 38 as one of the co-leads on the Testing &

Evaluation Working Group. The DAUK are considered ‘subject matter experts’ in the field of


4

Learning Assessments; therefore, they are a representative of the United Kingdom’s Ministry of Defence (MoD) Testing & Evaluation Team. DAUK’s responsibilities include:

a) Participating in all Testing & Evaluation Working Group meetings, teleconferences, and other activities required to determine the optimum statistical evaluation approaches;

b) Support the Program 38 Testing & Evaluation Team in conducting the analysis and data identification procedures;

c) Complete the Collaborative Institutional Training Initiative (CITI) Human Research Protection Program (HRPP) On-Line Training Course since they are conducting additional research related to m-Learning; and

d) Support the development of Testing & Evaluation Briefings, as required.

2.4.1 U.S. Air Force European Office of Aerospace Research & Development (EOARD)

a) Participate in Testing & Evaluation Working Groups and teleconferences to ensure

testing is in accordance with Human Research Protection Program (HRPP) Guidelines; and

b) Provide testing and evaluation advice to the MoLE Project Manager and Program 38.

2.4.2 U.S. Air Force Research Lab (AFRL) Warfighter Readiness Research Division (711 HPW/RHA)

a) Participate in all Testing & Evaluation Working Group meetings, teleconferences, and other activities required for determining optimal testing and evaluation processes.

2.4.3 Q&P, LTD

a) Perform as the Principle Advisor on Human Research Protection Program (HRPP) Issues;

b) Developed the appropriate Informed Consent Forms and seek Institutional Review Board (IRB) Approvals, as required;

c) Provide input for test plan, scenario, and use case development; d) Provide input to analysis, assessment plans, and reports; and e) Incorporate inputs from the Testing & Evaluation Team into the MoLE Final Report.

2.4.4 U.S. Army Technology & Advanced Telemedicine Research Center (TATRC)

a) Provide development and clarification of content and medical related uses cases that

meet the MoLE Project goals and objectives; and


5

b) Advise the MoLE Project Manager, Technology & Transition Working Group Lead, and the Program 38 Assessment Team of any condition that may affect execution of this assessment.

2.4.5 Tribal Education Limited

a) Provide a Mobile Technology Assessment Advisor to the Testing & Evaluation

Working Group in accordance with the respective MoLE S&T Research Grant; b) Assist in the development and deployment of a ‘transparent data collection’ strategy

that meets the MoLE Project’s goals and objectives; c) Provide access to transparent digital data in an agreed upon format that will ensure

data analysis can be performed using standard open source windows based tools. d) Provide access to survey questionnaire data in text comma delimited or open source

windows based data base format e) Ensure the transparent data collection strategy is in accordance with the Human

Research Protection Program Guidelines; f) Support the data recording requirements of this test plan; and g) Do not make any changes to the application after the agreed upon configuration

control date

2.4.6 Partner Nations

a) Each of the 24-nationsl involved in the MoLE Project will participate in one of many

ways. Each country will have a representative on the MoLE Testing & Evaluation Working Group, provide inputs on the development of the MoLE Proof of Concept (PoC) Questionnaire and assist in the deployment of the MoLE PoC Informed Consent.

2.5 Deviations from the Test Plan

The MoLE assessment Program 38 Project Lead, CDR Gary Anaya is authorized to deviate from this Test Plan as the operational situation and Professional Military Judgment (PMJ) dictates as long as it aligns with the MoLE Project goals and objectives, keeping the MoLE Project Manager and the MoLE Management Team updated on what changes have been made and potential impacts.

3 Release of Information

3.1 Assessment Data


6

The following is provided on the release of MoLE Proof of Concept (PoC) Assessment Data.

a) Operational assessment data will, as expeditiously as possible, be released or shared only within the MoLE Project Management Team;

b) The release/sharing of data will not be provided during the actual conduct of the evaluation but once a valid assessment can be made based on all data collected;

c) The data, including open and transparent data, will be gathered in a means to ensure

it protects the rights, privacy and confidentially of all MoLE PoC participants; d) The final analysis, which will be released within 90-days of the assessment period,

will only be released as part of the MoLE Final Report

3.1.1 Research Ethics & Informed Consent

The Testing & Evaluation Process used in the MoLE Proof of Concept (PoC) will meet the guidelines established by the US Department of Defense Directive (DODD) 5400.11 (DOD Privacy Program, DODD 3216.2 (Protection of Human Subjects in DoD-Supported Research, SECNAVINST 3900.39D, 32CFR219 (Common Rule), European Union (EU) Data Protection Requirements and UK Ministry of Defence JSP 536 (Research Ethics Committee).

In meeting the guidelines provided in the aforementioned documents, each MoLE PoC volunteer will not be required to answer questions, but have the option to ‘opt out’ as well as all participant information will be collected anonymously.

3.1.2 Proprietary Data

The MoLE Testing & Evaluation Data does not contain any proprietary or personal data. The collection of data from the assessment will be provided in aggregate form.

3.1.3 Release of Information to the Press or Commercial Vendors

No data will be released to the press, or to any other organization involved in the MoLE Project, prior to the release of the MoLE Final Report by the MoLE Project Manager. This report will be released 60-days after the MoLE Project terminates. All information about the project is available at https://wss.apan.org/1539/JKO/mole/SitePages/Home.aspx.

4 Pre-Proof of Concept Evaluation Activities


7

Before starting the PoC, the following steps will be requested to ensure each user is ready to begin the evaluation trial:

a) Complete a demographic online questionnaire for the following: 1) Age – less than 20, 20-29, 30-39, 40-49, and 50+ 2) Gender 3) How proficient are you in English? 4) Have you been given a device for the purpose of this trial? 5) How comfortable are you with using the mobile device that’s running the MoLE

app? 6) Have you previously been involved in humanitarian assistance or disaster relief

operations? 7) What is your professional expertise? (i.e., Medical, E-learning, Other) 8) Have you taken the Trafficking in Persons (TiP ) course within the last two years? 9) Acknowledge acceptance of the MoLE Proof of Concept (PoC) Informed Consent

b) Successfully download the MoLE application from the website

5 Proof of Concept Evaluation Objectives The goal of the MoLE PoC is to demonstrate that mobile technologies are a suitable tool for

providing training to and enhancing the collaboration among joint, multinational and coalition personnel. Specific objectives include:

a) Evaluate m-Learning Technologies, hosted on a wide range of different types of mobile devices, and their ability to effectively support joint, multinational and interagency interoperability by using training material provide by the U.S. Agency for International Development (USAID) field operations guide, reference videos and reference material such as the Non-Governmental Organizations (NGO) Guide, and other training-related material;

b) Evaluate m-Learning content design, which includes ease-of-use (e.g., easy to

operate, flexible, comfortable, easy interaction, etc), usefulness (e.g., easy to collaborate, user interface, learning effectiveness, etc) and m-Learning design; and

c) Evaluate the effectiveness of m-learning technology to conduct individual training by

hosting the DoD/NATO Trafficking-in-Persons (TiP) course, modified to m-Learning formats, and make this course available on the DDJ7 JCW JKDDC JKO portal.

5.1 Evaluation Criteria


8

The MoLE assessment is primarily based on addressing its utility, usefulness, self-efficacy and accessibility of the mobile learning technology and not focused on learning effectiveness. The following questions will be answered as a basis for the evaluation criteria.

a) Utility: What is the effectiveness or practicality of using mobile technologies to provide training?

b) Usefulness: What is the benefit or availability of using mobile devices in providing training?

c) Self-efficacy: Do persons who have used the mobile device provided training believe they are capable to perform a desired outcome?

d) Accessibility: What is the degree to which a mobile training application is available in austere environments to as many Role players as possible?

5.2 Scenario and Tasks The MoLE assessment is based on a notional Humanitarian Assistance & Disaster Relief (HA/DR) scenario which covers three basic phases and associated tasks a relief worker might encounter; Pre-Deployment, Traveling or in-route to the disaster site, and arrival phase. After having properly downloaded the MoLE application as detailed in section 4.X the following tasks will be executed. Questions and analysis for each task will be addressed in following sections of this test plan

a) Scenario-1 (Get Ready): This is the pre-deployment stage that involves preparation and training necessary to embark on a HA/DR mission. The goal of this stage is to demonstrate the utility and self-efficacy of using m-learning technology to use checklists and perform training. During this phase, a relief worker is travelling to an area where there has been a natural disaster. the following tasks will be exercised:

1) Task-1: The “what to pack” checklist will evaluate the use of mobile technology

to access and manipulate checklists. The evaluation will automatically direct the evaluator to within the “Mission Tools”, “Checklists – Packing Lists” section to “Select a minimum of three things you think you will need and then “Add an item you think is missing”. Rate your confidence in using the checklist.

2) Task-2: The pre-deployment training will evaluate the use of mobile technology to conduct training. For this task, the TiP course has been adapted into the smartphone application. This task involves two steps taking the TiP pre and post-test. “Go to the Trafficking in Persons (TiP) course.” Rate how useful are mobile devices for this type of training and how easy the CTIP content to navigate.

i. Step-1: “Do the Pre-test by answering the five module 1 questions ii. Step-2:“ Complete module 1 and answer the questions at the end”


9

b) Scenario-2 (En-Route): This is the traveling stage that involves preparation while in

transit to the site of the natural disaster. The relief worker is expected to use down time to complete the following tasks expected in preparation to work with external organizations including Non-Governmental Organizations (NGOs).

1) Task-3: The NGO Guide has detailed information to help with this. Read the

synopsis and try to open the e-book. Rate how useful is access to digital reference materials on a mobile device.

2) Task-4: It is expected that the relief worker might come across unrecognized

NGOs. Use the learning section to practice recognizing logos. Use the learning section to practice recognizing logos. Rate how easily could you explain to a colleague how to look up NGO logos.

3) Task-5: John Dunlop, from the United States Agency for International Development (USAID), was interviewed about working with many different agencies. Listen to a few sections of his video interview. Rate how useful are videos as a tool to enhance your understanding of USAID.

c) Scenario-3 (On the Ground): This is the arrival stage where the relief worker is

required to utilize the mobile device to access various reference materials. 1) Task-6: You need to deliver a box to deliver a box to the NGO DART. Use

the Network section to find out more about this organization. The Network section will allow access to a central, shared address book of local contacts. Rate How helpful are collaboratively updated contact details.

d) Final Questions: The following sets of questions are embedded within the application and will follow the tasks.

1) How useful are mobile devices for training? 2) How useful are mobile devices for refresher training? 3) How useful are mobile videos to enhance understanding? 4) How useful are digital reference materials on mobile devices? 5) How useful are collaborative contact pages to access real time information? 6) How easy was the Global MedAid application to navigate? 7) Please write up to five single words which best describe your overall

experience of using the mobile device as a tool for learning 8) What did you like the most about the mobile device as a tool for learning? 9) What did you like the least about the mobile device as a tool for learning? 10) Insert any additional Comments.


10

5.3 Test limitations and Uncontrolled Variables During the MoLE Proof of Concept (PoC) there will be several ‘uncontrolled variables;’ specifically,

a) Network connectivity. Although MoLE was developed to help mitigate the long-standing challenge of delivering low-bandwidth on-demand training by using mobile devices where there is a limited internet connectivity and limited infrastructure, Network connectivity will not be a controlled variable.

b) Mobile Devices. There is a strong possibility that not all participants will have their own mobile devices; therefore, the testing period may take up-to the last day since some may have to borrow mobile devices. To accommodate for this issues, each person will be assigned a unique five-digit user id.

c) Responses to Questions. In accordance with the Human Research Protection Program,

each participant cannot be forced to answer any question they do not desire. Therefore, there will be questions that participants do not wish to respond.

5.4 Data Analysis Data collected during the PoC consists of digital (transparent) data and questionnaire

survey data embedded into the application and logged onto the MoLE server as network connectivity is available. When the Role players log in to download the application they will be assigned a unique personal identification number (PIN) and all data for each Smartphone running the MoLE application will be referenced by each PIN. At the completion of each task listed above, the user will answer several questions embedded in the MoLE application. At the conclusion of the group of tasks there will be a list of questions rating aspects of the mobile learning experience. The user will then be asked to answer three questions online (not embedded as part of the MoLE application) about his/her experience using the mobile learning device and make additional comments. Table 4.4-1 shows the transparent digital data, survey questions for each task


11

Table 4.4-1. MoLE PoC Data Source Matrix

Scenario Task / Steps

Evaluation Attribute

Questionnaire Survey

Survey Analysis Transparent Data Transparent Data Analysis

Scenario-1 (Getting Ready) Task 1 Task-1: The evaluation will automatically direct the evaluator to within the “Mission Tools”, “Checklists – Packing Lists” section to “Select a minimum of three things you think you will need and then “Add an item you think is missing”

Self-Efficacy

Q1.1:Rate your confidence in using the checklist

Q1.1: User Interface: Likert scale (strongly disagree to strongly agree).

Analysis: Use z-score to analyze difference between agree scores and disagree scores.

Data display: Scatter plot

TD1.1: Time spent on each screen

TD1.2: Total time to complete Task-1

TD1.3: Total number of keystrokes to complete checklist

TD1.4: Transmittal of three things user added to the checklist. If users add more or less items, that would indicate inability to follow tasking on mobile device.

Time to complete Task-1

* Versus Phone Type

* Versus Country

Descriptive stats showing group ability to select three or more items.


12





Scenario-1 (Getting Ready) Task 2 1) This task involves

two steps taking the TiP pre and post-test. “Go to the Trafficking in Persons (TiP) course.”

Step-1: “Do the Pre-test by answering the five module 1 questions

Step-2:“ Complete module 1 and answer the questions at the end”

Utility Q2.1: Rate how useful are mobile devices for this type of training

Q2.1 Rate how easy the CTIP content to navigate.

Q2.1

User interface: Likert scale (strongly disagree to strongly agree).



TD2.1: Total time to complete the pretest

TD2.2: Elapsed Time to complete the pretest

TD2.3: Total time to complete TiP Module 1

TD2.4: Elapsed time to complete TiP Module 1

TD2.5: Number of pre-test module 1 questions answered correctly

TD2.6: Number of post-module 1 question answered correctly after completing module 1. We are only concerned with the same questions presented on the pre-test for comparative basis.


* Versus those who have/have not previously taken TiP

Show descriptive stats including 95% CI on the before module 1 scores

Show descriptive stats including 95% CI on the after module 1 scores

Compare process benchmarks-DPMO and Sigma levels to determine if improvements in score can be attributed to the presentation of TIP module 1 in a mobile format


13





Scenario-2 (En-Route) Task 3

The NGO Guide has detailed information to help with this. Read the synopsis and open the e-book.

Utility Q3.1: Rate the use of digital reference materials on mobile devices.

Q3.1




TD3.1:Total Time to complete Task-3

TD3.2: Number of keystrokes taken to navigate to and open e-book


* Versus Phone Type

* Versus Country

Calculate the minimal amount of clicks it takes to complete the task ie (the absolute minimum amount of clicks needed to navigate to and open e-book)- Lower Side Band (LSB+E3)

Compare the baseline number (see above) to the actual number of clicks users are caching.

Run a normality test and show data in a normal curve. Include LSB and USB boundaries


14






It is expected that the relief worker might come across unrecognized NGOs. Use the learning section to practice recognizing logos. Use the learning section to practice recognizing logos

Self-Efficacy

Q4.1: Rate how easily could you explain to a colleague how to look up NGO logos.

Q4.1




TD4.1:Total Time to complete Task-4

TD4.2: Number of keystrokes taken to navigate to reference section and ability to look select the correct logos.


* Versus Phone Type

* Versus Country

Calculate the minimal amount of clicks it takes to complete the task ie (the absolute minimum amount of clicks needed to navigate to and reference)



John Dunlop, from the United States Agency for International Development (USAID), was interviewed about working with many different agencies. Listen to a few sections of his video interview.

Usefulness Q5.1: Rate how useful are videos as a tool to enhance your understanding of USAID.

Q5.1




TD5.1: Total time spent watching the USAID Video

TD5.2: Elapsed time to complete USAID Video

TD5.3: Number of times the video froze

TD5.4: Number of times the video crashed the phone

Percentage of how many times the video froze or crashed the phone determines availability=

1-(# of user crashes/total # of user submissions)


15





Scenario-3 (On-the-Ground)) Task 6 You need to deliver a box to deliver a box to the NGO DART. Use the Network section to find out more about this organization. The Network section will allow access to a central, shared address book of local contacts. Rate How helpful are collaboratively updated contact details.

Self-Efficacy

Q6.1: . Rate How helpful are collaboratively updated contact details.

Q6.1




TD6.1: Number of key stokes taken to navigate to look up where nearest DART base is

Calculate the minimal amount of clicks it takes to complete the task ie(the absolute minimum amount of clicks needed to look up where nearest DART base is)- Lower Side Band (LSB)


Run a normality test and show data in a normal curve. Include LSB and USB boundaries


16





Summary EvaluationOn the Mobile Device Task 7

Utility Q7.1 Rate the use of mobile devices for training.

Q7.2 Rate the use of mobile devices as refresher training.

Q7.3 Rate the use of videos on mobile devices to enhance understanding

Q7.4 Rate the use of digital reference materials on mobile devices.

Q7.5 Rate the use of collaborative yellow pages to access real time information

Q7.1-7.9




Data analysis for Summary Evaluation on the mobile device should include would include aggregated values displayed as spider charts


17





Summary EvaluationOnline Task 8

Q7.7 Please write up to five single words which best describe your overall experience of using the mobile device as a tool for learning.

Q7.8 What did you like most about the mobile device as a tool for learning?

Q7.9 What did you like least about the mobile device as a tool for learning?

Data display will include word clouds extracted from the free text.

Mobile Learning Environment (MoLE) Proof of Concept Test Plan Jan 2012

18

5.5 Proof of Concept – Timeline Order of Events List

Following the Proof of Concept workshop the table below depicts the time ordered events for the Proof of Concept to include the specified intervals in which transparent data will be monitored for quality and to begin exercising data reduction and analysis routines.

Mobile Learning Environment (MoLE) Proof of Concept Test Plan

A-1

5.6 Data Collection and Analysis

Once registered, all devices will begin synchronizing data with the server either on a

periodic basis or when the user initiates a manual synch. All MoLE application activity will be logged on the server. A MoLE evaluation data collection web portal will be available for the assessment to view the status of all participants or select specific countries of interest. The evaluation web portal will show progress for each user identified by PINs only. Additionally, the web site will display at which task each user is at and the time the data was last updated. Fig 2 shows a snapshot of the MoLE evaluation website. This will serve primarily as a test monitoring tool. Not shown on the website below it the detailed Comma Separated Value (CSV) data file by date.

Figure 2 -MoLE Data Collection Website

6 Reports

6.1 Final Assessment Reports

Program 38 will submit a Final Assessment Report to the Q&P, LTD where the report will be incorporated into the MoLE Final Report and stored on the https://wss.apan.org/1539/JKO/mole/SitePages/Home.aspx web site. This website will also contain relevant briefings and meeting proceedings to document the process and analysis made by the MoLE Testing & Evaluation Team.