CSSP Task 25 Evaluation Framework and Data Analysis Plan · Task 25 Evaluation Framework ISR Report 6060-01-03 The Canadian Safety and Security Program is led by Defence Research

The Canadian Safety and Security Program is led by

Defence Research and Development Canada’s Centre for Security Science, in partnership with Public Safety Canada

CSSP Task 25 Evaluation Framework and Data Analysis Plan

ISR Report 6060-01-03 Version 3.0

18 April 2016

Presented to:

Exercise Program Manager Defence Research and Development Canada

Centre for Security Science 222 Nepean Street, 11th Floor

Ottawa, Ontario, K1A 0K2

Prepared by:

International Safety Research

38 Colonnade Road North Ottawa, Ontario

Canada K2E 7J6

Task 25 Evaluation Framework ISR Report 6060-01-03



i

QUALITY ASSURANCE AND VERSION TRACKING

Authorization

Title CSSP Task 25 Evaluation Framework and Data Analysis Plan

Report number 6060-01-03

Version 3.0 Signature

Prepared by

Dr. Kelly Forbes

Steven Dowker

Reviewed by Devin Duncan

Approved by Ian Becking

Approved for

Corporate Release by Mike McCall

Version Tracking

Ver. Action By Date

1.0 Release to Client M. McCall 27 Mar 2016

2.0 Release to Client M. McCall 12 Apr 2016

3.0 Release to Client M. McCall 18 Apr 2016

© Her Majesty the Queen in Right of Canada, as represented by the Minister of National Defence, 2016 © Sa Majesté la Reine (en droit du Canada), telle que représentée par le ministre de la Défense nationale, 2016




ii

TABLE OF CONTENTS

1. Introduction ............................................................................................................ 4

1.1 Background ....................................................................................................... 4 1.2 CAUSE Experiment Series ................................................................................ 4 1.3 CAUSE IV experiment ....................................................................................... 5

1.3.1 Vignette 1: Interoperability of paramedic services and health .................... 5 1.3.2 Vignette 2: Public alerting, notification, warnings and digital volunteers .... 7

1.4 Report Objectives.............................................................................................. 8

2. Evaluation Framework ........................................................................................... 9

2.1 Assumptions ..................................................................................................... 9 2.2 Experiment Controllers ...................................................................................... 9

2.2.1 Vignette Leads ........................................................................................... 9 2.2.2 Controllers ................................................................................................. 9 2.2.3 Evaluators ................................................................................................ 10

2.3 Participants ..................................................................................................... 10 2.3.1 Players ..................................................................................................... 10 2.3.2 Observers ................................................................................................ 11

2.4 Data collection plan ......................................................................................... 11 2.4.1 Prior to Experiment .................................................................................. 12 2.4.2 During the experiment .............................................................................. 13 2.4.3 After the experiment ................................................................................. 14

2.5 Data Analysis .................................................................................................. 15 2.5.1 Vignette 2 web-based survey data ........................................................... 15 2.5.2 Vignettes 1 and 2 observational data ....................................................... 16 2.5.3 Vignettes 1 and 2 rating scales data ........................................................ 16

3. Conclusion ........................................................................................................... 17

Annex A. Vignette 1 Pre-experiment questions .................................................. 19

Annex B. Vignette 2 Pre Experiment Survey ........................................................ 20

Annex C. Vignette 2 Post-Experiment Questionnaire .......................................... 30

Annex D. Observation Package ............................................................................. 39

Annex E. SA and Workload rating scales ............................................................. 40

Annex F. Communications and Information Sharing (CIS) ................................. 56

Annex G. Daily hotwash questions ....................................................................... 59

Annex H. System usability scale ........................................................................... 60

Annex I. Participant feedback questionnaire ...................................................... 62

Annex J. Final AAR Questions ............................................................................. 64

Annex K. Vignette 1 – Metrics ............................................................................... 65

Annex L. Vignette 2 – Metrics ............................................................................... 68

Annex M. CAUSE Metrics ....................................................................................... 72




iii

LIST OF FIGURES Figure 1: Canadian Communications Interoperability Continuum Model ......................... 6 Figure 2: SMEM Maturity Model ..................................................................................... 8

LIST OF TABLES Table 1 Data collection tools for Vignette 1 and 2 .......................................................... 11




4

1. INTRODUCTION

1.1 Background On December 7, 2011, President Barack Obama and Prime Minister Stephen Harper released the Beyond the Border (BTB) Action Plan. This plan sets out to increase the capability of Canada and the US to rapidly respond to and recover from disasters and emergencies on either side of the border by establishing interoperable cross-border plans and capabilities for emergency management [1]. This plan aims to show that by enhancing cross-border interoperability, a higher level of situational awareness (SA) can be achieved through the sharing of alerts, warnings and incident information. This increased SA is expected to lead to improved coordination of response efforts during bi-national disasters.

1.2 CAUSE Experiment Series In order to enhance cross border emergency management capabilities, the Centre for Security Science (CSS), Public Safety Canada (PS) and the Department of Homeland Security (DHS) Science and Technology (S&T) Directorate initiated a series of cross-border Canada-US Enhanced Resiliency (CAUSE) experiments. It is hypothesized that improving shared SA between emergency response agencies during large-scale emergency events leads to enhanced community resilience. This hypothesis has been tested by using interoperable technologies in each of the annual experiments within the series. The objectives of the CAUSE Resiliency experiment series are listed below [2]:

Connect, test and demonstrate emerging operational technologies;

Advance emergency management and responder SA capabilities;

Demonstrate the value of federal Science and Technology (S&T) investments;

Demonstrate enhanced resilience through improved interoperable shared SA and mutual aid during major events;

Enhance resilience in border regions by leaving behind working operational interfaces, processes, training and exercises that will improve shared SA; and,

Execute CAUSE as a catalyst to build trust relationships in support of the BTB Action Plan.

To date, three scenario-based experiments have been conducted in Canadian and American cross-border communities. The first experiment, CAUSE I was conducted on the West Coast in June 2011[3], the second experiment, CAUSE II, was performed on the East Coast in March 2013 [4] and the third experiment, CAUSE III was performed in both Eastern [5] and Western [6] Canada in 2014. The current experiment, CAUSE IV, will be conducted in the central region and will involve bordering communities in the province of Ontario and the state of Michigan.




5

1.3 CAUSE IV experiment The objective of CAUSE IV is to design an experiment that engages Canadian and American emergency response organizations during the management of a medical response (Vignette 1) and a large scale emergency (Vignette 2) that requires cross-border cooperation. The experiment will measure the impact of interoperable communications, information sharing and SA capabilities and the responders’ ability to perform their roles. In addition, CAUSE IV will identify gaps and challenges related to health and/or technology-related policies. The specific CAUSE IV objectives are identified in the list below [7] :

Establish and field test cross-border 700MHz PSBN wireless network capability;

Examine cross-border voice and data communications between paramedic services, healthcare and border officials (e.g., transmission of electrocardiogram tracing, live video, patient records, vehicle tracking);

Explore and test approaches to enable more efficient cross-border alerts, warnings and notification systems, including organization-to-organization, and organization-to-citizen alerts;

Develop, test and evaluate local, state and provincial mutual aid workflows, systems and applications; and

Develop a cross-border operational capability that incorporates digital volunteers and social media.

CAUSE IV is comprised of two experimental vignettes each of which is associated with a unique scenario. One vignette will be led by Canada and the other vignette will be led by the US. Both vignettes and their strategic objectives are briefly described below.

1.3.1 Vignette 1: Interoperability of paramedic services and health This Canadian-led vignette will compare and contrast existing and emerging technologies (LTE) that enable the real time and near real-time exchange of voice, video and data communications. This exchange will occur during a live, simulated cross-border patient transfer between Canada and the US. Throughout the CAUSE experiment series, the impact of interoperable technology on emergency response and resiliency has been characterized through the use of a multi-dimensional model referred to as the Canadian Communications Interoperability Continuum (Figure 1). There are five dimensions that are characterized by this model including governance, standard operating procedures, technology, training/exercises and usage. The Canadian Communications Interoperability Continuum model depicts the core elements and key attributes that are needed to develop a mature interoperable capability. It will be used as a benchmark in the current experiment to measure the impact of exchanging voice, video and data over LTE on the emergency response depicted in Vignette 1 [8]. Qualitative metrics, which are categorized according to the five dimensions within this




6

model, have been developed for use in CAUSE IV (Annex M). These metrics are based on the findings from earlier research studies and are applicable to the emergency responses depicted in Vignette 1. The metrics will be used by the study team to evaluate, using a 5 point Likert rating scale, the observations that are gathered by site evaluators during the conduct of the experiment.

Figure 1: Canadian Communications Interoperability Continuum Model

Two separate experiments will be conducted within the context of Vignette 1.

Experiment A: An interfacility transfer of a patient will be simulated from a Canadian hospital and transported to a US hospital in order to establish and field test a cross-border 700MHz public safety broadband wireless network capability allowing the exchange of voice, video and data. Applications demonstrated in this part of the experiment include electronic patient care record (ePCR), electrocardiogram (ECG) transfer, automatic vehicle locator (AVL), email, and VoIP

Experiment B: The concept of session persistence will be tested by establishing a real-time video conference that enables continuous cross-border communications and information sharing using the LTE technology. A live simulation of an ambulance transiting from the US to Canada will be performed. During this transfer, interactive real-time voice and video will be exchanged using publicly available applications (e.g., gmail) during a single three-way call between the transiting paramedic crew and two healthcare facilities (one in Canada and one in the US).




7

1.3.2 Vignette 2: Public alerting, notification, warnings and digital volunteers

This US-led vignette will compare and contrast existing and emerging social media technologies that enable public alerting, warnings and notifications during large scale cross-border emergencies. In addition, this vignette, based on a Canada-US cross-border concept of operations (CONOPs), will simulate the involvement of digital volunteers and the use of social media to support traditional emergency management organizations. The use of social media technologies and the involvement of digital volunteers was introduced to the CAUSE series during the CAUSE III experiment [9]. At that time, the impact of social media technologies and digital volunteers on the conduct of an emergency recovery operation was characterized using a theoretical model referred to as the Social Media Emergency Management (SMEM) Maturity Model (Figure 2). This model was developed by the CSS as a result of work focusing on enabling the cooperation between traditional emergency response organizations and the digital volunteer community [10]. Similar to the Canadian Communications Interoperability Continuum model, the SMEM Maturity Model is intended to summarize the main elements that are necessary to develop cooperation within the emergency management domain. There are four dimensions that contribute to the development of a mature capability including people, governance, technology, and implementation [10]. Advancements in these areas should lead to improved emergency response and community resiliency. Qualitative metrics, which are categorized according to the four dimensions within this model, have been developed for use in CAUSE IV (Annex L). These metrics are based on the findings from earlier research studies and are applicable to the emergency responses depicted in Vignette 2. The metrics will be used by the study to evaluate, using a 5 point Likert rating scale, the observations that are gathered by the site evaluators during the conduct of the experiment.




8

Figure 2: SMEM Maturity Model

1.4 Report Objectives This report presents the evaluation framework and the data collection tools that were developed for the CAUSE IV experiment. The impact of using interoperable technology to communicate and exchange critical information (voice, video and data), enable increase system performance and facilitate shared SA between the responding organizations and agencies will be measured for each of the vignettes.




9

2. EVALUATION FRAMEWORK A three phase evaluation framework was generated to measure the effects of the interoperable technologies investigated during CAUSE IV on emergency response operations. The following phases are represented in the evaluation framework:

Prior to the experiment;

During the experiment and,

Following the experiment. The framework will be applied over the course of the CAUSE IV experiment, at specific data collection milestones that are prompted by the Lead Controller, to test the hypothesis related to the CAUSE Resiliency experiment series.

2.1 Assumptions The following assumptions will be made during the design and conduct of the experiment:

The role of the patients in Vignette 1 will be simulated. No actual patients will be involved in the experiment;

The role of the public in Vignette 2 will be simulated. Actual public notifications will not be tested;

No real patient data will be shared or exchanged;

There will be no inferential statistics performed due to a small number of participants;

All analyses for Vignette 1 will be performed using qualitative and descriptive techniques. Analysis of the survey (pre- and post-experiment surveys) for Vignette 2 may include quantitative analyses if the participants’ response rate is sufficient.

2.2 Experiment Controllers The CAUSE IV experiment will be facilitated by an Experiment Design Team (EDT) and a contracted study team. These roles are described below.

2.2.1 Vignette Leads The Vignette leads are responsible for the overall supervision of the Vignette and provide direction to the Lead Controller on any issues relating to technology, safety and experiment flow.

2.2.2 Controllers There are two types of controllers for this experiment. These roles are described below.




10

2.2.2.1 Lead Controller The Lead Controller is responsible for the overall execution and progress of the experiment at a strategic level. The Lead Controller acts as a main point of contact for all other controllers and makes executive decisions in consultation with the Vignette Leads on any issues that may arise. 2.2.2.2 Site Controller The Site Controllers are responsible for managing their assigned experiment locations. The Site Controllers are responsible for ensuring that the scenario unfolds as planned in their assigned location. They are the primary guides through the experiment and must ensure that all safety and security regulations are followed.

2.2.3 Evaluators There are two types of evaluators for this experiment. These roles are described below. 2.2.3.1 Lead Evaluator The Lead Evaluator is responsible for ensuring that all evaluation data collection activities progress smoothly. The Lead Evaluator acts as a main point of contact for the Site Evaluators to provide support or executive decisions when required. 2.2.3.2 Site Evaluators Site Evaluators will observe the experiment play out and will take notes on their observations. Evaluators are not to interact with the participants during the experiment except to prompt the participants to record their ratings for SA and workload and to administer data collection tools at the end of the experiment day. Any participants who ask the Site Evaluator questions will be referred to the Site Controller at their location.

2.3 Participants The CAUSE IV participants will include players and observers. Feedback will be gathered from all participant groups. Participants will attend either in person or virtually.

2.3.1 Players It is anticipated that representatives from the following organizations will participate as players and provide feedback in the CAUSE IV experiment. 2.3.1.1 Vignette 1 Participating organizations

Lambton County Emergency Medical Services (EMS);

Tri-Hospital EMS;

Wallaceburg Central Ambulance Communication Centre (CACC);

Canadian Border Services Agency (CBSA);

Federal Bridge Corporation;

Bluewater Health;




11

Lake Huron Medical Centre;

Michigan Department of Transportation (DOT); and,

US Customs and Border Protection Services (US CBP). 2.3.1.2 Vignette 2 Participating organizations

St. Clair County;

Michigan 2-1-1;

Lambton County;

City of Sarnia;

211 Ontario; and,

Everbridge.

2.3.2 Observers The observers will be invited by the CAUSE IV EDT members to attend the experiment, as part of a separate Observers’ Day program that is developed and executed by CSS. These individuals will have no official role in the experiment however, they will be invited to observe aspects of the experiment and provide feedback through established channels such as the post experiment surveys.

2.4 Data collection plan Several data collection tools have been developed for the CAUSE IV experiment. These tools support the overall evaluation framework and will be administered prior to, during or after the experiments that are conducted for the vignettes. Each of these tools and the point at which they are expected to be used is identified in Table 1. A description of each data collection tool is also provided below.

Table 1 Data collection tools for Vignette 1 and 2

Vignette 1 Vignette 2

# Tool Prior During After Prior During After

1 Pre-experiment Interviews

X

2 Part 1 of Vignette 2 online survey

X

3 Part 2 of Vignette 2 online survey

X

4 Observation Gathering

X X

5 Situation Awareness X X

6 Workload scale X X

7 Daily AAR hotwash X X

8 System Usability scale

X X

9 Daily Participant Feedback Questionnaire

X X

10 Final AAR Hotwash X X




12

2.4.1 Prior to Experiment 2.4.1.1 Vignette 1 Interviews Prior to the experiment, the Canadian study team members will conduct interviews (in person or via teleconference) to support the investigation of technology usage in Experiments A and B in Vignette 1. The purpose of this data collection activity is to identify the current organizational role in cross-border patient transfer, documentation (policies, standard operating procedures, industry standards) which guide that role and the reasons and methods for exchanging information via voice and data communications. In addition, information related to the potential improvements in patient care that can be enabled through enhanced communications and the receptiveness to change within the organizations will be gathered. A set of questions has been developed for use during these interviews (Annex A). The team will conduct short interviews with personnel from Canadian and American border organizations (CBSA, US CBP), Canadian paramedic, Canadian dispatch, Blue Water Bridge Authority, Joe Fournier from CSS and a US healthcare organization. It is anticipated that each interview will be approximately 20-30 minutes in length. The questions will be provided to the individuals in advance of the interview in order to provide them with a sufficient opportunity to review the questions and, if required, locate relevant information that will be discussed during the their interview. The schedule for conducting the data gathering interviews will be determined by the personnel to be interviewed. The TA and/or representatives from CSS will attend any or all of the interviews their convenience. 2.4.1.2 Vignette 2 pre-experiment survey A two part survey will be used to gather data from participants. Prior to the experiment, the US study team members will gather information from Disaster Response personnel and digital volunteer organizations by using Part One of the web-based survey (Annex I). The purpose of this data collection activity is to identify the organizational roles, responsibilities and policies associated with the conduct of disaster responses. In addition, the survey will gather information related to the participants’ experience with emerging technologies and engaging digital volunteers to support the issuance of public alerting, warnings and notifications. The current gaps and challenges associated with the use of existing technologies will be identified. Participants will be directed to visit the ISR website (www.i-s-r.ca) and click on a survey link that will lead to the online survey. Also, it is anticipated that data will be gathered using the pre-experiment survey from April 18-22, inclusive. This schedule is subject to change depending upon the participants’ availability and the weekly technology sessions that will be held in April to inform the participants about the technologies that will be used during Vignette 2. The pre-experiment survey will be closed prior to the start of the CAUSE IV experiment.

http://www.i-s-r.ca/




13

Following the experiment, the US study team members will gather information from participants using Part Two of the web-based survey (Annex C). This survey is described in Section 2.4.3.3.

2.4.2 During the experiment 2.4.2.1 Vignettes 1 and 2 Observations and rating scales for SA and workload During the experiment, the Canadian study team will gather observations and qualitative data associated with Experiments A and B in Vignette 1 and Vignette 2. These observations will be based upon the objectives specified for each of the vignettes and the overarching CAUSE experiment series. Where possible, these observations will be augmented with qualitative comments provided by the participants. The observation sheets that were developed for gathering the observations are presented in Annex D. The evaluation team members will be strategically positioned at the experiment sites in order to observe the simulated activity. This will require some members of the evaluation team to move across the Canada-US border to observe multiple aspects of the simulated responses during the data collection phase. The Lead Evaluator will facilitate a brief teleconference at the end of each experiment day to ensure that all Site Evaluators are able to communicate the main observations gathered during the experiment and to confirm how the data will be transferred to the study team. The Canadian study team will gather measurements and qualitative feedback associated with the participants’ SA and workload during experiments A and B in Vignette 1 (Annex E.1). The US study team will also gather measurements and qualitative feedback associated with the participants’ SA and workload during Vignette 2 (Annex E.2). A standard approach to gathering SA and workload ratings during the experiment will be used. For both Vignette 1 and Vignette 2, participants will be provided with two data collection tools for use throughout the experiment. One tool will be used to gather measurements related to SA and the other tool will be used to gather measurements related to workload. Players will be prompted by the Site Evaluator to provide the measurements that reflect their level of SA and workload at specific points during the experiment. The term SA is meant to refer to the participants’ ability to identify and understand the information within their environment and to anticipate, based on this understanding, the events (related to their tasks) that will occur in the future. The requirements for establishing and maintaining SA will differ between operational roles [13]. For the SA measurements, a rating of ‘1’ will indicate that the participants thought they had ‘No SA’ whereas, a rating of ‘5’ will indicate that the participants thought they had ‘High SA’. A rating of ‘3’ will indicate that participants thought they had ‘Some SA’ during the experimental activities. The term workload is meant to refer to the demands that are placed upon the participants during the completion of their operational tasks. There are several types of demands (cognitive, physical, effort, frustration, etc.) that may influence their workload




14

[14]. This will be unique for each of the participants' roles. For the workload measurements, a rating of ‘1’ will indicate that the participants thought their workload was ‘Too low’ whereas, a rating of ‘5’ will indicate that the participants thought their workload was ‘Unmanageable’. A rating of ‘3’ will indicate that the participants thought their workload was ‘About right’. Gathering these ratings will require that the Site Evaluators briefly interrupt the participants to gather an indication of their SA level and workload and record any qualitative comments associated with their ratings. 2.4.2.2 Vignette 1 User experience assessment of session persistence An additional data collection tool will be used to gather data related to the participants’ experiences while using the LTE to exchange voice and video during Experiment B in Vignette 1. Participants will be asked to respond to a series of statements and provide qualitative feedback related to voice and video quality by applying a 5 point Likert rating scale. The rating scale will be used to reflect the extent to which they agree or disagree with the ideas presented in the statements (Annex F). A rating of ‘1’ indicates that the participants ‘Strongly Disagree’ with the statement whereas, a rating of ‘5’ indicates that the participants ‘Strongly Agree’ with the statement. A rating of ‘3’ indicates a ‘Neutral’ score.

2.4.3 After the experiment 2.4.3.1 Vignettes 1 and 2 daily hotwash sessions At the end of each experiment day, the participants, EDT members and potentially some observers will attend a brief (approximately a half hour in duration) hotwash session at their respective locations.. A set of questions was developed to guide the discussion (0). The Site Evaluator will gather the feedback and SA and workload ratings from his/her own site participants and provide this feedback to the Lead Evaluator through an evaluator teleconference conducted immediately after the hot wash meeting. Participants located at the individual sites are invited to attend the daily hotwash session at their convenience. The focus of these daily sessions will be to gather feedback about the participants’ experiences with the emerging technology (LTE, social media) and its impact on the roles and responsibilities depicted by the participating organizations. The daily hotwash sessions will also be used to set expectations for the following experimental day. In addition, a hardcopy of the SUS (Annex H) and participant feedback questionnaire (Annex I) will be administered to participants (Section 2.4.3.2). It is expected that an evaluator’s hotwash session will be held at the end of the experiment day via teleconference rather than having Site Evaluators travel from their venue to another location. The Lead Evaluator from the Canadian study team will facilitate the discussion and will gather feedback from the Site Evaluators for subsequent analysis. 2.4.3.2 Vignettes 1 and 2 daily questionnaires During the daily hotwash sessions, the Canadian study team will also gather




15

measurements using a set of questionnaires. These tools will measure system usability associated with the technology used during the experiment and will gather feedback from participants to evaluate the planning and execution of the experiment. These tools are briefly described below.

System Usability Scale (SUS): The SUS provides a quick tool for measuring the usability (Annex H) of the technology. Participants will provide a rating and qualitative feedback to indicate the extent to which they agree with each of the statements. It consists of a 10 item questionnaire and is accompanied by a 5-point rating scale. A rating of ‘1’ refers to ‘Strongly Disagree’ and a rating of ‘5’ refers to ‘Strongly Agree. A rating of ‘3’ refers to a ‘Neutral’ response [11].

Participant Feedback Questionnaire: A brief participant feedback questionnaire will be administered to AAR participants (Annex I). The intent of this tool is to gather qualitative feedback concerning the planning, preparation and execution of the CAUSE experiment as well as the technology that was utilized.

2.4.3.3 Vignette 2 post-experiment survey Upon completion of Vignette 2, the US study team will invite the participants to complete the post-experiment survey (Annex C). It is anticipated that a link to the post-experiment survey will be hosted on the ISR website in the same way as the link to the pre-experiment survey will be hosted. Participants will be directed to visit the ISR website and click on the survey link in order to complete the survey. Data will be gathered using the post-experiment survey from April 28-May 5, inclusive. This schedule is subject to change depending upon the real-world commitments that must be managed by the participants given that they work within the emergency management domain. It is anticipated that the post-experiment survey will be closed at the end of the day on May 5. Part two of this online survey was developed by the Canadian study team to gather feedback regarding the training for and the experience using of social media technologies and digital volunteers to support disaster response organizations (Annex C). 2.4.3.4 Overall AAR session Upon conclusion of the experiment, an AAR will be held with all participants, the EDT and potentially observers. A set of questions was developed to guide the AAR discussion (Annex J). Similar to the daily hotwash sessions, the qualitative feedback gathered during the final AAR session will be recorded on the Observation Package for inclusion in the metrics-based analysis.

2.5 Data Analysis

2.5.1 Vignette 2 web-based survey data The pre-experiment and post-experiment survey data will be subjected to a descriptive analysis to determine overall responses (e.g., mean, mode, range, standard deviations) provided by the participants. This analysis will also determine the overall response rate




16

for the surveys.

2.5.2 Vignettes 1 and 2 observational data The observational data gathered using the Observation Package tools (Experiments A and B associated with Vignette 1 and Vignette 2) will be subjected to a qualitative analysis. This analysis will use qualitative metrics that were developed on the basis of previous research to evaluate the impact of the technology. The following three sets of metrics will be applied by the study team to the observational data that is gathered by the Site Evaluators:

Vignette 1 Experiment A and B: This set of metrics will compare and contrast existing and emerging technologies (LTE) that enable the near real-time exchange of voice, video and data communications (Annex K);

Vignette 2: This set of metrics will compare and contrast existing and emerging social media technologies that enable public alerting, warnings and notifications during large scale cross-border emergencies (Annex L); and

CAUSE Metrics: This set of metrics is intended to evaluate the overarching objectives for the CAUSE experiment series (Annex M).

The metrics-based analysis will require the study team to assign a score for each individual metric based on documented observations gathered during the experiment. The following 5-point rating scale and associated language ladder will be applied to the metrics analysis. 1 = Little knowledge about information exchange or how SA is generated or enhanced within any organizations 2 = Information is monitored and shared within an organization 3 = Information is gathered from other organizations and used to determine actions 4 = Organizations inform others about their plans for action 5 = Multiple organizations plan a coordinated response All scores assigned by the study team will be compiled and consolidated for each of the vignettes. An overall qualitative score for each of the dimensions within the Canadian Communications Interoperability Continuum Model (Vignette 1) and the SMEM Maturity Model (Vignette 2) will be calculated to represent a final scoring. These overall scores will be used to identify areas of strength and opportunities for improvement.

2.5.3 Vignettes 1 and 2 rating scales data The participant data that will be gathered using 5-point rating scales will be subjected to a descriptive analysis to determine overall responses (e.g., mean, mode, range, standard deviations) provided by the participants. A separate analysis of the ratings data will be performed on data gathered using the following tools:

Workload Scale (Vignettes 1 and 2);

SUS (Vignettes 1 and 2);

SA Scale (Vignettes 1 and 2);

Session Persistence Scale (Experiment B in Vignette 1); and,

Participant feedback Questionnaire (Daily Hotwash).




17

3. CONCLUSION This evaluation framework was designed to measure the impact of the emerging technology systems (LTE, social media) on the emergency responses depicted in the CAUSE IV experiment. This framework includes a set of data collection tools and a procedure for administering these tools prior to, during and after the CAUSE IV experiment. The qualitative tools have been developed on the basis of previous research studies and industry best practices [6][9]. The findings will be used to identify opportunities for enabling information exchanges through improved voice and data communications during cross-border emergency responses. It is hypothesized that such improvements will lead to the enhanced SA and improved resilience within communities affected by large scale disasters. Further, these findings support the national BtB policy between Canada and the US. These findings will inform the actions taken to improve the coordination of cross-border emergency responses between the partnering nations.




18

REFERENCES [1] Government of Canada. Beyond the Border: A Shared Vision for Perimeter Security

and Economic Competitiveness. February 2011. Accessible from: http://actionplan.gc.ca/en/page/bbg-tpf/beyond-border-action-plan

[2] Macdonald, S. CAUSE III Experiment Plan- Public Broadband Network, 2014.

[3] Galbraith, J. Miller, M. Li, G. CAUSE 1 – West Coast Report, 2012. Accessible from: http://cradpdf.drdc-rddc.gc.ca/PDFS/unc118/p536604_A1b.pdf

[4] CAUSE 2- CA/US Joint Report, 2013. Accessible from: http://cradpdf.drdc-rddc.gc.ca/PDFS/unc128/p537613_A1b.pdf

[5] Duncan, D. Forbes, K. Becking, I. Digital Volunteer Supported Recovery Operations Experiment After-Event Report, March 2015

[6] Forbes, K. Becking, I. Canada-U.S. Enhanced (CAUSE) Resiliency III Western

Scenario After-Event Report. March 2015

[7] Duncan, D. CAUSE IV Experiment Plan, 2016.

[8] Canadian Communications Interoperability Continuum Model. Accessible from: http://doit.maryland.gov/marylandfirst/PublishingImages/Interoperabilitycontinuum.JPG

[9] Kaminska, K., Dawe, P., Forbes, K, et al. (2015). Digital volunteer supported recovery operations experiment, Defence Research and Development Canada, Scientific Report.

[10] Kaminska, K., Dawe, P. & Rutten, B. Social media for emergency management expert roundtable workshop: Summary of findings, DRDC CSS TN 2013-046, 2013.

[11] Roscoe, A. Assessing pilot workload in flight, 1984.

[12] Accessed from : http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html

[13] Banbury, S. and Tremblay, S. (Eds.). A cognitive approach to situation awareness: Theory and application, Aldershot, U.K.: Ashgate and Town.

[14] Human Performance Research Group, NASA Ames Research Center. NASA Task Load Index Paper and Pencil Package http://humansystems.arc.nasa.gov/groups/tlx/downloads/TLX.pdf

http://www.citig.ca/what-is-interoperability.aspx

http://www.citig.ca/what-is-interoperability.aspx

http://humansystems.arc.nasa.gov/groups/tlx/downloads/TLX.pdf




19

ANNEX A. VIGNETTE 1 PRE-EXPERIMENT QUESTIONS

1. Can you explain the primary role that your organization has in the cross-border

transfer of patients and any policies/SOPs/industry standards that guide that role?

2. Who do you communicate with (voice and data) during the transfer process and why?

3. From your perspective what types of challenges/barriers exist with the transfer process?

4. What opportunities to interact with different organizations would help improve patient care or system performance?

5. How receptive would your organization be to accommodate newer technology by

changing operational procedures if it would help improve cross-border patient transfer?




20

ANNEX B. VIGNETTE 2 PRE EXPERIMENT SURVEY

Welcome Page

Welcome to the CAUSE IV Vignette 2 Pre-Experiment Questionnaire Background On April 26-28, 2016, the U.S. Department of Homeland Security Science & Technology Directorate (DHS S&T) and the Defence Research and Development Canada's Centre for Security Science (DRDC CSS) will conduct the Canada-U.S. Enhanced Resiliency Experiment (CAUSE) in Sarnia Ont. and Port Huron, MI. This experiment is the fourth in the series of bi-national experiments focused on improving the interoperability of communications and shared situational awareness of responders using new technologies. This CAUSE IV survey will gather information about your prior experience within the field of disaster response and also with the use of social media and other technologies. This information will be used to support the development of a bi-national data report. Your participation will provide valuable input to the CAUSE experiment series. Your individual responses will remain confidential. Please contact Steven Dowker by email ([email protected]) or by phone (613.241.4884 ext. 236) if you have any questions or concerns regarding this survey. This survey The survey will be available on April 18th, 2016 and will be closed prior to the start of the CAUSE IV experiment on April 26, 2016.

mailto:[email protected]




21

Part 1: Role in CAUSE IV Experiment

1. What role has been assigned to you during the CAUSE IV experiment?

a. Player with a specific role

b. Observer

c. Technical Support

d. Other (Please specify) ______________________

2. What player role has been assigned to you?

_________________________________________________

3. Where will you be located during the experiment?

a. St. Clair County EOC

b. Sarnia/Lambton County EOC

c. 211 Call Centre in Ontario

d. 211 Call Centre in Michigan

e. In the field (i.e., simulated damage assessment team)

f. State/provincial/federal site

g. Other (Please Specify) ___________________________

Part 2: Role in your organization

4. What type of organization are you representing?

a. Federal Government

b. State/provincial Government

c. County/local Government

d. Private business

e. Digital volunteer organization

f. Other (Please specify) _________________________




22

5. What federal organization will you be representing? ___________________

6. What state / provincial organization are you representing _______________

7. What county / local government are you representing __________________

8. What private organization were you representing?_____________________

9. What digital volunteer organization were you representing?______________

10. What is your role within your organization?

a. Supervisor

b. Operational role

c. Tactical role

d. Strategic role

e. Public relations/community engagement

f. Technical Support

g. Volunteer

h. Other (Please specify)_______________

Part 3: Information sharing during emergencies

11. Does your organization develop plans to respond to cross-border emergency

events?

a. Yes

b. No

c. Don’t know

12. Has your organization defined cross-border priority information sharing

requirements? Check all that apply.

a. For normal operations (i.e., steady state)?

b. In times of emergency?

13. Does your organization actively share some kinds of information across the

border?




23

a. Yes

b. No

c. Don’t know

14. What information does your organization share during cross-border events?

Please list.

_________________________________________________

15. When does your organization provide assistance to other organizations or

citizens? Check all that apply.

a. Preparedness activities before an event occurs

b. Immediate response to an event

c. Long-term recovery from an event

16. What kinds of information would you categorize as essential to complete the

duties of your job? Check all that apply.

a. Transportation Status

b. Weather

c. Utilities (e.g., electricity, gas)

d. Mutual aid resources (e.g., fire, EMS, law enforcement, equipment)

e. Mass care (e.g., shelters, hospital status)

f. Communication status

g. Other (please specify)___________________________________

Part 4: Social media use within organization

17. Which social media platforms does your organization use during planning or

responding to emergency events? Check all that apply.

a. Facebook

b. Twitter

c. Instagram




24

d. Google +

e. LinkedIn

f. Skype

g. YouTube

h. Other____________(Please specify)

i. My organization doesn’t’ use any of these technologies

j. Don’t know

18. In your experience, what is the biggest barrier for using social media to support

disaster response operations? Check all that apply.

a. Lack of awareness of social media

b. Lack of acceptance of social media

c. Lack of policy and processes related to the use of social media

d. Accessibility and availability of technology

e. Training for disaster response officials and volunteers

f. Lack of available personnel to assist

g. Other: __________________________________________

Part 5: Technology use within organization

19. Does your organization currently use any of the following kinds of technology

when planning or responding to emergency events? Check all that apply.

a. Alerting and Warning systems (e.g., IPAWS, NAADS)

b. Mapping programs (e.g., ArcGIS)

c. Active Operating Picture tool (e.g., Latitude Geographic)

d. Local Alert/Warning System (e.g., MyCNN, WINS)

e. Other (please specify)__________

f. Don’t know

20. How is the technology currently used by your organization during an emergency

response operation? Check all that apply.




25

a. To inform or alert the public

b. To monitor public information

c. To identify public safety issues and vulnerable citizens

d. To respond to public’s questions, concerns and requests

e. To locate emergency response resources

f. To gather information that will support decision making

g. To assess damage in affect areas

h. To link the visualization of plans with operations

i. To send alerts and notifications to cross-border messaging and alerting

systems

j. To support the request and acquisition of mutual aid during response

operations

k. Other (Please specify) ____________________________

l. None of the above

m. Don’t know

21. Do you have any experience using technologies in the following ways within your

organization?

a. Operation of these systems

b. Producing messaging or content for distribution via these systems

c. Viewing or receiving information using these systems

d. Providing technical support for these systems

e. Other (Please specify) _______________________

f. I have no experience with technologies

22. Based on your current experience and knowledge, to what extent do you think

the use of interoperable technologies could improve your organizations'

emergency planning and response capabilities?

a. Not at all

b. A little bit




26

c. Somewhat

d. A lot

e. A great deal

Part 5: Experience with digital volunteer organizations

23. Has your organization already partnered with a digital volunteer organization

during an emergency?

a. Yes

b. No

c. Don't know

24. What type of support did digital volunteer organization(s) provide? Check all that

apply.

a. Built maps

b. Inform traditional disaster response organizations and/or the public

c. Generated situation reports for the traditional disaster response

organizations

d. Gathered data

e. Performed data analyses

f. Don't know

g. Other: ______________

25. What tools did the digital volunteers use when providing support to your

organization? Check all that apply.

a. Facebook

b. Twitter

c. Instagram

d. Google+

e. LinkedIn

f. Skype




27

g. YouTube

h. Hootsuite

i. Other (Please specify) _______________________

j. Don’t Know

26. What would enhance the abilities of digital volunteer organizations when

providing support to an emergency response? Check all that apply.

a. Access to more technologies/applications

b. Increased social media training for volunteers

c. Increased technology training for volunteers

d. Increased interoperability between the existing systems and technologies

e. Don't know

f. Other (Please specify)________________________

Part 6: Current knowledge level

27. Please read each item and select a rating that reflects how knowledgeable you

are of interoperable technologies as a method of supporting response

operations.

A score of ‘1’ indicates ‘not at all knowledgeable’, a score of ‘3’ indicates ‘somewhat knowledgeable’ and a score of ‘5’ indicates ‘extremely knowledgeable’.




28

5-Point

Rating Scale

N/A

# Statement

Your understanding of HOW technology is used….

1 2 3 4 5 6

1 ...to support your organizational planning efforts.

2 ...to visualize and share resource plans, capability gaps, and assign

resource partners.

3 ...to link the visualization of plans with the visualization of operations.

4 ...to monitor steady state status of operations in a cross-border

community.

5 ...to receive and amplify alerts originating from National Weather

Service or Environment Canada.

6 …to send alerts and notifications to the public about operations.

7 ...to use predictive analytics to precisely inform decisions such as

which sirens to turn on in the event of an impending storm.

8 ...to monitor vulnerable citizen calls for service, and aid in decision

making to support the needs of vulnerable populations.

9 ….to provide situational awareness that may inform timeline of

preventive / protective actions.

10 ...to send alerts and notifications to cross-border messaging and

alerting systems

11 ...to enable cross-border communities of practice to jointly plan and

provide situational awareness.

12 …by digital volunteer organizations to support disaster response

operations.

13 …to gather information to assess damage in affected areas (e.g.,

storm damage).

14 ...to help public officials triage and prioritize immediate response

actions.




29

15 ...to help public officials streamline processes for monitoring calls for

service and quickly identify areas of need.

16 …to help the public locate and replenish needed supplies.

17 ...to help public officials determine the where to obtain mutual

assistance.

18 …to spread the updates from authorities to a wider audience.

19 …to provide updates on the status of pre-determined essential

elements of information

20 ...to speed the process of damage assessments and time to

response.

21 ...to improve information access and remove bottle necks in

communications.

22 ...to harmonize emergency management processes amongst

disparate organizations in multiple jurisdictions.

23 …to support the request and acquisition of mutual aid during

response operations.

Thank-you for participating in the CAUSE IV pre-experiment survey for Vignette 2.

This information will be used to support the development of a bi-national data report for

CAUSE IV.

Your participation will provide valuable input to the CAUSE experiment series. Your

responses will remain confidential.

Please contact Steven Dowker by email ([email protected]) or by phone (613.241.4884

ext. 236) if you have any questions or concerns regarding this survey.




30

ANNEX C. VIGNETTE 2 POST-EXPERIMENT QUESTIONNAIRE

Post-Experiment Survey:

Welcome Page Welcome to Vignette 2 Post-Experiment Questionnaire Background The U.S. Department of Homeland Security Science & Technology Directorate (DHS S&T) and the Defence Research and Development Canada's Centre for Security Science (DRDC CSS) ran the Canada-U.S. Enhanced Resiliency Experiment (CAUSE) in Sarnia Ont. and Port Huron, MI. This experiment took place April 26-28, 2016 and was the fourth, referred to as CAUSE IV, in a series of bi-national experiments focused on improving the interoperability of communications and shared situational awareness of responders using new technologies. This CAUSE IV post-experiment survey will gather information about your experience participating in the experiment. This information will be used to support the development of the After Action Report (AAR). Your participation will provide valuable input to the CAUSE experiment series. Your responses will be confidential and only a summary of all the feedback will be presented in the final AAR. Please contact Steven Dowker by email ([email protected]) or by phone (613.241.4884 ext. 236) if you have any questions or concerns regarding this questionnaire. This survey The survey will be available when the experiment ends on April 28th, 2016 and is planned to remain open until May 5th, 2016.

mailto:[email protected]




31

Part One: Experience participating in the experiment. 1. What was your role during the CAUSE IV experiment?

___________________________________________________________________

2. What Player role was assigned to you?

___________________________________________________________________

3. Where were you located during the experiment?

a. St. Clair County EOC

b. Sarnia/Lambton County EOC

c. 211 Call Centre in Ontario

d. 211 Call Centre in Michigan

e. In the field (i.e., simulated damage assessment team)

f. State/provincial/federal site


4. What type of organization were you representing?

a. Federal Government

b. State/provincial Government

c. County/local Government

d. Private Business

e. Digital Volunteer Organization

f. Other (Please specify) _________________________

5. What Federal organization were you representing? _________________

6. What Provincial / State organization were you representing? __________

7. What County/ Local organization were you representing? _____________

8. What Private organization were you representing?____________________

9. What Digital Volunteer organization were you representing?_____________




32

10. What is your role within the organization that you are representing?

a. Supervisor

b. Operational role

c. Tactical

d. Strategic

e. Public Relations/ Community Engagement

f. Technical Support (eg. IT support)

g. Volunteer

h. Other (please specify)

Part Two: Integrating interoperable technologies into disaster response operations

11. Please indicate which of the following technologies and capabilities that you learned

about during the CAUSE IV experiment. Check all that apply.

a. 211 Michigan services

b. 211 Ontario services

c. Latitude Geographic's Active Operating Picture (AOP) tool

d. ArcGIS Online

e. Everbridge notification platform (MyCNN, WINS)

f. Mutual Aid Planner app

g. Facebook and Twitter groups

h. NAADS and IPAWS bridge

i. Not applicable

j. Other (Please specify)______________

12. Please indicate whether the technologies and services used in CAUSE IV were

valuable for addressing the main activities during the disaster response:

a. Planning

b. Early Warning

c. Reporting

d. Local Response

e. Cross Border Response




33

13. What types of information were essential (i.e., Essential Elements of Information- EEIs) to complete the duties of your job? Check all that apply

a. Transportation Status

b. Weather

c. Utilities (eg. electricity, gas)

d. Mutual aid resources (e.g., fire, EMS, law enforcement, equipment)

e. Mass care (e.g., shelters, hospital status)

f. Communication status

g. Other (please specify)____________________

14. How did the information provided by the CAUSE IV technologies improve the Essential Elements of Information during the experiment?

__________________________________________________________________

15. How did the type of information provided by the CAUSE IV technologies compare to

the information that is delivered through your current technologies?

__________________________________________________________________

16. Are there any lessons learned that can be transitioned within your own organization?

(ex. new technology, applications, uses for technology).

a. Yes

b. No

c. Don’t know

17. Please identify any lessons or technologies that you think could be transitioned to

your organization.

__________________________________________________________________

18. During the experiment, which of the following ways did you use or observe alerting

and warning systems being used? Check all that apply.

a. Operation of these systems

b. Producing messaging or content to be distributed through these systems

c. Viewing or receiving information using these systems

d. Providing technical support for these systems




34

e. No experience

f. Don't know


19. To what extent has your knowledge of alerting and warning systems changed as a

result of participating in the CAUSE IV experiment.

a. No change

b. Slightly

c. Some

d. A lot

e. A great deal

20. Did your participation in the CAUSE IV experiment inform you about any of the

following ways that Alerting and Warning systems can be used? Check all that apply.

a. How the Alerting and Warning systems work

b. What types of information Alerting and Warning systems can provide

c. How to improve the effectiveness of the information that is released

d. How to use Alerting and Warning systems to coordinate cross-border

activities

e. How to compile information from multiple sources into a usable format

f. How Alerting and Warning systems can provide information that initiates

operational action

g. How to reach a larger audience through the messaging capability

h. Other (Please specify)________________

21. Please select any of the positive impacts that you think these new technologies could

have on the response effort today or in the future.

a. Increased quality of information being collected

b. Increased quality of information released to the public

c. Better flow of information within your organization

d. Better flow of information between all organizations

e. Improved timeliness of response coordination efforts




35

f. More effective and informed decision making

g. Higher level of situation awareness

h. Don't know

i. Other (Please specify)_______________

22. Did any of the new technologies within the experiment have a negative impact on

your response efforts.

a. Yes

b. No

c. Don't know

23. Please indicate what was negatively impacted by the technologies during the

experiment.

a. Quality of information being collected

b. Quality of information released to the public

c. Flow of information within your organization

d. Flow of information between all the technologies

e. Timeliness of response coordination efforts

f. Decision making

g. Don't know

h. Other (Please specify)_______________




36

Part Three: Evaluation of Technology Sessions

A series of bi-weekly orientation sessions was held prior to the CAUSE experiment. These sessions were intended to familiarize the participants with the technologies that will be used during the experiment. We would like to gather some feedback about how valuable this familiarization was to your participation in the experiment.

24. Please rate the following learning sessions using the rating scale provided. If you did

not attend some or all of these sessions please select "not applicable".

1- Not at all valuable 2- Somewhat valuable 3- Valuable 4- Very valuable 5- Extremely valuable 6- Not applicable

# Technology Session

5-Point Rating Scale

1 2 3 4 5 6

1 Cross Border Situation Awareness

2 Alerts, Warnings and Notifications

3 2-1-1 Reporting

4 Digital Volunteer Reporting

5 Local Officials Reporting

6 Requests for Assistance and Mutual Aid

25. Please list any advantages you experienced as a result of participating in the

technology sessions.

26. Please list any of the knowledge that you can apply within your own organization.




37

Part Four: Knowledge of how interoperable technology is used to Plan and Deliver a Disaster Response 27. Please read each statement and use the rating scale to indicate the extent to which

your knowledge changed as a result of participating in CAUSE IV. A score of ‘1’

indicates ‘I didn’t learn anything new', a score of ‘3’ indicates ‘Somewhat more

knowledgeable’ and a score of ‘5’ indicates ‘I am significantly more knowledgeable’.


# Statement

Your knowledge of HOW interoperable technology can be used….

1 2 3 4 5

1 ...to support your organizational planning efforts.

2 ...to visualize and share resource plans, capability gaps, and assign resource partners.

3 ...to monitor steady state status of operations in a cross-border community.

4 ...to receive and amplify alerts originating from the National Weather Service or Environment Canada.

5 …to send alerts and notifications to the public about operations.

6 ...to use predictive analytics to precisely inform decisions such as which sirens to turn on in the event of an impending storm.

7 ...to monitor vulnerable citizen calls for service, and aid in decision making to support the needs of vulnerable populations.

8 ...to send alerts and notifications to cross-border messaging and alerting systems

9 ...to enable cross-border Communities of Practice to jointly plan and provide situational awareness.

10 …to engage digital volunteers to support disaster response operations.

11 ...to speed the process of damage assessments and time to response.

12 ...to help public officials determine where to request and obtain mutual assistance.

13 …to spread the updates from authorities to a wider audience.

14 …to provide updates on the status of pre-determined Essential Elements of Information

15 ...to harmonize emergency management processes amongst diverse organizations in multiple jurisdictions.

Part Five: Impact of using interoperable technology on the quality of information exchanged

28. Each statement below is related to the impact of interoperable technologies on the

information that was exchanged during the disaster response. Please use the 5-point




38

rating scale to indicate the extent to which you agree or disagree with each

statement. A rating of ‘1’ refers to ‘Strongly Disagree’ and a rating of ‘5’ refers to

‘Strongly Agree’. A rating of ‘3’ refers to a ‘Neutral’ response.


# Statement The use of interoperable technologies provided higher quality information about...

1 2 3 4 5

1 …location of deployed response crews.

2 …areas with the greatest level of infrastructure damage.

3 …areas with the greatest level of need for response crews.

4 …areas at greatest risk for further damage due to the disaster.

5 …public's response to the disaster.

6 ...targeted alert messages and protective actions

7 ...available resources to support mutual assistance requests

8 ...local reports from 211, 911, and digital volunteers

9 ...when and how to use various situational awareness, alerting and reporting tools




39

ANNEX D. OBSERVATION PACKAGE This Observation Package provides an example of the data collection tool that will be used by the evaluators to gather observations at their assigned venue. Each expected inject in the experiment will be associated with a unique data collection table in the Observation Package.

Vignette: 1 Experiment: A

Venue: Evaluator Name:

Date: Time:

Situation:

Notes:




40

ANNEX E. SA AND WORKLOAD RATING SCALES

E.1 EXPERIMENTS A AND B FOR VIGNETTE 1

Vignette 1 Situation Awareness

Experiment : A

Venue: ______________________________________

Player Role: _________________________________

When requested by your Site Evaluator please provide a rating that indicates your level of situation awareness during

each of the main activities listed below. The events associated with each of the main activities are presented in

a bulleted list.

Only one rating is requested for each main activity.

Please make any comments which can help the evaluators understand your reason for assigning the rating.

The term situation awareness refers to your ability to identify and understand the information that you need to complete your task and anticipate what will occur in the

future. This will be unique for each player role.

Main Activities and Events

Rating (Please circle):

Main Activity: Transit to hospital: Events:

CACC dispatches ambulance

1 2 3 4 5

Notes:




41

Main Activity: Initiate/prepare for transfer

Events:

Paramedics arrive at hospital and communicate arrival

Paramedics assist with CBSA Emergency Medical Transfer form

Paramedics load patient

1 2 3 4 5

Notes:

Main Activity: During transfer on CDN Side

Events:

Paramedic en route from hospital

Paramedics provide bridge with ETA

Paramedics inform CBP of impending arrival

Paramedics call US hospital with ETA and patient update

Paramedics provide CDN Dispatch with patient update

1 2 3 4 5

Notes:

Main Activity: Transiting CDN-US border: Events:

Arrive at border and proceed to transit

Automatic transfer from CDN to US LTE

1 2 3 4 5

Notes:

Main Activity: During transfer on US side Events:

Paramedic provide ETA to US dispatch

Paramedic provide US Hospital with ETA, 12 lead ECG

Paramedic provide US dispatch with patient update

1 2 3 4 5

Notes:




42

Main Activity: Arrival at US hospital Events:

Paramedic to notify US dispatch of arrival

Paramedic upload patient care record and 12 lead ECG

1 2 3 4 5

Notes:




43

Situation Awareness Vignette 1B


Experiment : B

Venue: ______________________________________

Player Role: _________________________________


each of the main activities listed below. The events associated with each of the main activities are presented in a

bulleted list.







Main Activity: Start of communications

Events:

US Paramedics confirm connection to US PSBN network

1 2 3 4 5

Notes:




44

Main Activity: Communications prior to network switch

Events:

US paramedics call US dispatch en route to bridge

US paramedics, US +CDN Hospitals have 3-way videoconference

US paramedic arrive at border + be processed

1 2 3 4 5

Notes:

Main Activity: Communications immediately after network switch Events:

Automatic transfer from US to CDN LTE

US paramedic, US hospital and CDN hospital continue 3 way videoconference

1 2 3 4 5

Notes:

Main Activity: Upon arrival to location Events:

Final sign-off

1 2 3 4 5

Notes:




45

Vignette 1 Workload

Experiment : A

Venue: ______________________________________

Player Role: _________________________________

When requested by your Site Evaluator please provide a rating that indicates the workload that you are experiencing during each of the main activities listed below. The events associated with each of the main activities are presented in

a bulleted list.



The term workload refers to the demands that are placed upon you during the completion of your tasks. There are several types of demands (cognitive, physical, effort, frustation, etc.) that may influence your workload. This will be unique for each player role.



Main Activity: Transit to Hospital. Event:

CACC dispatches ambulance

1 2 3 4 5

Notes:




46

Main Activity: Initiate/prepare for transfer

Events:

Paramedics arrive at hospital and communicate arrival

Paramedics assist with CBSA Emergency Medical Transfer form

Paramedics load patient

1 2 3 4 5

Notes:

Main Activity: During transfer on CDN Side

Events:

Paramedic en route from Hospital

Paramedics provide bridge with ETA

Paramedics inform CBP of impending arrival

Paramedics call US hospital with ETA and patient update

Paramedics provide CDN Dispatch with patient update

1 2 3 4 5

Notes:

Transiting CDN-US border: Events:

Arrive at border and proceed to transit

Automatic transfer from CDN to US LTE

1 2 3 4 5

Notes:

Main Activity: During transfer on US side Events:

Paramedic provide ETA to US dispatch

Paramedic provide US Hospital with ETA, 12 lead ECG

Paramedic provide US dispatch with patient update

1 2 3 4 5

Notes:




47

Main Activity: Arrival at US hospital Events:

Paramedic to notify US dispatch of arrival

Paramedic upload patient care record and 12 lead ECG

1 2 3 4 5

Notes:




48

Vignette 1 Workload

Experiment : B

Venue: ______________________________________

Player Role: _________________________________

When requested by your Site Evaluator please provide a rating that indicates the workload that you are experiencing during each of the main activities listed below. The events

associated with each of the main activities are presented in a bulleted list.



The term workload refers to the demands that are placed upon you during the completion of your tasks. There are several types of demands (cognitive, physical, effort, frustation, etc.) that may influence your workload. This will be unique for each player role.



Main Activity: Start of communications

Events:

US Paramedics confirm connection to US PSBN network

1 2 3 4 5

Notes:




49

Main Activity: Communications prior to network switch

Events:

US paramedics call US dispatch en route to bridge

US paramedics, US +CDN Hospitals have 3-way videoconference

US paramedic arrive at border + be processed

1 2 3 4 5

Notes:

Main Activity: Communications immediately after network switch Events:

Automatic transfer from US to CDN LTE

US paramedic, US hospital and CDN hospital continue 3 way videoconference

1 2 3 4 5

Notes:

Main Activity: Upon arrival to location Events:

Final sign-off

1 2 3 4 5

Notes:




50

E.2 VIGNETTE 2


Venue: ______________________________________

Player Role: _________________________________


each of the main activities listed below. The events associated with each of the main activities are presented in

a bulleted list.







Main Activity: Planning Events:

Develop and share plans in the Mutual Aid Planner app; integrate results into operational GIS

St. Clair County Hazard Plan

Cross-Border Resource Allocation and Mutual Aid Plan

Monitoring steady state community Essential Elements of Information

1 2 3 4 5

Notes:




51

Main Activity: Early Warning

Events:

NWS and Environment Canada Severe weather alert to citizens

Local targeted alerts

Vulnerable citizen request detection

Cross-border warnings at the local level

Cross-border warnings at the national level

Monitoring public safety EEIs

Non-automated notification after chemical spill

1 2 3 4 5

Notes:

Main Activity: Reporting

Events:

Citizens call 911 dispatch

Citizens call 2-1-1 dispatch

Citizens share sentiments and report damage via social media

Analysis generates prioritized list of damage assessment location

Citizen critically injured, requires ambulance transport to specialized hospital


1 2 3 4 5

Notes:




52

Main Activity: Local Response Events:

Lambton EMS responds to call for service

Tri-County Hospital Ambulance responds to call for service

St. Clair County sends damage assessment team

Request for assistance originating from Marysville

St. Clair County request for assistance

Sarnia request for assistance

Sarnia EOC request for assistance

1 2 3 4 5

Notes:

Main Activity: Cross Border Response Events:

NEMAC request from Canada to U.S. for Digital Volunteer MRP

NEMAC request from U.S. to Canada for Hazmat MRP

Cross-border situation report to national stakeholders

1 2 3 4 5

Notes:




53

Vignette 2 Workload

Venue: ______________________________________

Player Role: _________________________________

When requested by your Site Evaluator please provide a rating that indicates the workload that you are experiencing during each of the main activities listed below. The events associated with each of the main activities are presented in

a bulleted list.



The term workload refers to the demands that are placed upon you during the completion of your tasks. There are

several types of demands (cognitive, physical, effort, frustation, etc.) that may influence your workload. This will

be unique for each player role.



Main Activity: Planning Events:

Develop and share plans in the Mutual Aid Planner app; integrate results into operational GIS

St. Clair County Hazard Plan

Cross-Border Resource Allocation and Mutual Aid Plan

Monitoring steady state community Essential Elements of Information

1 2 3 4 5

Notes:




54

Main Activity: Early Warning

Events:

NWS and Environment Canada Severe weather alert to citizens

Local targeted alerts

Vulnerable citizen request detection

Cross-border warnings at the local level

Cross-border warnings at the national level

Monitoring public safety EEIs

Non-automated notification after chemical spill

1 2 3 4 5

Notes:

Main Activity: Reporting

Events:

Citizens call 911 dispatch

Citizens call 2-1-1 dispatch

Citizens share sentiments and report damage via social media

Analysis generates prioritized list of damage assessment location



1 2 3 4 5

Notes:




55

Main Activity: Local Response Events:

Lambton EMS responds to call for service

Tri-County Hospital Ambulance responds to call for service

St. Clair County sends damage assessment team

Request for assistance originating from Marysville

St. Clair County request for assistance

Sarnia request for assistance

Sarnia EOC request for assistance

1 2 3 4 5

Notes:

Main Activity: Cross Border Response Events:

NEMAC request from Canada to U.S. for Digital Volunteer MRP

NEMAC request from U.S. to Canada for Hazmat MRP

Cross-border situation report to national stakeholders

1 2 3 4 5

Notes:




56

ANNEX F. COMMUNICATIONS AND INFORMATION SHARING (CIS)

Vignette 1 Communications and Information Sharing (CIS) – Technical Perspective

Experiment : A and B

Venue: ______________________________________

Experiment Support Role: _________________________________

Provide a rating to indicate your level of agreement with each of the statements listed below.

Key Observations Rating (Please circle):

The experiment proceeded as expected.

1 2 3 4 5

Notes:




57

There was a low level of technical issues during experiment preparation.

1 2 3 4 5

Notes:

There was a low level of technical issues during actual experiment.

1 2 3 4 5

Notes:

There was an ability to resolve technical issues quickly.

1 2 3 4 5

Notes:




58

The applications performed as expected.

1 2 3 4 5

Notes:

There was an ability to address known technical gaps; achieve technical breakthroughs.

1 2 3 4 5

Notes:

New technical gaps were identified as a result of the experiment.

1 2 3 4 5

Notes:




59

ANNEX G. DAILY HOTWASH QUESTIONS The following questions will be presented at the daily hotwash sessions that occur at the end of the experiment day.

1. What are the main experiences/observations you made today?

2. Did you gain any new information or experience that helped you with your role throughout the day?

a. Are there key positive experiences you gained?

b. Were there any challenges you faced today? (technology, information that you needed?)

3. Do you have any suggestions for the next experiment day? (logistics, process,

people, etc




60

ANNEX H. SYSTEM USABILITY SCALE The System Usability Scale (SUS) provides a quick tool for measuring the usability. It consists of a 10 item questionnaire and is accompanied by a 5-point rating scale. Participants will provide a rating to indicate the extent to which they agree with each of the statements. A rating of ‘1’ refers to ‘Strongly Disagree’ and a rating of ‘5’ refers to ‘Strongly Agree. A rating of ‘3’ refers to a ‘Neutral’ response.

Vignette 1 (Experiment A or B) or Vignette 2 (Please circle one)

Venue: ______________________________________

Player Role: _________________________________

Provide a rating to indicate your level of agreement with each of the statements listed

below.

No. 1 2 3 4 5 Notes

1 I would like to use this system frequently.

2 I found the system unnecessarily complex.

3 I thought the system was easy to use.

4

I would need the support of a technical person to be able to use this system.




61

5

I found the various functions in this system were well integrated.

6

I thought there was too much inconsistency in this system.

7

I would imagine that most people would learn to use this system very quickly.

8 I found the system very cumbersome to use.

9 I am confident using the system.

10

I needed to learn a lot of things before I could get going with this system.




62

ANNEX I. PARTICIPANT FEEDBACK QUESTIONNAIRE This participant feedback questionnaire will be administered to AAR participants at the end of the CAUSE IV experiment. The intent of this tool is to gather feedback concerning the planning, preparation and execution of the CAUSE experiment. Player Feedback Survey Date ________________________ Location during experiment ______________________________ What role was assigned to you during the CAUSE IV experiment?

Player with a specific role ____________________________

Evaluator _________________________________

Observer _________________________________

Other ___________________________________ 1. How would you rate the organization and preparation prior to the experiment? (circle

most appropriate)

A. Not effective B. Somewhat effective C. Met expectations D. Exceeded expectations E. Exceptional

2. How effective was the training provided by the facilitators for your role in the experiment? (circle most appropriate)


3. From your perspective, how would you rate the execution of the experiment? (circle most appropriate)





63

4. How well equipped was your location to participate in the experiment? (circle most appropriate)


5. How effective was the training and familiarization you received to prepare you to

participate in the experiment? (circle most appropriate)


6. How effective were the daily hotwash sessions at identifying key observations from

experiment conduct? (circle most appropriate) A. Not effective B. Somewhat effective C. Met expectations D. Exceeded expectations E. Exceptional

7. What are the main points of learning that you would like to see carried forward to

your organization?

8. Please identify any issues regarding the experiment design and conduct that could

be considered for future CAUSE experiments.




64

ANNEX J. FINAL AAR QUESTIONS The following questions will be presented at the AAR hotwash session that occurs at the end of the CAUSE IV experiment.

1. How did your understanding of your role evolve throughout the experiment?

2. Were current processes improved as a result of the technologies involved in CAUSE IV?

3. If you faced challenges with the technology used on CAUSE IV, what were they

and how did you mitigate them? How could improvements be made (especially with regard to voice, data and video communications)?

4. Based on your experience with CAUSE IV, what aspects of the technology would

assist your organization with the execution of your tasks?

5. Did you experience any delays in your normal work processes as a result of the new technologies? If yes, were these delays because of your unfamiliarity with the technology or from gaps in the technology itself?

6. Was there anything that did not meet your expectations? Why?

7. Would you like to add any final comments about your experience participating in

this experiment?




65

ANNEX K. VIGNETTE 1 – METRICS The following set of metrics will be used to evaluate the effectiveness of an interoperable LTE system to facilitate cross-border information exchange during the transport of a medical patient from a Canadian to a US hospital.

Item Resiliency Indicator

Governance

Vig1-G1 Critical information is shared between multiple stakeholder organizations (including within province and state and across national borders).

Vig1-G2 Leadership is accessible and can exchange information from multiple points via technology to exchange information.

Vig1-G3 Assessment and prioritization of response needs is performed and the output is shared.

Vig1-G4 All relevant Points of Contact to be included in the information sharing framework have been identified.

Vig1-G5 Provide feedback from information-gathering entities to the authorities to support decision-making.

Vig1-G6 Pre-identified mechanisms (MOUs, MAAs) to request assistance from other cities, counties and levels of government are in place.

Vig1-G7 Stakeholders can report indicators/warnings and information to lead authorities.

Standard Operating Procedures (SOPs)

Vig1-S1 Clearly defined and documented mechanisms/processes/formats for sharing information

Vig1-S2 Steps required to transmit and receive information via technology are known and can be described by operators.

Vig1-S3 Information sources are obvious so that clarification is not required.

Vig1-S4 Processes for requesting information from other organizations are in place.

Vig1-S5 Requests for initiating processes (patient-transfer) can be communicated with technology.

Vig1-S6 Communication protocols can be accommodated via technology

Vig1-S7 Technology permits multiple stakeholders to contribute information to status reports.




66

Vig1-S8 Processes or procedures can be used to disseminate information and products between organizations.

Vig1-S9 Establish and maintain communications with all responder organizations involved in the incident.

Vig1-S10 Establishment of leading authority is communicated to all organizations and changes are clearly identified.

Technology

Vig1-T1 Feedback and/or follow-up on information can be provided through the use of tools.

Vig1-T2 Information exchanged through the use of technology reflected the current status of the emergency.

Vig1-T3 Disseminate relevant information to other stakeholders in a usable and expected format.

Vig1-T4 Personnel understand how to use the technology to communicate with other responders.

Vig1-T5 Responders are able to send and receive data needed to satisfy information requirements.

Vig1-T6 Amount of information sharing between emergency response organizations can be adjusted to accommodate dynamic response and recovery efforts.

Vig1-T7 Technology supports the exchange of useful/actionable information between organizations.

Vig1-T8 Technology supports the enhancement of SA at the unit level.

Vig1-T9 Technology enhances the timeliness for information sharing and supports decision making.

Training & Exercises

Vig1-TEx1 Information exchange is enabled through the coordinated use of multiple technologies.

Vig1-TEx2 Evaluate, revise and prioritize tactics to meet incident developments

Vig1-TEx3 Information exchange reflects the time critical nature of emergency (i.e., patient transfer, public safety issue)

Vig1-TEx4 Existing technology is interoperable with emerging technology

Vig1-TEx5 Processes for exchanging information are tested and refined.

Vig1-TEx6 Communications plans include all relevant data and voice communications that are available for the exercise.

Vig1-TEx7 Terminology is understood by senders and receivers




67

Vig1-TEx8 Information exchanges enhance ability to quickly detect changes in success of the response.

Vig1-TEx9 Unit with new information provides initial on-scene report to appropriate organization(s).

Usage

Vig1-U1 Receipt of information is acknowledged using the same technology through which the information was received.

Vig1-U2 Plans address establishing key transportation avenues (e.g. best routes for personnel and equipment to arrive at location, etc.).

Vig1-U3 Develop a common operating picture (COP) for ongoing status of operation.

Vig1-U4 Deploy and transport resources to appropriate, pre-determined locations.

Vig1-U5 Issue corrective messages using the same technology when errors are recognized.

Vig1-U6 Horizontal information exchange requirements can be met.

Vig1-U7 Vertical information exchange requirements can be met.

Vig1-U8 Information bottlenecks created by communications process OR technology can be identified by organizations

Vig1-U9 Request for information can be issued and understood in a single transmission




68

ANNEX L. VIGNETTE 2 – METRICS The following set of metrics will be employed to help evaluate the reach and effectiveness of digital volunteers in assisting local Emergency Managers respond to a disaster event within their jurisdiction.


People

Vig2-P1 Establish methods to identify and mobilize support for gathering information relevant to recovery operations.

Vig2-P2 Educate the public on what they should expect during the recovery phase.

Vig2-P3 Evaluate compliance of citizens to follow directions during the recovery operations.

Vig2-P4 Identify citizen's willingness and availability to provide support.

Vig2-P5 Evaluate citizen preparedness and participation to provide support.

Vig2-P6 Recognize exemplary citizens in recovery programs.

Vig2-P7 Identify indicators that the information provided by the public is monitored.

Vig2-P8 Develop and sustain volunteer opportunities for citizens to support local emergency responders and community safety efforts.

Vig2-P9 Public's information needs are clearly defined and understood based on the monitoring of social media.

Vig2-P10 Provide information to volunteers who want to register and provide help during the recovery operation.

Vig2-P11 Coordinate voluntary support/activities with community leadership and liaise with local agencies.

Vig2-P12 Match individual volunteers with requests from agencies.

Vig2-P13 Identify public information needs and media types and utilization with the affected area.

Vig2-P14 Detect and analyze trends in social media data that are posted by the networked public.

Vig2-P15 Detect and analyze trends in social media data that are posted by the networked public.

Vig2-P16 Public receives messages (information and requests) from authorities and humanitarian organizations.




69

Governance

Vig2-G1 Establish methods to identify and mobilize support for gathering information relevant to recovery operations.

Vig2-G2 Educate the public on what they should expect during the recovery phase.

Vig2-G3 Evaluate compliance of citizens to follow directions during the recovery operations.

Vig2-G4 Identify citizen's willingness and availability to provide support.

Vig2-G5 Evaluate citizen preparedness and participation to provide support.

Vig2-G7 Identify indicators that the information provided by the public is monitored.

Vig2-G8 Develop and sustain volunteer opportunities for citizens to support local emergency responders and community safety efforts.

Vig2-G9 Public's information needs are clearly defined and understood based on the monitoring of social media.

Vig2-G10 Provide information to volunteers who want to register and provide help during the recovery operation.

Vig2-G11 Coordinate voluntary support/activities with community leadership and liaise with local agencies.

Vig2-G12 Pre-identify resources available to supplement command and control capabilities through partnerships.

Vig2-G13 Identify public information needs and media types and utilization with the affected area.

Vig2-G14 Detect and analyze trends in social media data that are posted by the networked public.

Vig2-G15 Detect and analyze trends in social media data that are posted by the networked public.

Vig2-G16 Public receives messages (information and requests) from authorities and humanitarian organizations.

Vig2-G17 Plan and coordinate warnings, instructions, and information updates through collaboration with partners.

Vig2-G18 Disseminate information to a wider audience at the request of authorities and in accordance with governance processes.

Implementation

Vig2-I1 Report and document the incident by completing and submitting required forms, reports, documentation, and follow-up notation.

Vig2-I2 Provide geo-coded status report of community, homes and facilities that are relevant (e.g., safe or unsafe to re-enter and re-occupy) and are best described graphically.




70

Vig2-I3 Plans address establishing key transportation avenues (e.g. best routes for personnel and equipment to access disaster locations, etc.) to avoid or to use for certain parts of the recovery effort.

Vig2-I4 Develop a common operating picture (COP) for ongoing status of recovery operations for stakeholder reference and review.

Vig2-I5 Implement a process to evaluate citizen-related component to ensure alerts and warnings, public education, emergency public information (e.g., evacuations, mass care) are effective.

Vig2-I6 Implement a process to track numbers of volunteers that would participate in ongoing volunteer programs and surge capacity roles (e.g., logs to know which volunteers to contact for physical or digital volunteer tasks)

Vig2-I7 Provide evacuation assistance in response to public requests for assistance.

Vig2-I8 Evaluate, revise and prioritize tactics to meet incident developments.

Vig2-I9 Provide transportation for resources to appropriate, pre-determined locations.

Vig2-I10 Develop outreach plan to educate the community about the functions of the volunteers (e.g., trust building to encourage citizens to provide information concerning requests and updates to the volunteer organizations and free up emergency organizations)

Vig2-I11 Develop and distribute public education materials on evacuation preparation, plans, and procedures (e.g., neighbourhoods, hospital, nursing homes, special events venues, animal shelters, transient populations, tourists, non-English speaking).

Vig2-I12 Participate in citizen preparedness activities to ensure that public information on evacuation preparation and processes is effectively communicated.

Vig2-I13 Develop a community relations plan for ensuring continued communications with citizens.

Vig2-I14 Issue corrective messages when errors are recognized in previous public announcements.

Vig2-I15 Develop guidelines for addressing public safety issues identified through analysis/monitoring of social media data.

Technology

Vig2-T1 Monitor social media applications to gather information from the public (e.g., storm damage, public awareness, public sentiment, etc.).

Vig2-T2 Plans (e.g., structural damage assessments) include street maps for determining alternate routes and positioning.

Vig2-T3 Make best practices and guidance for recovery easily available via accessible tools (e.g., checklists, policies, shifts, schedules, rest locations, etc.)

Vig2-T4 Begin using information-sharing mechanisms to communicate with stakeholders about lifeline restoration activities.




71

Vig2-T5 Identify indicators that can be monitored through social media tools that the public is aware of alerts and warnings that are issued and are responding appropriately.

Vig2-T6 Alerts, warnings and emergency public information will be provided in multiple formats and through multiple technologies.

Vig2-T7 Mechanisms/processes for sharing information are technologically possible for all stakeholders.

Vig2-T8 Feedback and/or follow-up on information (e.g., respond to specific citizen requests) can be provided through the use of tools.

Vig2-T9 Key officials are notified in the event of an incident using relevant tools and technologies (e.g., call down lists, SMS messages).

Vig2-T10 Information was organized, linked, searchable, and easily retrievable using tools (e.g., data gathered during the recovery effort that can be referenced).

Vig2-T11 Develop and update geo-coded database of all pre-designated mass care and shelter operations.

Vig2-T12 Monitor the changing population levels and circumstances in mass care and shelter operations.

Vig2-T13 Tools allow public and organizations to identify citizens in need of assistance during evacuation and sheltering.

Vig2-T14 Alert/warning systems and media outlets are monitored to assure the proper message is delivered from the authorities.

Vig2-T15 Results of situation assessments are compared and contrasted using tools that can share information between stakeholders.

Vig2-T16 Ensure information is accessible to all stakeholders via available tools

Vig2-T17 Summarize the inflow of gathered information from all technology sources in a timely fashion.

Vig2-T18 Products produced by DVs' tools are usable in the format in which they are received.




72

ANNEX M. CAUSE METRICS


Governance

CAUSE-G1 Frameworks exist that are used to support decision making (e.g., LACES).

CAUSE-G2 Partner organizations are notified that assistance may be required or provided if needed.

CAUSE-G3 Source of additional resources and support from stakeholders is known in advance.

CAUSE-G4 Authorities make an effort to understand the information that is gathered through social media.

CAUSE-G5 Evidence that officials are aware of information provided through social media.

CAUSE-G6 Stakeholders can assess whether actions and plans are likely to lead to reduced loss of life, business and property compared to previous disasters.

CAUSE-G7 Evidence of cross-border cooperation (e.g., sharing of resources, trade resuming under modified circumstances) to plan response and recovery operations.

CAUSE-G8 Evidence that the authorities understand their accountability and their responsibility to inform the public (e.g., operational, financial, psycho-social).

CAUSE-G9 Volunteer organizations have a visible role during response and recovery that is clearly supported by the officials.

CAUSE-G10 Evidence that volunteer organizations and officials have a cooperative relationship (e.g., present at the same briefings, identified in educational material).

Standard Operating Procedures (SOPs)

CAUSE-SOP1 Citizens, businesses and organizations expect to find information online from authorities.

CAUSE-SOP2 Debriefs are performed once normal operations are resumed.

CAUSE-SOP3 Evidence of enhanced coordination of response (e.g., assignment of resources, established services, activation of mutual aid, staging of equipment).

CAUSE-SOP4 Plans identify and refer to lessons learned from previous disasters.

CAUSE-SOP5 Plans have flexibility to be able to accommodate requests for assistance from officials.




73

CAUSE-SOP6 Responders are empowered to make time critical decisions based on their training and experience.

CAUSE-SOP7 Public commentary regarding the management of the current and previous emergency events is assessed at important milestones.

CAUSE-SOP8 Volunteer organizations clearly support the authorities and provide complementary assistance (e.g., amplify the official information).

CAUSE-SOP9 Potential requirements for assistance and support are communicated to relevant organizations.

Technology

CAUSE-T1 Multiple modes of information exchange can be employed.

CAUSE-T2 Interoperable technology has a high level of 'ease of use' and can support emergency response and recovery operations.

CAUSE-T3 Technology allows officials to verify, organize and filter out the noise to meet their own information requirements.

CAUSE-T4 Multiple device types are used to access information (e.g. social media sites).

CAUSE-T5 Multiple tools are used to present summarized information updates.

CAUSE-T6 Technology can accommodate required planning changes.

CAUSE-T7 Technology investment can be linked to reduced economic impacts in the affected regions.

CAUSE-T8 Interoperability allows citizens to remain connected in various ways during a response and recovery operation.

CAUSE-T9 Developments in interoperability are linked to lessons learned from previous disaster.

Training & Exercises

CAUSE-TEx1 Training on technology best practices is available to operators.

CAUSE-TEx2 Interoperable technology is used during regular exercises.

CAUSE-TEx3 Lessons are documented following completion of response and recovery operations.

CAUSE-TEx4 Knowledge is shared within the operational community.

CAUSE-TEx5 Proactive control of hazards to stop them from becoming disasters.

CAUSE-TEx6 Experience through practical training allows responders to recognize when information gathering requirements change.




74

CAUSE-TEx7 Experience through practical training allows responders to recognize when alerts, warnings and notifications must be updated and shared with the public.

CAUSE-TEx8 Experience through practical training develops the understanding of which communication method will be most critical in the unique circumstances of the event and recovery.

CAUSE-TEx9 Training on social media analysis can assess the public's compliance with emergency response and recovery operations.

CAUSE-TEx10 Evidence that the greatest economic impacts are known and are being addressed through training and exercises.

Usage

CAUSE-U1 Interoperable technology is regularly used during normal daily operations.

CAUSE-U2 Interoperable technology use increases the reach and range of information sharing.

CAUSE-U3 Information containing actionable details (e.g., requests, damage, safety issues, crime) is submitted through social media sites and a response is expected.

CAUSE-U4 Sharing of information within responder community (e.g., schedules, refreshment/rest locations) is achieved through interoperable technology use.

CAUSE-U5 Traditional media outlets are used to push information to the public and are augmented with interoperable technology.

CAUSE-U6 Social media used by authorities to exchange (push/pull) information with audiences beyond the affected community(ies).

CAUSE-U7 Citizens' perceptions of the effectiveness of local government are monitored through response and/or recovery operations.

CAUSE-U8 Evidence exists that citizens trust authorities to provide timely information and response during and after an event.

Documents

CSSP Task 25 Evaluation Framework and Data Analysis Plan · Task 25 Evaluation Framework ISR Report 6060-01-03 The Canadian Safety and Security Program is led by Defence Research