EDSS Program Review: June 2002

6/19/2002 Applied Physics Laboratory/ Naval Research Laboratory

1

EDSS Program Review: June 2002

David Jones:APL-UWJim Ballas: NRL


2

Outline

David Introduction Project Overview HCI Experience Progress Report


3

Outline…

Jim UCD Process Task Analysis Evaluations Future Plans


4

APL-UW Team Members Bob Miyamoto- PI David Jones Troy Tanner & Bill Kooiman


5

APL-UW Background Miyamoto’s Group

Env Effects on Sensors

TDA development Training Tools

Me


6

Project Overview General Philosophy- the design model

fits the user’s mental model: UCD Perform task analyses that feed into the

interface design & its evaluation Support EDSS developers with UCD and

HCI standards and guidelines Provide iterative feedback


7

Example of UCD Application: DMARS

Study the user’s information needs

Study how the user performs given tasks

Create an intuitive process

Involve users in design


8

High Seas Workflow System Helps produce the High

Seas Warning Heard from the supervisors Then heard from actual

users Different stories Created a system for the

users Flexible design


9

HSW cont… Funded by DARA & SPAWAR Written in JAVA Uses Polexis’ XIS for map

functions Running operationally at SD

METOC Center


10

FNC Project: EVIS Studying human-system

component of METOC support

Worked closely with users and customers

Conducted experiments Performed Cognitive Task

Analysis Gone to sea for evaluation

RULE MISSION OPERATION EVOLUTION PARAMETER MARGINAL SEVERE T00 T12 T24 T36 T48 T60 T721 AIR AERIAL REFUEL ALL LEVELS VISIBILITY 1600-4800 M <1600 M 2 AIR AERIAL REFUEL ALL LEVELS TURBULENCE LIGHT MODERATE 3 AIR AERIAL REFUEL ALL LEVELS ICING LIGHT MODERATE 4 AIR AERIAL REFUEL ALL LEVELS THUNDERSTORMS WITHIN 25 NM WITHIN 10 NM 5 AIR AIR BASE LAUNCH/RECOVERY VISIBILITY 1600-4800 M <1600 M 6 AIR AIR BASE LAUNCH/RECOVERY CEILING 1000-3000 FT <1000 FT 7 AIR ATTACK CLOSE AIR SUP CEILING 1000-3000 FT <1000 FT 8 AIR ATTACK CLOSE AIR SUP VISIBILITY 1600-4800 M <1600 M 9 AIR ATTACK CLOSE AIR SUP TURBULENCE LGT-MDT HEAVY 10 AIR BOMBING LOW LVL +500 AGL CEILING 0500-1000 FT <500 FT 11 AIR BOMBING LOW LVL +500 AGL VISIBILITY 4800-5600 M <4800 M 12 AIR BOMBING LOW LVL +500 AGL TURBULENCE LIGHT MODERATE 13 AIR BOMBING LOW LVL +500 AGL ICING LIGHT MODERATE 14 AIR BOMBING MID-HI LEVEL CLOUD COVER 40-60% >60% 15 AIR RECON GROUND LEVEL VISIBILITY 1000-3000 M <1000 M 16 AIR RECON HIGH LEVEL CLOUD COVER 50-60% >60% 17 AIR RECON HIGH LEVEL VISIBILITY 4800-5600 M <4800 M 18 AIR RECON LOW LEVEL CEILING 1000-3000 FT <1000 FT 19 AIR RECON UAV (DRONES) CEILING 4000-6000 FT <4000 FT 20 AIR RECON UAV (DRONES) VISIBILITY 4800-5600 M <4800 M 21 AIR RECON UAV (DRONES) ALTITUDE WIND 40-60 KT >60 KT 22 AIR RECON UAV (DRONES) PRECIPITATION 0.1-0.3 IN/HR >0.3 IN/HR 43 C4ISR HIGH ALTITUDE ISR CLOUD COVER 40-60% >60%


11

Progress Report Initial Task Analysis -Jim Gaining Domain Knowledge Training Observations- David Organizing a UCD Workshop


12

Gaining Domain Knowledge EDSS User’s Guide Draft Mission Needs Statement for

Distributed Collaborative Planning Systems for Expeditionary Forces

COMPHIBGRU THREE 041605Z OCT99 4.X Tiger Team User Input Spreadsheet Pubs: NWP 3—02.1; ATP 3 ch 6, JP 3-02


13

Training on the USS Tarawa

Attended training in San Diego: Apr

Enthusiastic users

Hands-on training well received

Great audience for usability evaluation


14

Example of User reactions Staff personnel were excited

about creating overlays for planning

But.. Menu headings caused

some confusion “Are Assault Plans part of

AOA Mgmt?” “When do I use the Env

DB?”


15

User reactions Navigation

among the different windows was difficult at times

Some windows require expertise that all users might not have or forgot


16

Quick Thoughts after Training Most users want it to look like

Windows DII UIS provides HCI guidance Some ideas

Back Arrows & Undo command Web-based and searchable user’s guide Tooltip help- On mouse of menu title Workflow wizards or web-based training


17

UCD Workshop Scheduled for 30 July 2002 At SAIC Tysons Corner office Our ideas

UCD Processes DII HCI Standards HCI Design Principles with examples HCI Evaluation

SAIC Ideas?


18

Jim Ballas

UCD & HCI Task Analysis Evaluations Future Plans


19

NRL-WDC Team Members

Jim Ballas Ph.D. in Applied Experimental Psychology

Derek Brock M.S. in Computer Science, HCI emphasis

Beth Kramer M.S. in Human Factors Psychology

Janet Stroup B.A., some graduate CS coursework


20

NRL-WDC: Interface Design and Evaluation Section4 Ph.Ds on staff of 10

Expertise in HCI, Human factors, Cognitive Science, Computer Science, Auditory perceptionProjects include AEGIS (with LMC), DDX (with Raytheon), NATO S&T, KSA EVIS Management6.1 to 6.3 projects


21

NRL-WDC: Interface Design and Evaluation Section

HCI Research cited in Major reference documents

Handbook of Human Computer Interaction

ACM CHI Conference 2001 paper: “Demystifying

Direct Manipulation”Wrote and Revised Operator

Workstation Evaluation section for IUSW-21 at Sea Test this Sept


22

User Centered Design

Following approach outlined in NATO COADE document

Additional principles: HCI as an instance of language use


23

User Centered Design: COADE


24

User Centered Design: COADE


25

Viewing HCI as an Instance of Language Use The design and implementation of an

effective software application and its user interface is ultimately a communication problem that always involves both the designer’s meaning and the user’s understanding

The principles at work in people’s use language form a comprehensive framework for the design of human-computer interaction


26

Principles of Language Use Any form of communication between

people is an instance of language use; language is used to do things together

Language use requires people to coordinate their actions and their attention (cognition); it always involves “speaker’s” meaning and “addressee’s” understanding.

Meaning and understanding require common ground


27

Common Ground Common ground is knowledge that

people establish they can use with each other on the basis of shared experience

When common ground is missing, meaning and understanding breakdown

Building common ground is always a serial process - even though the resulting shared knowledge may contain gaps


28

Layers in Language Use and in HCI Language use frequently involves more than one

conceptual layer of activity; telling a story, for instance, involves at least two layers:

The story teller and the listener participate as themselves in the first layer

In the second layer, the events of the story take place Similarly, HCI has two principal layers of activity:

The designer and user participate as themselves in the first layer

In the second layer, the user interacts with the computer as if it (and not the designer) were the user’s counterpart

Each layer in an instance of language use makes different demands of the user’s language use skills


29

Language Use Issues in HCI In the first layer of HCI, designers, through

the software’s presentation, must help users to compensate for gaps that direct access (through menus, etc.) imposes on the process of building coherent common ground

In the second layer, wherever possible, interfaces should be designed to allow users to establish and use common ground with the interface itself as a regular part of their interaction with the computer


30

Guidelines, Standards, and Relevant Literature DII User Interface Standards

MilStd2525 Research on distributive planning:

Klein & Miller Work directed by NRL Cited in MCDP-5

General human factors and HCI literature


31

Work to Date: Initial Task Analysis Partially Complete

Initial Task List:EDSS planning

Make Basic Decisions Create Operational Area Determine Landing Craft Complete Default Craft Parameters Table Make Navigation Decisions Design Sea Echelon Areas

Free Hand 4W Grid

Select HLZ Select Beach Center and Boat Lane Design Routes Design Display Determine Ship-to-Shore Movement

1 2 3 4 5

Make Basic Decisions Create Operational Area Determine Landing Craft

Complete Default Craft Parameters Table Make Navigation Decisions Free Hand

Goal

Amphibious Task Force objectives and plan general courses of action

Make master plan: amphibious landing plans for a variety of different geographic situations

Determine landing craft availability as basis for landing craft assignment to the ship-to-objective movement

Establish default planning speeds (speed made good) of the landing craft and the helicopters

determine the unit of measurement that will be used

used when the geography does not fit a 4W grid pattern, e.g., in confined waters

Product

Landing plan directory which can contain numerous landing plans

Table of landing craft available for each ship Table

decision to promulgate unit of measurement in the OPTASK Amphib to ensure that all staffs and ships plot the same information

Performance Requirements

Type of chart being used; degree of accuracy that can be plotted on the paper chart; degree of the accuracy required

Information Requirements Default craft parameters

types and number of landing craft available for each ship Default craft parameters table

Actions Fill in table

Decisions

Determine beachheads, landing areas, landing sites, beaches, helicopter landing zones, etc., H-Hour

Asset type (choices are surface, air, infantry platoon & company, mech infantry platoon & company, and armor platoon & company) Decision re. changes to default parameters

Nav Units-Sea Min (DDMM.MM); Nav Units-Sea Secs (DDMMSS; Land-measurement in Grid format

DESIGN SEA ECHELON AREAS

Task Name

6


32

Task Analysis: other tasks Administrative/file operation tasks

Log On And Initializing Log Off Installing New EDSS Software Exporting Plans Importing Plans

GCCS-M Tasks Saving Slides Deleting Slides Exporting Slides Installing Maps And Charts Retrieving Maps Or Charts Removing Maps Or Charts Uninstalling Maps, Charts, And Imagery Line Of Sight (Los) Profile


33

Future Work Complete task analysis

Including cognitive analysis using Critical Decision Method

Workshop:To include illustrations of design issues, e.g. illustration of lighting/ filtering effect on color images


34

Future Work: Evaluation Approach: evaluate user

performance and compare to desired performance

Examples: DMARS EVIS Software Development Tools and

Processes


35

DMARS Evaluation Summary Observe and compare use of three media

Paper (NAVOCEANO Mine Warfare Pilot publication) WWW (based on MWP, so called RP-WEB UC-CD (User Centered Digital METOC Acoustic

Reference Manual - DMARS) Five METOC tasks: prepare brief and answer 4

questions 12 METOC officers and enlisted personnel

NAS Patuxent River NAS North Island

Each person tested on five tasks using three media


36

DMARS Evaluation: Process Measures Task timing logged with Activity Catalog Tool

a NASA sponsored tool Coding scheme distinguished following tasks

Acquire information Browse: search for topic in the METOC document by

navigating from one section to another to another (Search for and Move To)

Interpret: read information from a specific section; Assemble briefing document;

Compose/edit: generate and/or modify the document using a word processor or presentation software;

Copy/paste: copy material from the METOC document into the briefing document.


37

DMARS Evaluation: Outcome measures Accuracy

On Problem 1, Subject presented brief and experimenter graded eight items

Required analysis, not just picture Problems 2-5, Subject supplied

answers.

Number of images used in briefing Preference


38

DMARS Evaluation: Time to Find Information to Prepare Briefing

Significant effect of document type

F(2, 16) = 5.62, p < .01 RP-WEB> [UC-CD, PAPER]

Discussion Effect on key problem

Only on browse time No briefing preparation

differences (when time to manually prepare slides added to PAPER condition

No interpretation time differences

Magnitude RP-WEB 160s longer than UC-

CD on a task which overall takes 1260 s (13% time increase)

0

200

400

600

800

1000

Browse time (s)

UC-CD PAPER RP-WEB


39

DMARS Evaluation: Accuracy

Significant effect f document type F(2,16) = 3.48 , p =.055

UC-CD > [PAPER, RP-WEB] @ p = .087

Discussion UC-CD errors on problems

4 and 5 due to omission of location selection

UC-CD errors on winds in Problem 1 due to user inexperience with new form of wind vector--only a short training period used.

0

0.2

0.4

0.6

0.8

1

UC-CD PAPER RP-WEB


40

CCTV

Desk

Door

Storage (document)

IT21

Doo

r

NITESServer

NITESNT

NITESNT

NITESNT

IT21

IT21

Printer

SPA-25(Radar)

: Video Camera

CCTV

F: Forecaster

T: Technician

O: Project Observer

NITESServerRAID

Desk

: Workstation

NOWCOPIER Coffee

FILE

T

O

O

Wind

FILE

FILE

SMOOS

SMQ-11(Satellite)

: Chair

FIT21

T

T

: Port Hole

Example of At Sea Observations: USS CARL VINSON during Battlegroup training (COMPTUEX)


41

At Sea Observation Methodology Office Environment

Observed forecasters workflow (not-interfering basis) Two watches a day (12 on, 12 off) Each session is about 12 hours long Two Observers for each watch; two observations per watch

Equipment Three video cameras Note taking Palm Pilot for recording the timing of the task Performance Questionnaires Interviews


42

Future Work Summary Workshop Task Analysis Design recommendations Evaluation

Documents

EDSS Program Review: June 2002