Fall MeetingOctober 2013

International Technology Alliance in Network and Information Sciences

Human-Machine Conversations to Support

Coalition Missions with QoI Trade-Offs

Alun Preece (Cardiff)Dave Braines (IBM UK)

Diego Pizzocaro (Cardiff)Christos Parizas (Cardiff)

Tom La Porta (PSU)

Human-machine information interaction in coalition mission support

High-level tasking of network resources in terms of mission objectives

Enabling exploitation of soft (human) sources in addition to physical sensing assets

“Locate & track high value targets in Border Zone”

Information quality in coalition mission support networks is highly variable, both due to the nature of the sources and the network capacity

Coalition mission support

“Conversational” model for human-machine and machine-machine interactions

Interactions flow from natural language (NL) to Controlled English (CE) and back

Scenario-based experimentation

Natural language-based approach

Users request information from the network (ask), while also being sources of information (tell)

3

Human-machine conversationsHuman-machine conversations

Human-machine example:

Soldier on patrol reports a suspicious vehicle by means of a text message from their mobile device.

Machine-human example:

A software agent sends a brief “gist” report to a human analyst indicating the vehicle is associated with a known high-value target (HVT).

Machine-machine example:

A broker agent queries or updates a database agent managing HVT sightings.

Contributions

Natural language (NL) based conversational approach that includes support for:

requests for information (with specified QoI requirements)

provision of information (at specified QoI levels)

human-machine reasoning and information fusion (with aggregated QoI levels)

Show how the model can be used to support realistic information exchanges in a coalition mission support context, offering flexibility in dealing with trade-offs associated with quality-of-information (QoI).

A new approach using Controlled Natural Language

A new approach using Controlled Natural Language

TA6 context P4.3: Coalition Context-Aware

Assistance for Decision Makers Link to P4.1: how coalition

communications environment affects cognition

Link to P4.2: advanced fact extraction & reasoning

Link to P5.2: high-level CE-based policies for asset management

Types of interaction

NL to CE query or CE facts (confirm)

CE query to CE facts (ask-tell)

exchange of CE facts (tell)

Gist to full CE (expand)

CE to CE rationale (why)

Controlled English Conversation Cards (CE-Cards)

Controlled English Conversation Cards (CE-Cards)

We conceptualise a conversation as a series of cards exchanged between agents, including humans and software services

A conversation unfolds through a series of primitive communicative acts (e.g. queries, assertions, requests)

natural language Controlled English a form of template-based CE that

provides the “gist” of complex sets of CE sentences for brevity and easier human-readability

Cards are modelled in CE with types as shown and attributes that include:•is from•is to•is in reply to•has content•has timestamp•has resource (for linked non-text content)

3 kinds of card content

CE card model

CE-Cards Conversation PoliciesCE-Cards Conversation Policies

Examples

gist: “the red SUV is a threat”

expand: “red SUV”

tell: “there is a vehicle named v12345 that has `red SUV’ as description and has XYZ456 as registration and…”

tell: “there is a vehicle named v12345 that is a threat and is located at central junction and…”

why: “v12345 is a threat”

tell: “v12345 is owned by HVT John Smith and…”

Conversation sequence rules*

*can be defined in CE as part of the model

Vignette IVignette I

Interacting agents:human patrol (location A)human intelligence analystbroker software agent that mediates between humans and other agentstasker software agent that handles access to database and sensor resources

Step 1Patrol on North Road (A) reports a suspicious black saloon car, vehicle registration ABC123, moving south

[confirm interaction between patrol & broker; QoI on certainty of observation]

Vignette IIVignette II

Interacting agents:human patrol (location A)human intelligence analystbroker software agent that mediates between humans and other agentstasker software agent that handles access to database and sensor resources

Step 2Broker sends the patrol's report to tasker agent, and a DB query reveals the vehicle is linked to a HVT

[ask-tell interaction between broker & tasker; QoI on certainty that vehicle is linked to HVT]

Vignette IIIVignette III

Interacting agents:human patrol (location A)human intelligence analystbroker software agent that mediates between humans and other agentstasker software agent that handles access to database and sensor resources

Step 3Broker sends a request to tasker to track the vehicle. Tasker assigns a UAV to perform this task

[tell interaction between broker and tasker; QoI requirements on accuracy of tracking]

Vignette IVVignette IV

Interacting agents:human patrol (location A)human intelligence analystbroker software agent that mediates between humans and other agentstasker software agent that handles access to database and sensor resources

Step 4UAV reports that the vehicle stops near Central Junction (B); the broker sends an alert to analyst

[tell interactions between tasker, broker, & analyst; aggregate QoI from steps 1-3]

Summary of interactions (Steps 1-4)Summary of interactions (Steps 1-4)

there is a tell card named '#2b' that is from the agent tasker and is to the agent broker and is in reply to the card '#2a' and has content the CE content 'there is an HVT sighting named h00453 that has the vehicle v01253 as target vehicle and has the person p670467 as hvt candidate'.

Detailed interactions: Step 1Detailed interactions: Step 1

Detailed interactions: Step 2Detailed interactions: Step 2

Detailed interactions: Step 3Detailed interactions: Step 3

Handling Quality of Information (QoI) Issues

Handling Quality of Information (QoI) Issues

Example 1 – QoI from the patrol (step 1)If the patrol is a trained team, likely that the word “suspicious” has a codified meaning and can be directly mapped to some kind of certainty range (symbolic or numeric).

For a message from an untrained team or social media, the subjective meaning of “suspicious” would need to be estimated and the certainty recorded. e.g. based on past performance.

These cases can be handled by the receiver of the message – broker in our vignette – using rules (conversational pragmatics)

Example 2 – quality of information retrieved from database (step 2)We may have uncertainty that the vehicle is associated with HVT John Smith (the information may be out of date or inaccurate). Subjective logic may play a role in combining different uncertainty representations (symbolic, numeric). CE rationale plus provenance information offers a means of communicating the compounded result, gist offers a way to make it easily-digestible while also being expandable to users on request (or during subsequent audit

Architecture for experimentationArchitecture for experimentation

CE-Card server supports exchange & storage of cards

Software agents representing human and machine parties can be implemented in any programming language

We are experimenting with a speech-based interface, in conjunction with a Google Glass-style display for gist content

We would envisage full CE being directed to a user's handheld device

Conclusions & future workConclusions & future work

The conversational model appears robust enough to handle realistic scenarios

The model can be extended with pragmatics rules to handle a range of QoI cases

Next:Incorporate (richer) QoI cases / pragmatics.Extend/enrich scenario.Extend and link approach across P4 and to P5.Develop and run experiments with human subjects.