SIFT Playbook and m:N

Control Overview

For other staff and projects, please see

http://www.sift.net

Smart Information Flow Technologies

Dr. Phillip Walkerpwalker@sift.net703-244-9096

26 March 2021

Smart Information Flow Technologies

2

• 35-person small business• 18 years in business.

• $7M+ per year in revenues.

• Distributed throughout U.S.• Based in Minneapolis.

• Branches in DC, Boston, Dallas, San Diego, San Francisco.

• R&D expertise:• 35 advanced degrees (Computer Science,

AI, Psychology, Control).

• 150+ years combined industrial R&D experience.

• Research spanning computer science, human centered systems, and beyond.

• Customers: DARPA, NASA, AFRL, ONR, ARL, OSD, NIST, AFOSR, Lockheed, BBN, BAE Systems…

Downtown Minneapolis

Lexington, MA

26 March 2021

26 March 2021 3

How do you manage complex Agent(s)?Supervisory control: how humans achieve effective, scalable control of complex, heterogeneous agents

➢ By giving up some decision and execution tasks and, instead, “controlling” at a higher level

➢ Long history in Control since Sheridan and Verplank, 1978.

Delegation: How humans perform/achieve supervisory control (aka Training, Instructing)

1. Goal (partial state) to be achieved

2. Plan (action sequence) to be

performed

3. Constraints (states or actions) to be

avoided

4. Stipulations (actions or states to be

achieved)

5. Value statement on states and

actions

Supervisor Method

Subordinate Responsibility

Goal Achieve goal if possible; report if incapable

Plan Follow plan if possible; report if incapable

Constraint Avoid actions/states if possible; report if not

Stipulation Achieve actions/states if possible; report if not

Value Statement

Work to optimize value

4

A Playbook® Approach to Delegation

• A means of Delegation

• Require a shared knowledge of

domain Goals, Tasks and Actions

• Supervisor calls plays; Agents

have autonomy within the play’s

scope

• Plays reference a defined range

of plan/behavior alternatives

• Supervisor can further

constrain/stipulate/compose

• Delegation by goals/plans

/constraints/values

• “Service Requests” are like plays,

but without full command

authorityA page from Alonzo Stagg’s 1927 Playbook

426 March 2021

Playbook® Reference Architecture

Analysis &

Planning

Component

Algorithms

Control

Instructions

Feedback

Shared Task Model

Playbook UI

Vehicle(s)

Special Purpose Planners(e.g., Route, Sensor)

Pla

yb

oo

k®

Pro

per

Execu

tio

n

En

vir

on

men

t

Executive

Incoming Data

Streams

Playbook balances and integrates Planning, Execution and Human-

Autonomy Interaction

526 March 2021

15+ years of Playbook-Related Work

6

• Ground-based UAV mission planning Tactical Mobile Robots--DARPA

• Mixed Initiative Control of Automateams(MICA)--DARPA

• Playbook-enhanced Variable Autonomy Control System—DARPA SBIR

• Teamed with Geneva Aerospace• Heterogeneous UAVs; MOUT Surveillance

• TUSC—AFRL SBIR• Air Force recon and surveillance missions, MIIIRO

testbed integration

• PUMA– NRL SBIR• Heterogeneous Naval UAVs UI enrichment

• Army USI– AFDD task order• Develop Army Air-controlled demo

• HURT (SIFT Personnel at Honeywell)--DARPA• Persistent goal management; service requests• Flight demo– managed 6 heterogeneous UAVs to satisfy

imagery requests for 6 soldiers with dynamic no-fly zones

• Levels of Automation3--Army SBIR• Integrated persistent goal management into PB• Extensive human experiments in MUSIM testbed• Army air-ground flight demo

• FLEX-IT—AFRL Task Order• Multi-modal PB concepts, UIs and demos for

Predator/Reaper

• IMPACT ARPI– Joint Autonomy Research

Multi-UAV simulation (3)Touch/Sketch/

MultiTouch Gesture

HOTAS(Hands On

Throttle & Stick)

HOKAM(Hands On Keyboard & Mouse)

Audio Outout/ Speech Input

TSD

UAV Video Outpu

t

A FLEX-IT Workstation

626 March 2021

Levels of Automation3

7

NUGGET: Development and experimental exploration of Playbook concept for multi-UAS (6+) supervisory control

APPLICATION:

• Developed Playbook®

implementations in MUSIM testbed

• Extensive Experimental Validation (including workload and breakdown conditions)

• Flexible plays:• Reduce Wx (esp. on secondary tasks)

• Enhance RT and SA

• Are NOT susceptible to automation complacency and error effects

STATUS:

• 2.5 years SBIR funding (Ph1&2)

• 2 related flight demonstrations (RT-Max) by US Army AFDD

• Ongoing use AFRL’s FLEX-IT and AFDD’s DelCon

From Fern & Shively, 2009

26 March 2021

FLEX-IT

8

NUGGET: A demonstrator testbed for HCI concepts using flexible delegation and play calling for multiple Unmanned Vehicles

APPLICATION:

• Emphasis on Flexibility– user can “call plays” at any level, can manually fly, can interrupt and resume, can tune and tweak parameters, etc.

• Multi-modal: Voice I/O, touch, sketch, HOTAS

• Multiple novel UI and Human Automation Interaction concepts

• E.g., Noodle: a “fly ahead” capability using HOTAS

STATUS:

• 3 years AFRL funding

• Migrated to Vigilant Spirit

• Ongoing use on IMPACT ARPI

Multi-UAV simulation (3)Touch/Sketch/

MultiTouch Gesture

HOTAS(Hands On Throttle &

Stick)

HOKAM(Hands On

Keyboard & Mouse)

Audio Outout/ Speech Input

TSD

UAV Video Output

A FLEX-IT Workstation

Manual: Convention

al “hands-

on-throttle-

and-stick”control.

Higher Level

Plays: Pilot’s verbal

commands

initiates a

planning

interaction with

automation &

then automates

execution steps.

Maneuver

Plays: Pilot’s

verbal

command

initiates

short,

simple

maneuver.

“Noodle”: Pilot’s

inputs on

stick &

throttle

defines

UAS’s near

future path.

HOTAS

“Hook Left”

Multiple, Composable Methods of Control

26 March 2021

Playbook from FLEXIT to OFFSET

9

FLEX-IT, AFRL,

2010-2013

IMPACT ARPI,

AFRL, 2014-2016

CODE, DARPA,

2015-2017

ISHIS SBIR, ONR,

2016-

OFFSET, DARPA,

2017-

Multi-UAV simulation (3)

Touch/Sketch/ MultiTouch Gesture

HOTAS(Hands On Throttle

& Stick)

HOKAM(Hands On Keyboard &

Mouse)

Audio Outout/ Speech InputTSD

UAV Video Output

A FLEX-IT Workstation

26 March 2021

CODE Program Overview

• … extend mission capabilities and improve U.S.

forces’ ability to conduct operations in denied or

contested airspace

• …create a modular software architecture beyond

the current state of the art

CODE intends to focus in particular on developing and demonstrating improvements in collaborative autonomy—the capability of groups of UAS to work together under a single person’s supervisory control

…develop the operational concepts for CODE-enabled missions and validate their effectiveness through detailed modeling and simulation. It also intends to develop the most promising capabilities and demonstrate them in flight using multiple surrogate UAS

Distribution Statement A Approved for Public Release, Distribution Unlimited

SIFT RolePh1– a Track B performer creating a holistic user interface for pre-mission planning and in-mission communication and SA– even in comms denied environmentsPh2 & 3– subcontracted to Lockheed providing planning and plan evaluation UI aid

Map Display

Play Creator

Alternate Plan Evaluator

1026 March 2021

11

ISHIS*:

NUGGET: Initial design of a Behavior Shaping User Interface for swarms, and definition of an experimental platform to validate it

APPLICATION:

• Human interaction with (awareness and control) of swarms is understudied

• Communications restrictions and lags, coupled with swarm uncertainty makes traditional UIs problematic

• Applying multiple techniques to create behavior aggregation awareness, decision aids and controls

• Initial Undersea focus

STATUS:

• Partnered with Katia Sycara (CMU) and Mike Lewis (UPitt)

• Ph2 redirected to focus on “red teaming” tools for countering swarms

* Greek for Strength or Power

Interface System for Human Interaction with Swarms

N162-126, ONRDr. Marc Steinberg

26 March 2021

12

PINNA:

NUGGET: Review Playbook concepts and suggest applicability for UAS in the NAS design, regulation and experimentation

APPLICATION:

• Near term commercial application scenarios (sustained monitoring)

• Larger scale cargo and passenger scenarios

• Pilot/operator focus but exploring interaction with ATC and Dispatch

• Leveraging prior NASA work on Detect & Avoid and Maintain Well Clear

STATUS:

• 6 month initial effort, 2&3Q 2018

• Articulated play concepts

• Demoed “listening in” for plan inferencing

NASA ARCDr. Jay Shively

Playbook Interactions for Non-Piloted vehicles in the National Airspace

https://www.youtube.com/watch?v=piVjsLPqitA26 March 2021

“Displaced Transparency”

• The dilemma of transparency is not new… and not limited

to human-machine interactions

• Human organizational solution is Displaced Transparency

• Much human work interaction with human partners involves

delegation of intent:

13

“Displaced Transparency” = Relaying transparent information “displaced” (in

time or distance) from the LOCA

• All about conveying one’s mindset/intent to another mind

• Entails communication of intent via language, symbols, procedures, priorities,

• Implies oversight and correction• But “mind melds” take time… how do we

accomplish it efficiently and avoid errors?

26 March 2021

Transparency across Temporal Spans

14

minshrsdayswksmos mins hrs days wks mos

Action

“Normal” Transparent UI Usage Window

Negotiation about:

• Intent

• Plans

• Policies

• Design

Discussion about:

• Explanations

• AARs

• “Tuning”

Before Future Action After Past Action

26 March 2021

Why does it work?

• Q: What is it about transparency, planning, intent

communication, trust and debriefing that make them all

pertinent to improved human-automation teaming?

• A: It’s all about mental model synchronization through

negotiation (or having negotiated, and trained)

• That can happen in the moment of need…

• But it doesn’t HAVE to…

• …and it’s probably best not to leave it until then if you can avoid it

• …And frequently, you can.

15

That’s why high performing teams interact less, and less explicitly1

That’s why debriefings are such a highly effective strategy for team development2

That’s why plans are nothing but planning is everything2

That’s why plays work in all the ways they do

That explains multiple trust effects4

1Entin & Serfaty, 19992Tannenbaum & Cerasoli, 2013

3Eisenhoser, 19574Lee & See, 200426 March 2021

Some n:M Lessons Learned

• Flexibility is key:

• As flexible as you want it to be; as automated as you need it.

• But there will be dual temptations:

• To use precise control and focus where one shouldn’t

• To rely on and expect automation where one shouldn’t

• As multi-agent systems become more complex, working

out problems in real-time becomes less tenable.

• “Displacing Transparency” to planning, policy, mental model

sharing, negotiation and after-action review

1626 March 2021

Contact Information

www.sift.net

17

Dr. Phillip Walkerpwalker@sift.net703-244-9096

Dr. Christopher Millercmiller@sift.net612-716-4015

26 March 2021

OFFSET : I3

For other staff and projects, please see

http://www.sift.net

Smart Information Flow Technologies

Josh Hamelljhamell@sift.net

26 March 2021

Table of Contents

19

26 March 2021

• Program Description

• OFFSET vs m:N

• Immersive Interactive Interface

• Support for Scale

• Lessons Learned

• Q&A

OFFSET : Program Description

20

26 March 2021

• OFFensive Swarm-Enabled Tactics

• Heterogenous composition

• Up to 250 platforms

• Urban operations

• Twice yearly exercises at MOUT/CACTF facilities, real hardware, representative scenarios

• Final program phase

(DARPA)

• Single Operator vs Multiple

• Magnitude of platforms per operator

• ISR, mock engagement

• Tolerant to platform failure/loss

Contrast m:N

21

26 March 2021

(DARPA)

OFFSET : SIFT

22

26 March 2021

• Working under CCAST

(BBN Raytheon)

integration team, with

OSU

• I3 – Immersive

Interactive Interface

• Multimodal

• “Near the battle”

• Control at varying levels

of aggregation

(DA

RP

A)

I3 Architecture

23

26 March 2021

• Unity application, VR / Windows 10• Valve Index

• I3 consumes telemetry, world state; issues tactics and commands

Platform

State

Object

Recog.

‘Comms’

Artifacts

Swarm

Commands

World Models

Scenario Information

Tactic Specifications

Initial Geometries

I3 User

STOMP

Dispatcher

Sprinter/Ext

Eval Server

Real/Virtual

Platforms

I3 : In Scenario

24

26 March 2021

Dispatcher

(USGS)

I3 : Visualization

25

26 March 2021

World Model

Mission Plan, Scenario Status

Platforms : Status, Tactics, Groups

Artifacts

Inspection Tools

• Controllers

• Object inspection, selection, designation

• Haptic feedback

• Gesture Recognition

• Speech

• Tactic Invocation

• Mission Plan Control

I3 : Control

26

26 March 2021

Support For Scale

27

26 March 2021

• Mission Plans

• Swarm Level Interaction

• Intelligent Automation

Lessons Learned

28

26 March 2021

• Emphasize higher levels of automation and aggregation

• Design to discourage low level interactions

• Resilient (and/or adaptive) behavior w.r.t. comms

• Automation backplane support (authority)

• Highlight goals and tasks

• Real world is messy

Questions?

29

26 March 2021

?

Contact Information

www.sift.net

30

Dr. Christopher Miller

cmiller@sift.net

26 March 2021

Josh Hamelljhamell@sift.n

et