Playing games with the development of machine vision algorithms using ViPER

8/9/2019 Playing games with the development of machine vision algorithms using ViPER

1/18

Playing games with the development of

machine vision algorithms using ViPER

Presented by:

Benjamin WheelerNAVAL SURFACE WARFARE CENTER DAHLGREN DIVISION

UNMANNED AND ROBOTIC SYSTEMS INTEGRATION BRANCH (G82)

540.653.6030 (office phone)

540.642.2973 (mobile phone)

[email protected]


2/18

Introduction to Concept, and Motivation Dev. of machine vision for UxS is often dependent on data from real systems.

This work investigates the use of virtual environments to create synthetic datato feed perception system algorithms in robotic systems.

Background and Virtual Prototyping Utilization of open source, freeware, and govt owned simulation tools.

Demonstrates integration with existing machine vision code used in the

SUMET UGV program (Office of Naval Research Code 30).

Experiment Design and Setup Built a real and virtual experiment setup to generate stereo disparity images.

Compared real and synthetic imagery.

Results and Application Created stereo disparity images with an average disparity difference of 3.5

pixels between real and synthetic data for 656x492 pixel images.

Use of simulation can be effective when the world is sufficiently modeled.

OUTLINE: What are we talking about?

Utilizing M&S for computer vision


3/18

INTRO to CONCEPT: Virtual imagery for dev. of machine vision for UxS

Current development of machine vision for UxS is highly

dependent on access to the platform and real data. Size and complexity of UxS systems is ever increasing.

Test and development of algorithms for these systems is a technical and

resource challenge due to the multi-disciplinary nature and complexity of

current state-of-the-art robotic systems.

Progress is often bottlenecked by resource limitations,making synthetic data an attractive alternative. A simulated system would reduce resource requirements, speed

development time, and lower overall development cost.

However, it is difficult to correlate the true benefit of this approach, and the

ease at which high-fidelity simulated data may be created.

To test feasibility, using a real UxS perception system, we

attempted to create identical real and virtual data sets

Limited resources bottleneck development


4/18

ViPER enables software-in-the-loop dev.

The ViPER simulation was created under the SAF-T program

by ONR, to create a new paradigm in fire-control for RWS.

It was designed to integrate with Software-in-the-Loop

capability, and implements a preliminary design of the

autonomous target detection, integrated with a Linux-based

embedded system implementing semi-autonomous fire control.

In Sept. 2013, the initial baseline of ViPER for SAF-T was usedto conduct a user experiment to compare human performance

differences between tele-operated and semi-autonomous fire-

control modalities.

V i P E R

ViRT

UALPROTOTYPINGENVIRONMENTforROBOTICS

INTRO to CONCEPT: Virtual Prototyping Environment for Robotics (ViPER)


5/18

This work demonstrates the abili ty to create near identical

disparity maps using an existing UGV perception system.

Real Imagery Synthetic Imagery

Average of 3.5 pixel difference per dispari ty calculation

INTRO to CONCEPT: Virtual imagery for dev. of machine vision for UxS

Investigate use of fake imagery for real algo.


6/18

Serious Gaming is a focus area for training and system dev., often

leveraging the commercial gaming industry Examples include: VBS II, DSTS, OneSAF, Army Gaming Studio,

Delta3D,

Dev. examples include: RIVET, MODSIM (FCS), EODRS,

Many open-source, freeware, and low-cost game engines have beendeveloped for aspiring game creators

For warfighter-centric systems, todays games share many of the

features necessary for rapid virtual prototyping and virtual

demonstration of user-intensive systems.

Real-enough physics make interaction with the world believable

High fidelity graphics create an immersive environment for free

Easy to manipulate game environments enable creation of tailor-made

experiments, and the ability to rapidly test different system designs.

VIRTUAL PROTOTYPING: Why use a video game?

Allows creation of system-in-a-box


7/18

Utilize M&S to perform RWS vs.

supervised autonomous RWS (saRWS)experiments to generate requirements and

inform system design

In simulated world, information is known a-priori, allowing precise modeling of varying

levels of object detection, and creation of

training and test data.

Algorithms transition into ROS from

simulation, allowing concurrent

development of the real system

Record MoEs to generate KPPs

VIRTUAL PROTOTYPING:Why use a video game?


8/18

BACKGROUND: UGV Perception SystemsSUMET and ROS Small Unity Mobility Enhancement Technologies (SUMET)

developed an advanced perception system (APS) forunmanned ground vehicle based on stereo perception algorithms.

The SUMET system is implemented in the Robotic Operating System

(ROS). ROS is a meta-operating system enabling effective

integration of re-use of software elements common to robotic

systems.

Autonomous off-road navigation


9/18

Auto ID,

track & fire-

solution- but -

Operator

decides to

engage

Purpose of SAF-T Weaponization of unmanned systems,

and increase effectiveness of RWS

Autonomously identify, track, and

computes firing solutions for targets

Improved use of RWS requires a new C2 paradigm Systems suffer from limited SA and latency.

Technologies such as shot detection provide

recognized value-added to the warfighter.

Simulation-based approach used to defineparameters of desired future system Leveraging commercial video game

software and existing tools

Focus on Warfighter-System interaction

Shoot more stuff, faster, using a computer

BACKGROUND: Advanced Target Detection and Tracking


10/18

Unity3D

Unity3D is a video game engine developed by Unity Technologies,providing a convenient content-oriented editor.

Freeware and low-cost licensing options are available.

Unity3D is being using by the Supervised Autonomous Fires Technology (SAF-T)

program to virtually prototyped an unmanned weapons system, and through this

effort has integrated ROS with Unity3D.

Related Works Virtual Environments for Cognitive Architecture Development (VECAD - ONR)

Robotics Interactive Visualization and Experimentation Toolkit (RIVET - GDRS)

BACKGROUND: Tools and Related Works

Leverage open source and freeware


11/18

EXPERIMENT DESIGN & SETUP

Created test-board in the lab, and virtual world within

Unity3D using the Unity Editor. Created textures for the virtual world with still photographs of the real scene.

Created real and virtual world


12/18

12


Created test-board in the lab, and virtual world within

Unity3D using the Unity Editor. Cameras implemented using 115mm baseline, with FOV matching the real

system, with resolution of 656x492 pixels, and 3.5mm (68.82 degree FOV)

Stereo calibration parameters were shared between real and virtual system.

Created real and virtual world


13/18


Real and Simulated Systems run the same ROS code to

calculate stereo disparity from imagery Real system uses imagery generated by the stereo camera system Simulated system used imagery generated by the Unity3D simulation

Unity3D communicates to the ROS system via a custom ROS driver node that

communicates via the RTP protocol using a UDP Client / Server relationship.

Real orVirtual

cameras

produce the

same ROS

Topics


14/18

Real and Simulated Systems run the same ROS code to

calculate stereo disparity from imagery

RESULTS & CONCLUSIONS

Real System Synthetic SystemDisparity Difference

Disparity Map

Temporal di ff. over 100

images [1.2 per pixel]

Temporal di ff. over 100

images [~ 0 per pixel]

Disparity MapAverage di ff. over 100

images [3.5 per pixel]

Std. dev. diff. over 100

images [.006 per pixel]

Uncharacterized noise in real-world data

ImageResolu

tion=656x492

pixels:322752

totalpixelsinim

age


15/18

See notes for sl ide explanation

ViPER acts as a synthetic sensor

APPLICATION: ViPER usage in the SAF-T program


16/18

Warfighter Workshops Used simulator to motivatetalks with Marine RWS operators.

Marines played simulator in TO

and SA mode, commentedon concept and design,

informing algorithm requirements

User-based Experimentation

IRB protocol for participants to play scenario in TO and SA mode. Recording overall measures of performance at di fferent error levels

to determine necessary false posit ive and false negative rates of the

detection and tracking pipeline to maintain system capabil ity

APPLICATION: ViPER usage in the SAF-T Program

Inform system level design requirements


17/18

APPLICATION: ViPER usage in the SAF-T Program

Generate synthetic data for algorithm dev.


18/18

QUESTIONS:What else would you like to know?

Playing games with the development of

machine vision algorithms using ViPER

Presented by:

Benjamin Wheeler540.653.6030 (office phone)

540.642.2973 (mobile phone)

[email protected]

Documents

Playing games with the development of machine vision algorithms using ViPER