C omputer V ision and S ystems L aboratory SSGRR - 2000 UNIVERSITE LAVAL vision/ 1 SSGRR - 2000 International Conference on Advances

1

Computer Vision and

Systems Laboratory SSGRR - 2000UNIVERSITELAVAL

www.gel.ulaval.ca/~vision/

SSGRR - 2000International Conference on Advances in Infrastructure for Electronic Business, Science and Education on the Internet,

July 31 - August 6, L’Aquila, Italy

2

Computer Vision and



Designing Virtual Environments for Critical Transactions and Collaborative Interventions:

the VERTEX / APIA Framework for Networked, Physics-Compliant Objects

Denis PoussartDenis LaurendeauFrançois BernierMartin Simoneau

Nathalie HarrisonDenis OuelletChristian Moisan

Computer Vision and Systems Laboratory and Interventional MRI Unit, CHUQ, Université Laval

Québec, CANADA

Supported, in part, by grants from NSERC, FCAR, the Institute for Robotics and Intelligent Systems and the Canadian Foundation for Innovation

[email protected]


3

Computer Vision and



Critical Interventions represent cases where{errors, delays, lack of optimization}may have very negative consequences

for safetyfor the environmentfor healthfor costs …

In the future, as more and more complex situations arise, we may anticipate that operational support from Virtual Environments will become paramount and prevalent in the

planning training execution phases of delicate tasks

4

Computer Vision and



The inspection, maintenance and repair of hydroelectric facilities is just one example

5

Computer Vision and



Computer Graphics

Virtualizing “reality”

HumanMachine

Interaction

Computer Vision

Automated generation of

geometry of objects and scenes

6

Computer Vision and



But something is missing ...

For critical tasks, visual illusion is not sufficient.

There is more to “real things” than just shape, or forms, even if they are augmented with some “behaviors”.

Accurate physical modeling, laws, and simulation capabilities must be integrated within the virtual environment.

Reality includes PHYSICS!

7

Computer Vision and



Computer Graphics

Virtualizing “reality”

Computer Vision

HumanMachine

Interaction

Simulation

Physicsactual world

8

Computer Vision and



This opens up a huge question space!

what is relevant to be physically modeled?

what would be appropriate forms of models?

what level of detail?

perhaps multiple levels of detail, depending ...

how to develop scenarios ?

and brings about many integration issues ...

9

Computer Vision and



A project of Phase 3 of the Institute of Robotics and Intelligent Systems (IRIS) of the Network of Centers of Excellence program of Canada.

VERTEX: Virtual Environments: from 3D Representations to Task planning and EXecution

Objective is to optimize the execution of delicate tasks by combining the accurate simulation of actual scenes, tools and processes with advanced human machine interfaces.

CRIM

International Submarine Engineering Ltd.

We are exploring this approach in:

10

Computer Vision and



VERTEX

On site acquisition

11

Computer Vision and



Geometric modeling, 3D and photometric

ModelingModels of behaviors

Models of augmented scenes

Tools, materials, processes

VERTEX

On site acquisition

12

Computer Vision and







VERTEX

Planning

Task simulation in “VR” mode

Reactive Interaction Predictive evaluation

Task decomposition

Optimized scenarios

Acquisition sur le site

13

Computer Vision and



Planning



Task decomposition

Optimized scenarios





TrainingSimulated scenarios

VERTEX

Acquisition sur le site

14

Computer Vision and



ExecutionTask supervision &

Teleoperation in augmented VR

modeReal time control of robot and tools





On site acquisition

Planning



Task decomposition

Optimized scenarios


VERTEX

15

Computer Vision and








On site acquisition

VERTEX

Planning



Task decomposition

Optimized scenarios

ExecutionTask supervision & Teleoperation in augmented

VR modeReal time control of robot and tools

16

Computer Vision and



Design issue:

User-centric design

Who is the user?What are his / her needs?

Actually, complex interventions typically involve several users, of various types, with different needs, perspectives and internal models

Hix, D., Swan, E., Gabbard, J., McGee, M., Durbin, J., King, T. (1999) User-Centered Design and Evaluation of a Real-Time Battlefield Visualization Virtual Environment. In Proceedings of IEEE Virtual Reality '99

These users, acting cooperatively, might very well be in different locations

17

Computer Vision and



On site Acquisition

VERTEX

Modeling: off-line

Planning:interactive, soft real-time

VR

Execution: hard real-time

AR

18

Computer Vision and



it might just flow out of the action loop,

A key aspect of physics relates to the handling of time. Real time????

Design issue:

But different physical components may require different time resolution: run time optimization requires fine grain control of time.

and during the direct, immediate {supervision. control, execution} of the task (Augmented Reality), timing accuracy is mandatory: this is the realm of hard real time.

during strategic planning activities, it blends with predictive evaluation,

it may relate to factors which impact upon the user’s sense of interactivity, such as latency jitter,

In (critical) VE’s, time has many different flavors:

19

Computer Vision and



Design objectives:

The (physics) simulation engine is at the core of the

system

Beside its predictable real-time behavior, the engine should be capable of supporting:

dynamic extensibility (non - stop)

internal coherence, robustness

multi-resolution behavioral modeling

precisely known degradation

20

Computer Vision and



Design objectives (cont):

From an implementation point of view, the engine should seek

modularity, reusability

close match to current and foreseeable trends

- Moore’s law

- high-speed networking

capability to operate from heterogeneouscomponents with run-time binding

networked deployment (multiple users, geographica extent of tasks)

21

Computer Vision and



Implementation choice:

Use the Common Object Request Broker Architecture (CORBA) as the software bus - the glue - in assembling the Vertex system.

Why? CORBA is “heavy”, with significant overhead ...

True, but as time will unfold

complexity of relevant problems

CPU, network performance will both and

and the benefits of a robust architecture a more and more significant asset. Silicon, bandwidth free!

22

Computer Vision and



network

Actors * Properties * Interactions Architecture

APIA

To insure time accuracy and conformity to physics, we locate the main driving loop in the simulation engine, “away” from the HMI component.

*

23

Computer Vision and



network

APIA

Maintains an on-going representation of the “world”

Implemented on a cluster of COTS (à la Beowulf) Runs (preferably) on hard real-time OS (OS’s)

Physics engine

A Lego-like approach, with hierarchical capabilities

May include heterogeneous components

24

Computer Vision and



network

APIA

Overall management

Scenario authoring

Repository of model objects …

Controller

25

Computer Vision and



network

APIA

I/O links to the actual physical world

Sensors &Actuators

26

Computer Vision and



network

APIA

Multiple and different views / interactions easily implemented

To suit the representation levels required by different types of users

HMI’s

27

Computer Vision and



network

APIA

Provides the physical glue between the components

RT-CORBA provides the logical glue Designed to fully exploit high performance networking QOS, CaNet3

Geographically - distributed computing, users

28

Computer Vision and



CA*net3 IPv6

29

Computer Vision and



Implementation choice

Douglas C. Schmidt, Center for Distributed Object Computing,Washington University

Vertex uses ACE™ and TAO™, a CORBAimplementation under development at the Center for Distributed Object Computing,Washington University.

ACE / TAO is designed to support real-time networked applications, with rigorous control of task priorities and QOS.

30

Computer Vision and



VERTEX / APIA is currently being deployed in other areas, such as breast and liver cancer treatment through cryosurgery.

minimally invasive surgery shares many aspects of telerobotics,

a collaborative project with the Imaging Research Unit of Hopital St-François d’Assise (Dr. C. Moisan) and the Finite Element Research Group at Laval.

A generic approach ...

31

Computer Vision and



MRI Acquisition

3D Geometrical and Tissue Modeling

Modeling

Execution Task Supervision & Teleoperation in

AR modeReal-Time Cryogenic

Probe Control

VERTEX

Behavior Modeling

Models of Augmented Scenes

Tools, Materials and Processes

Planning

Task Simulation in VR mode

Reactive Interaction & Predictive Evaluation

Task Decomposition

Optimized Scenarios


32

Computer Vision and



On-going 3D visualization of cryoprobelocation and orientation

Real-time display of (simulated) cold front spatial distribution

33

Computer Vision and



Current status andfuture work

Video

Documents

C omputer V ision and S ystems L aboratory SSGRR - 2000 UNIVERSITE LAVAL vision/ 1 SSGRR - 2000 International Conference on Advances