Assembly Simulation on Collaborative Haptic Virtual Environments

Rosa Iglesias, Elisa Prada Sara Casado, Teresa Gutierrez

Ainhoa Uribe, Alejandro Garcia-Alonso

Fundacion Labein

Universidad del Pais Vasco (UPV-EHU)

Outline

• Objectives

• Haptic devices, haptic rendering

• Haptic Assembly Simulation

• Collaborative Haptic Assembly Simulation

• Network Topologies

• Client/Server (two implementations)

• P2P

• Conclusions

GOAL: Create a Distributed Virtual Environment for collaboration, whereby users can carry out assembly and maintenance tasks

• VE Assembly + simultaneously + haptic devices • to design and evaluate computer generated mock-ups prior to

building any physical prototype.

Collaborative Haptic V Environment for Assembly Tasks

Haptic devices

Haptic rendering

• “Haptic rendering is the process of computing and

generating forces in response to user interactions with virtual objects” (Salisbury, 1995).

• “Haptic rendering allows users to “feel” virtual objects in a simulated environment” (Salisbury, 2004)– To touch objects, move them,…– To create different haptic effects: texture perception,

deformable elements or collision impacts.

• Haptic devices require an update frequency of 1000 Hz.

Sphere Section

1 kHz

Haptic ThreadHaptic Thread

Get position P

Send force

Touch:

-check if P is inside

-if inside, compute the contact point on the surface

-compute force

NormalVector

User’s point

Touching a sphere

Sphere Section

1 kHz

Haptic ThreadHaptic Thread

Get position P

Send force

Touch:

-check if P is inside

-if inside, compute the contact point on the surface

-compute force

NormalVector

User’s point

Touching a sphere

1 kHz

Haptic ThreadHaptic Thread

Get position P

Send force

Touch:

-check if P is inside

-if inside, compute the contact point on the surface

-compute force

1 kHz

Haptic ThreadHaptic Thread

Get position P

Send force

Touch:

-check if P is inside

-if inside, compute the contact point on the surface

-compute force

NormalVector

User’s point

NormalVector

User’s point

Touching a sphere

Haptic loopHaptic loop

Haptic rendering

• An update frequency of 1 KHz means:

Haptic assembly simulation (HAS)

• Description

video

Introduction: HAS

• Examples of assembly constraints:– Assembly along an axis

– Assembly along a common plane

Assembly Assembly LineLine Hole

Pin

Hole

Pin

Hole

Pin

Assembly Assembly LineLine

Assembly Assembly LineLine Hole

Pin

Hole

Pin

Hole

Pin

Hole

Pin

Hole

Pin

Hole

Pin

Hole

Pin

Assembly PlaneAssembly PlaneAssembly PlaneAssembly PlaneAssembly PlaneAssembly Plane

Haptics require new architectures

• Classical simulation loop :

– Process input events (move objects)– Simulation– Image rendering– Haptic rendering ??? it is not possible

• Due to high frequency of haptic rendering: required two independent processes (threads).

HAS : two processes

• Get last position

• Simulation (collisions, constraints, …)

• Image rendering

• Send “haptic requirements”

• Read last position (from haptic device)

• Send last position

• “haptic requirements” received ?

• Compute force (haptic rendering)

• Send force to haptic device

Simulation (20-60Hz) Haptic (1000 Hz)

Problems : (1) haptic cycles without simulation updates, (2) updates mean brusque changes (haptic feedback instability)

Objectives & Challenges

- Broad objective: Several users can work Broad objective: Several users can work simultaneouslysimultaneously to to

achieve a common goal using haptic devices in assembly achieve a common goal using haptic devices in assembly

operations.operations.

- Challenges:Challenges:

• Consistency (virtual scene synchronization) must be guaranteed,

because users simultaneously interact with the same scene

• Effective and stable haptic feedback when users collide or assemble

their grasped objects.

• Scalability: number of users, virtual objects and so on.

Objectives & Challenges

• KEY FACTOR: Network conditions– Delay– Variation of delay (jitter)– Message Loss,….

• They affect: VE consistency and user perception– Visually:

• Sluggish scene update of the remote object

– Haptically• Unstable haptic feedback

Objectives & Challenges in CHAS

• GOAL: Collaborative Haptic Assembly Simulator GOAL: Collaborative Haptic Assembly Simulator (CHAS), (CHAS), whereby two users can simultaneously carry out assembly tasks using haptic devices.

• Two types of interaction:– INDEPENDENT INTERACTION:

interacting with static objects (not grasped).– DEPENDENT INTERACTION:

depends on other user’s action.

Objectives & Challenges in CHAS

• Two types of dependent interaction:– Dependent collision: Dependent collision:

when both grasped objects collidewhen both grasped objects collide

– Remote assembly: Remote assembly: when a grasped object is being assembled into other grasped when a grasped object is being assembled into other grasped

objectobject

CHALLENGES in CHASCHALLENGES in CHASIn the case of a remote assembly + dep. coll.: • consistency must be guaranteed.

• each user must receive adequate haptic feedback

To build a distributed VE:

Client/Server (C/S)

Peer-to-Peer (P2P)

Mixture of themC/S

P2P

Network topology

Multiple Servers

Client/Server (A1)

• PROS: Consistency is automatically satisfied,

since data are validated centrally,

and then distributed to clients.

• CONS: computations for simulation are

made sequentially

Client 1

Network

Server

Visualization

InteractionLocal DB

Client 2

TouchSound Sight

Central DB

Administration

Simulation

Client n

Client 1

Network

Server

Visualization

InteractionLocal DB

Client 2

TouchTouchSoundSound SightSight

Central DB

Administration

Simulation

Client n

Simulation

AdministrationDB

Local DBInteraction

Visualisation

Client 1

Network

Server

Visualization

InteractionLocal DB

Client 2

TouchSound Sight

Central DB

Administration

Simulation

Client n

Client 1

Network

Server

Visualization

InteractionLocal DB

Client 2

TouchTouchSoundSound SightSight

Central DB

Administration

Simulation

Client n

Simulation

AdministrationDBDB

Local DBLocal DBInteraction

Visualisation

A1

Simulation or validation: check if new object position is colliding or computes the assembly constraints

Simulation

Client/Server (A3)

Observing its disadvantage: sequential computation.

A new C/S architecture was designed.• PROS: well-balanced distribution of the workload

and parallel computation.

• CONS: efficiency comes from the worst

computer conditions

Client 1

Network

Server

Visualization

InteractionLocal DB

Client 2

TouchSound Sight

Central DB

Administration

Simulation

Client n

Client 1

Network

Server

Visualization

InteractionLocal DB

Client 2

TouchTouchSoundSound SightSight

Central DB

Administration

Simulation

Client n

Consistency

AdministrationDB

Local DB

Interaction

Visualisation

Simulation

Client 1

Network

Server

Visualization

InteractionLocal DB

Client 2

TouchSound Sight

Central DB

Administration

Simulation

Client n

Client 1

Network

Server

Visualization

InteractionLocal DB

Client 2

TouchTouchSoundSound SightSight

Central DB

Administration

Simulation

Client n

Consistency

AdministrationDBDB

Local DBLocal DB

Interaction

Visualisation

Simulation

A2

Consistency: server checks if there is any inter-object penetration. In that case, synchronise user’s scenes

Consistency

Simulation

Network topology: Analysis using C/S

RESULTS using C/S architectures:

• With A1 and A3:

several users can simultaneously interact using keyboard/mouse.

• A3 seemed to have better performance when > 2 users

• A user can haptically interact while others watch with A3.

• After experiments, it was shown that simulation/validation must be

placed at each user for avoiding unstable haptic feedback, even

with independent interaction.

• A1 and A3 are very sensitive to the network delay.

Network topology: Analysis using C/S

• They address: – collaboration taking turns (any delay)

– cooperative manipulation = a carrying task

• Two users grasping the same object.

• Only low-delay interaction between both users, because

haptic interaction is affected.

• Two goals still not achieved :– simultaneous assembly tasks– high delay

Network topology: Analysis C/S vs P2P

Comparing C/S and P2P:• C/S: Consistency is more easily achieved.

Local scene update after a round-trip delay to the server.

• P2P: Consistency challenge.

Update delay is only one-way - Updates are less dependent on network

conditions.

- Permit a better performance with haptic interaction.

- Potential to scale to a larger number of users.

Then, a P2P architecture is chosen

Consistency-maintenance scheme

• In our research, to build the CCollaborative HHaptic

AAssembly SSimulator (CHASCHAS), whereby two users can

simultaneously carry out assembly tasks using

haptic devices.

• A new consistency-maintenance scheme has been

designed and tested.

Consistency-maintenance scheme

Main features:

• Gives priority to the validation of interactions between objects grasped by users for fast haptic response.

• Small messages size: < 200 bytes

• Not need to store and manage a history of movements.

• Not specific messages to deal with inconsistency because

each user manages consistency locally.

Experimental set-up

• Many experiments were carried out varying end-to-

end delays: less than 1 ms, 50 ms, 100 ms, 200 ms

and with jitter (0 ms – 100 ms).

• The experimental platform:

Switch 1Switch 2

NetDisturb

Switch 1Switch 2

NetDisturb

Switch 1Switch 2

NetDisturb

Experimental set-up

• The experimental platform: – Experimental platform: 3 computers via a

100Mbps links connected by two high-speed switches.

– Two computers are each connected to an OMNI device.

– The other computer to simulate the different values of delay and jitter.

Switch 1Switch 2

NetDisturb

Switch 1Switch 2

NetDisturb

Switch 1Switch 2

NetDisturb

Results between Labein & QUB: VE & tasks

Virtual scene: Electrical box for an aircraft engine

•User Labein grasps the handle

•User Queen’s grasps

the screw

•Perform collisions

•Assemble both grasped

objects

USER QUEEN’S

USER LABEIN

USER QUEEN’S

USER LABEIN

CAD design: Electrical box for an aircraft engine by SENER

Results between Labein & QUB: Consistency

• Consistency after a dependent collision

SCREW X-POSITION (LABEIN & QUEEN'S) ( First 1200 ms)

-50

-30

-10

time(ms)mm

screw at User Queen's screw at user Labein

501 1001

Results between Labein & QUB: Forces

• Collision force magnitude

after a dependent collision

User Queen's (screw): collision force magnitude (Dependent Collision)

0,0

0,2

0,4

0,6

0,8

1,0

1 11 21 31 41 51 61 71 81 91time (ms)

Newton averaged force

Results between Labein & QUB: Forces

• Force magnitudes

during a remote assembly

User Queen's (screw): assembly force magnitude

0

0,3

0,6

0,9

1 401 801 1201 1601 2001 2401 2801 3201

time (ms)

Newton insertion

Results between Labein & QUB: Conclusions

• With the consistency-maintenance scheme in CHAS, users

had a realistic sensation of the VE and the performance was

satisfactory.

• CHAS provided an adequate haptic interaction when both

users perform remote assemblies or dependent collisions.

• Such haptic feedback showed that the sense of co-presence

between the users continue to improve, with regard to only

visual feedback.

THE END

QUESTIONS ?

THANK YOU VERY MUCH !

MUCHAS GRACIAS !

ESKERRIK ASKO !

DONOSTIA-SAN SEBASTIAN