Design of a Multi-Threaded Distributed Telerobotic

Framework Mayez Al-Mouhamed, Onur Toker, and Asif IqbalMayez Al-Mouhamed, Onur Toker, and Asif Iqbal

College of Computer Science and EngineeringCollege of Computer Science and Engineering

King Fahd University of petroleum and MineralsKing Fahd University of petroleum and MineralsDhahran 31261Dhahran 31261

Kingdom of Saudi ArabiaKingdom of Saudi Arabia

OUTLINEOUTLINE

• Background on TeleroboticsBackground on Telerobotics• A multi-threaded distributed frameworkA multi-threaded distributed framework• Telerobotic server componentsTelerobotic server components

– PUMA componentPUMA component– Force componentForce component– Component hierarchy Component hierarchy – Integrated schemeIntegrated scheme

• Telerobotic client componentsTelerobotic client components– Integrated schemeIntegrated scheme

• Multi-threaded distributed telerobotic Multi-threaded distributed telerobotic systemsystem

• ConclusionConclusion

Background on Background on telerobotics telerobotics

• Telerobotics is an approach that extends a human operator’s sensing and manipulating capabilities to a remote environment.

• Telerobotics has numerous applications in hazardous or hostile environments.

• The purpose of this work is to design a reliable and efficient man-machine interface between a slave arm (robot) and a master arm over a network which preserves a large degree of operator dexterity.

• Computer vision and force feed techniques are implemented to enhance the maneuverability of the operator by providing him force feeling.

• To avoid communication delays in frequently interacting telerobotic systems, the operator will have a library of local automation tasks. Only coarse operator supervision is required. For example, picking up a tool, moving tool to a certain location, returning tool to initial position, etc.

• Telerobotics is a multidisciplinary area integrating knowledge from robotics, real-time operating systems and programming, network programming, 3D stereo vision, mechanical linkage engineering, ergonomics, etc.

A Multi-Threaded Distributed A Multi-Threaded Distributed FrameworkFramework

• Client-Server distributed component Client-Server distributed component telerobotic system.telerobotic system.

• A telrobotic server has components A telrobotic server has components (PUMA, Force Sensor, and Decision (PUMA, Force Sensor, and Decision Server) and interfaces (Proxy Robot, Server) and interfaces (Proxy Robot, Sensor, and DecisionServer).Sensor, and DecisionServer).

• One or more telerobotic client One or more telerobotic client componentscomponents

• An integrated scheme of client-server An integrated scheme of client-server componentscomponents

• A multi-threaded distributed A multi-threaded distributed telerobotic systemtelerobotic system

Telerobotic server Telerobotic server componentscomponents• PUMA ComponentPUMA Component• Acts as a software proxy (thread) of the Acts as a software proxy (thread) of the

slave robot arm diagram. slave robot arm diagram. • Public methods Public methods

• ConnectRobot, ConnectRobot, • InitializeRobot, InitializeRobot, • incremental joint and Cartesian motion commands, incremental joint and Cartesian motion commands, • tool and world frame motion coordination, etc.tool and world frame motion coordination, etc.

• Public propertiesPublic properties• Booleans that represent robot state. Booleans that represent robot state.

• Public EventsPublic Events• motion commandsmotion commands• communication and synchronization mechanisms.communication and synchronization mechanisms.

PUMA componentPUMA component

• Force Sensor ComponentForce Sensor Component• A service thread for a wrist force sensor A service thread for a wrist force sensor

attached to slave robot arm (3 force+3 attached to slave robot arm (3 force+3 moments)moments)

• Provides reflected force feedback from Provides reflected force feedback from slave arm (server) to operator (client) slave arm (server) to operator (client)

• Public propertiesPublic properties• start and stop readingstart and stop reading• TimerValue, etc.TimerValue, etc.• SensorThreadPriority, SensorThreadPriority,

• Real-time streaming of force data from Real-time streaming of force data from server to clientserver to client

Telerobotic server Telerobotic server componentscomponents

Component hierarchyComponent hierarchy

Integrated scheme - serverIntegrated scheme - server

Telerobotic Client Telerobotic Client ComponentsComponents• Client GUI (Graphic User Interface) for Client GUI (Graphic User Interface) for

remote testing and maintenance remote testing and maintenance operationsoperations

• IDecisionServerto interface to server IDecisionServerto interface to server through .NET Remotingthrough .NET Remoting

•All the definitions to execute methods on All the definitions to execute methods on PUMA and ForceSensor componentsPUMA and ForceSensor components

• After initialization, the client carries an After initialization, the client carries an empty un-referenced copy of IDecisionServerempty un-referenced copy of IDecisionServer

•Following the network connection, the client Following the network connection, the client can reference any instance of DecisionServercan reference any instance of DecisionServer

Integrated scheme - clientIntegrated scheme - client

Integrated Scheme of Client-Integrated Scheme of Client-Server ComponentsServer Components

• Inherits DecisionServer from Inherits DecisionServer from IDecisionServer, IProxyRobot, IDecisionServer, IProxyRobot, IForceSensor interfacesIForceSensor interfaces

• To invoke DecisionServer events, To invoke DecisionServer events, Shim Classes are used as agents to Shim Classes are used as agents to forward DecisionServer events to forward DecisionServer events to client.client.

A Multi-threaded A Multi-threaded distributeddistributed Telerobotic Telerobotic SystemSystem

• Simultaneous activation of many Simultaneous activation of many threads likethreads like

•Two digital cameras generate stereo pictures Two digital cameras generate stereo pictures which are sent to the clientwhich are sent to the client

•Grabbing and transfer of stereo video dataGrabbing and transfer of stereo video data

•Reading and transferring force sensor dataReading and transferring force sensor data

•Sending and receiving robot control signals Sending and receiving robot control signals over the LAN to one or more clients over the LAN to one or more clients

• Both the stereo data and the Both the stereo data and the distributed component calls share the distributed component calls share the same LAN using different ports for data same LAN using different ports for data transfertransfer

Integrated systemIntegrated system• DecisionServer component provides slave supervisory DecisionServer component provides slave supervisory

control.control.• Server Side Interfaces and .NET RemotingServer Side Interfaces and .NET Remoting• A set of definitions of public methods and properties. A set of definitions of public methods and properties. • Servers as a contract for any component that Servers as a contract for any component that

implements this interface. implements this interface. • Allows hiding the actual component to increase Allows hiding the actual component to increase

security. security. • Uses IProxyRobot and IForceSensor to communicate Uses IProxyRobot and IForceSensor to communicate

with PUMA and Force Sensor components with PUMA and Force Sensor components • IDecisionServer inherits both of the above interfaces. IDecisionServer inherits both of the above interfaces. • Allows defining a unified set of methods, properties, Allows defining a unified set of methods, properties,

and events within DecisionServer component. and events within DecisionServer component. • .NET Remoting publishes an instance of .NET Remoting publishes an instance of

DecisionServer component on the LAN. DecisionServer component on the LAN. • .NET Remoting enables access to remote objects .NET Remoting enables access to remote objects

using SOAP.using SOAP.

Events forwarding using Shim Events forwarding using Shim classesclasses

Server system of distributed Server system of distributed frameworkframework

Client system of distributed Client system of distributed frameworkframework

ConclusionConclusion

• A reliable and efficient man-machine interface between a slave robot (server) and a master arm (client) over a network while preserving some degree of operator dexterity

• A reliable distributed components (.NET Remoting) is implemented:

• Software reusability, ease of extensibility, debugging, and data Encapsulation

• Advanced software tools like .NET Framework (comp. to DCOM)

• The components communicate using .NET Remoting and SOAP

• Automatic handling of network resources and data transfer

• Isolation of components from network protocol issues

• Enhances data security as well as facilitates deployment

• Multi-threaded execution for multi-streaming of force, command and stereo data.