The Fawkes Robot Software Framework - Universität Kassel · PDF fileNov 28th 2016 – MSL Workshop 2016 The Fawkes Robot Software Framework Software Stack for the RoboCup Logistics

Nov 28th 2016 – MSL Workshop 2016

The Fawkes Robot Software FrameworkSoftware Stack for the RoboCup Logistics League

Matthias LöbachTim Niemueller

Carologistics

RoboCup participant since 2012Joint team of FH Aachen University of Applied Sciences &RWTH Aachen UniversityWinner of the RoboCup Logistics League2014, 2015 and 2016

1 / 30

1 RoboCup Logistics League

2 Fawkes Robot Software Framework

3 Software Components

4 Conclusion

2 / 30

RoboCup Logistics League - At a Glance

RoboCup Logistics League (RCLL)

In-factory manufacturing logisticsin Smart FactoryMaintain and optimize materialflow in productionMulti-Robot planning/schedulingand coordination

Goals

Production logistics autonomyBenchmarking of robots in aSmart FactoryLong-term autonomy withreasoning/planning systems

3 / 30

RoboCup Logistics League

Physical processing machines based on Festo MPSAlmost 250 product variants

4 / 30

RoboCup Logistics League

Playing Field

Team colors: cyan and magentaExclusive machines randomly spreadacross fieldMirrored at middle axis

4 / 30

RoboCup Logistics League - Game Phases

Setup Phase

Duration: 5 minutesTeams prepare robots in insertion zone

Exploration Phase

Duration: 4 minutesPoints for correct light signal reportsPoints for correct machine position

Production Phase

Duration: 15 minutesPoints for delivered productsPoints for intermediate steps

5 / 30

RoboCup Logistics League - Production Machines

Mobile production machines

Conveyor as contact point forrobotsAR-Tag for identification andlocalizationSignal light for identification andstatus

6 / 30

RoboCup Logistics League - Robots

Robotino 3 platform

Laser range finderAdditional laptopCustom gripperCameras

Signal Camera

Custom Gripper

Additional Laptop

Conveyor Camera

Marker Camera

Internal Computer

Laser Scanner

7 / 30

RoboCup Logistics League - Referee Box

Game ControlMaintain game state/score

CommunicationPublish production plans

Data RecordingCollect benchmarking data

Visualization and InstructionReferee/visitor monitoring

Machinery ControlInstruct field machines

Logs game information and all communication

8 / 30

RoboCup Logistics League – Production Example

Get base with capFill cap station with capBring surplus base to ring station asresourceRetrieve base with ordered color at basestationInsert base into cap station for assemblyRetrieve assembled product from capstationDeliver product at delivery station

Most simple product has seven steps

BS CS 2

CS

RS

BS

DS

9 / 30


Get base with cap

Fill cap station with capBring surplus base to ring station asresourceRetrieve base with ordered color at basestationInsert base into cap station for assemblyRetrieve assembled product from capstationDeliver product at delivery station


BS CS 2

CS

RS

BS

DS

9 / 30


Get base with capFill cap station with cap

Bring surplus base to ring station asresourceRetrieve base with ordered color at basestationInsert base into cap station for assemblyRetrieve assembled product from capstationDeliver product at delivery station


BS CS 2

CS

RS

BS

DS

9 / 30


Get base with capFill cap station with capBring surplus base to ring station asresource

Retrieve base with ordered color at basestationInsert base into cap station for assemblyRetrieve assembled product from capstationDeliver product at delivery station


BS CS 2

CS

RS

BS

DS

9 / 30


Get base with capFill cap station with capBring surplus base to ring station asresourceRetrieve base with ordered color at basestation

Insert base into cap station for assemblyRetrieve assembled product from capstationDeliver product at delivery station


BS CS 2

CS

RS

BS

DS

9 / 30


Get base with capFill cap station with capBring surplus base to ring station asresourceRetrieve base with ordered color at basestationInsert base into cap station for assembly

Retrieve assembled product from capstationDeliver product at delivery station


BS CS 2

CS

RS

BS

DS

9 / 30


Get base with capFill cap station with capBring surplus base to ring station asresourceRetrieve base with ordered color at basestationInsert base into cap station for assemblyRetrieve assembled product from capstation

Deliver product at delivery station


BS CS 2

CS

RS

BS

DS

9 / 30




BS CS 2

CS

RS

BS

DS

9 / 30




BS CS 2

CS

RS

BS

DS

9 / 30




BS CS 2

CS

RS

BS

DS

9 / 30




4 Conclusion

10 / 30

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

G

W

C��

��

11 / 30

Integration

Middleware

Interconnect software componentsMake data accessible and observableStructure the data

Framework

Run-time organization and execution flowAssert certain properties of the system

Toolbox

Provide libraries for typical robotics taskIntegrate libraries and make available through framework

12 / 30

Integration

Middleware


Framework


Toolbox


12 / 30

Integration

Middleware


Framework


Toolbox


12 / 30

Fawkes at a Glance

Component-based architecture (plugins)Hybrid BlackBoard/messaging data exchangeMulti-threaded and distributableAspect-oriented framework feature accessStructured and synchronized main loopAutomated coordinate transform systemWeb interface for introspection and control

13 / 30

Run-time Coordination

Fawkes provides a main loopThreads can be hooked into themain loopThreads can also run concurrentlyMain loop is replaceableThreads for each hook are wokenup concurrentlyThreads sleep during execution ofother hooks

Pre Loop

Sensor/Vision

Sensor Proc.

World State

Think

Skill

Act

Act Exec

Post Loop

ConcurrentThread

14 / 30

Run-time Coordination

Fawkes provides a main loopThreads can be hooked into themain loopThreads can also run concurrentlyMain loop is replaceableThreads for each hook are wokenup concurrentlyThreads sleep during execution ofother hooks

Pre Loop

Sensor/Vision

Sensor Proc.

World State

Think

Skill

Act

Act Exec

Post Loop

ConcurrentThread

14 / 30

BlackBoard

Fawkes Process

BlackBoard

Interface A

Interface B

Thread 1

Thread 2

Remote App

BlackBoard created by Fawkes main application

15 / 30

BlackBoard

Fawkes Process

BlackBoard

Interface A

Interface B

Thread 1

Thread 2

Remote App

BlackBoard created by Fawkes main applicationInterface storage in the BlackBoard memoryInterface definition via XML (fields/messages)

15 / 30

BlackBoard

Fawkes Process

BlackBoard

Interface A

Interface B

Thread 1

Thread 2

writing

Remote App

Fawkes Threads access the BlackBoard via these Interfaces

15 / 30

BlackBoard

Fawkes Process

BlackBoard

Interface A

Interface B

Thread 1

Thread 2

reading

Remote App

Fawkes Threads access the BlackBoard via these Interfaces

15 / 30

BlackBoard

Fawkes Process

BlackBoard

Interface A

Interface B

Thread 1

Thread 2

Remote App

Fawkes Threads access the BlackBoard via these InterfacesRemote applications can access BlackBoard via networkTransactions (read/write)Only one writer at a time

15 / 30

BlackBoard

Fawkes Process

BlackBoard

Interface A

Interface B

Thread 1

Thread 2

Remote App

Message passing as command channelMessages can only be sent from reader to writerAny number of messages in queue

15 / 30

Plugins

ComponentsProvides specific functionalityIdeally: parameterizable blackboxCan – ideally – be easily replacedInterconnected through middleware

PluginsDynamically loadable librariesSet of threads and their initializationFramework can introspect threads

Soft Guarantee for Plugins⇒ Either all threads are successfully

initialized, or none is ever started

Plugin

Thread 1

Thread 2

· · ·

16 / 30

Plugins





Plugin

Thread 1

Thread 2

· · ·

16 / 30

Plugins





RobotinoPlugin

ComThread

SensorThread

ActThread

16 / 30

Plugins





LaserFilterPlugin

FilterThread 1

FilterThread 2

FilterThread 3

· · ·(dynamically allocatedbased on config)

16 / 30

Aspects

Framework FeaturesThreads must access featuresClassic: inquire/get featuresControl executed by requester

⇒ Framework has only limited information

AspectsPlugin threads denote requirementsFramework initializes aspectsSoft guarantee of feature availability

ImplementationAspect as simple classThreads inherit aspect classFramework asserts initialization

Framework

Plugin/Thread

get

17 / 30

Aspects





Framework

Plugin/Thread

Aspects

Req

uire

men

ts

Assert A

spects

17 / 30

Aspects





LaserLinesThread

ClockAspect

LoggingAspect

BlackBoardAspect

TransformAspect

PointCloudAspect

· · ·

17 / 30

Transforms

18 / 30

Transforms

Various coordinate reference framesKeep data in local frame for accuracyNeed transformations to act onsensed objects

Generalized transformation system(ported from ROS tf2)Tree of linked frames

18 / 30

Transforms Graph

Represent transforms as graph(tree)Walk the treeto calculate transformPerforms time matching andinterpolationAPI to transform points, posesetc.Static vs. dynamic transforms

API from C++, Lua, and CLIPS

19 / 30

Behavioral Architecture

Agent

Behavior Engine

Vision · · ·

Localization Motion ComponentsActuators/SensorsData Processing

Reactive BehaviorsSkill execution/monitoring

DeliberationDecision making/planning

20 / 30

Behavioral Architecture

Agent

Behavior Engine

Vision · · ·

Localization Motion Blac

kboa

rdor

RO

STo

pics

20 / 30

Webview

21 / 30

Webview

21 / 30

Webview

21 / 30

Webview

21 / 30

Webview

21 / 30

Webview

21 / 30

Further Features

Text and data logging facilitiesConfiguration sub-systemNetwork messaging infrastructure and discoveryROS integrationPlugins for performance analysis (RRD, mongodb-log, ...)

⇒ Batteries included

22 / 30




4 Conclusion

23 / 30

Perception

Machine SignalRecognize light signal on MPSCombine laser and image data todetect signalRobust to disturbances

Conveyor DetectionDetect conveyor on MPSBased on point clouds from 3DvisionUsed for precise positioning

24 / 30

Navigation

Self-localizationAdaptive Monte Carlo LocalizationParticle filter on posesWorks on 2D laser data

Global Path PlanningTopological graph searchProvides points of interestStatically collision-free paths

Local Path PlanningGrid map approach on laser dataAvoid dynamic and static obstaclesTake global path as guide line

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

G

W

C��

��

25 / 30

Simulation

Gazebo-based SimulationFull 3D simulation with physicsBased on well-known Gazebosimulator

RCLL in SimulationUse referee box forenvironment agencyVery close to the real game

Gazebo

LLSF Environment

Models

Gazebo Robot 1Motor, Laser, Cam, ...



Gazeb

oAPI

Referee Box

Visualization

Robot 2Fawkes, ROS,

...

Robot 1Fawkes, ROS,

...

Robot 3Fawkes, ROS,

...

26 / 30

Lua-based Behavior Engine

Basic actions for reasoning layerEmphasize description over programmingAllow programming where necessaryModeled using Hybrid State MachinesAbstract low-level systemImplemented for Fawkes and ROSWritten in the Lua scripting language

Variable tablex 5.2y 4.3error· · ·

27 / 30

Incremental Task-Level Reasoning

Only commit to single step at a timeStrategic behavior with coarse tasksReason about current knowledge

CLIPS rule-based systemEfficient reasoning with many updatesDistributed, local-scope, incremental

Vision · · ·

AMCL Motion

Behavior Engine

CLIPS Agent

(defrule s1-t23-s0(state IDLE) (holding S1)(machine (mtype ?mt&M2\_3) (name ?n)

(loaded-with $?lw&:(contains$ S0 ?lw)) )=>(assert (task-candidate goto ?n))

)

28 / 30




4 Conclusion

29 / 30

Conclusion and Questions

Fawkes as a versatile software framework is the foundationfor the the publicly released Carologistics software stack.

Hybrid blackboard middlewareMassively multi-threaded software componentsVersatile Behavior Engine and reasoning agentFocus on integration with reasoning componentsRCLL software stack released as open source software

https://www.fawkesrobotics.org/https://www.carologistics.org/

30 / 30

https://www.fawkesrobotics.org/

https://www.carologistics.org/

Documents

The Fawkes Robot Software Framework - Universität Kassel · PDF fileNov 28th 2016 – MSL Workshop 2016 The Fawkes Robot Software Framework Software Stack for the RoboCup Logistics