The Making of XRobots

Jan Köhnlein Miro Spönemann

The Idea

• A showcase for Xtext

• Robot wrestling: push the opponent off the arena or tip him over

• Players write programs to control the robots

• Web-based editor

• A distributed real-time system

lego.com

Lego Mindstorms EV3

4 motors

4 sensors

USB

Bluetooth

SD card slotlego.com

Unpacking

Robot Design

2 driving motors

2 arms with 1 motor

infrared sensor

color sensor

marker

Operating System

+ Executes Java programs

+ Supports network communication

+ Open source

o Eclipse plug-in

− Still beta

− More than 2 min. booting time

First choice: LeJOS (lejos.org)

Robot API

drive(10)

drive(-10)

10 cm

10 cm

driveForward

driveBackward

non-blocking

stop

stop

rotate(90) rotate(-90)

90° 90°

rotateLeft rotateRight

stop stop

non-blocking

scoop(1)

scoop(-1)

scoop(0)

The Arena

x axis

y axis

ownPosition

opponentPosition

ownPosition.viewDirection

opponentPosition.viewDirection

opponentBearing

centerBearing

IRobot.java

Robot.java (LeJOS implementation)

Networking

Communication

robots game server

•Keeps the game’s state

•Referee / Security

•Interprets player scripts in a sandbox

•Sends commands to robots

•Receives state from robots

➡Need for very fast communication

Server Side

Average Latency

Bluetooth java.io ~250ms

Bluetooth Java RMI ~120ms

Bluetooth java.nio ~40ms

WiFi (USB dongle) java.nio ~12ms

Communication Protocol

Client Server

Shared

Robot.java RobotExecutor.java RobotClientState.java

RemoteRobotProxy.java RobotServerState.java

IRobot.java

internal state

commands

• Programming language based on Java

• Write cleaner code

• Integrated in the Eclipse IDE

• Compiled to Java

• Active Annotations

xtend-lang.org

Generated RMI@SimpleRMIclass Robot {! def drive(double distance) { ... }! def rotate(double angle) { ... }!}

IRobot.java Robot.java RobotExecutor.java RobotClientState.java RemoteRobotProxy.java RobotServerState.java

Changing the Interface@SimpleRMIclass Robot {! def drive(double distance) {…}! def rotate(double angle) {…}!!!!!}

def setDrivingSpeed(double speed) {…}

def double getGroundColor() {…}

command protocol is updated

state protocol is updated

Scripting

• Define your own language

• Xtext generates a text editor with syntax highlighting, code completion, error markers, etc.

• Add a code generator or interpreter

xtext.org

An XRobots Scriptrobot 'Hello, World!'author Miro!TurnToOpponent on opponentBearing.length > 35 { rotate(opponentBearing.angle) drive(10)}!Destroy { if (abs(opponentBearing.angle) < 90) drive(100) else drive(-100)}

The XRobots Languagegrammar org.xtext.xrobot.dsl.XRobotDSL with org.eclipse.xtext.xbase.Xbasegenerate xRobotDSL ".../XRobotDSL"!Program: 'robot' (name=ID | name=STRING) 'author' (author=ID | author=STRING) (modes+=Mode)*;!Mode: name=ID ('on' condition=XExpression)? action=XBlockExpression;

Xbase: connect your language to the JVM

XRobots &

• Can use all expressions of Xtend

• A Program instance is mapped to a class implementing IRobot

• Can access all methods of IRobot: drive(double), rotate(double), etc.

Simple Game Loop public void gameLoop() { while(!isGameOver()) { readSensorData(); performSomeMove(); } }

• Enforces boring face and hit strategy

• Hard to implement sequences of moves

• Finite moves unappealing

Execution Model• Mode conditions are checked periodically

• The first mode with satisfied condition is the active mode

• If the active mode changes, the previous mode is canceled

• Cancellation is built into the runtime

XRobots Interpreter

RobotExecutor

XRobotInterpreter

internal state

commands

RobotClientState

RemoteRobotProxy

RobotServerState

XbaseInterpreter

Robot

Robot Vision

Infrared Sensor

Sender Receiverdetects angle (± 40°) and distance

lego.comlego.com

Infrared Sensor: Angle

-25 -20 -15 -10 -5 0 5 10 15 20 25

-40°

-30°

-20°

-10°

0°

10°

20°

30°

40°

-25.00 -20.00 -15.00 -10.00 -5.00 0.00 5.00 10.00 15.00 20.00 25.00

-40.00

-30.00

-20.00

-10.00

0.00

10.00

20.00

30.00

40.00

f(x) = 1.6981305442x + 3.3567996758

measured value

corresponding angle

lego.com

Infrared Sensor: DistanceCharts

Page 1

20 25 30 35 40 45 50 55 60

0 cm

20 cm

40 cm

60 cm

80 cm

100 cm

120 cm

measured valuecorresponding distance

lego.com

Image RecognitionImage processing library: OpenCV (opencv.org)

+ Read and process camera images + Very fast + Open source − Image recognition methods are quite fuzzy

Fiducial TrackingreacTIVision (reactivision.sourceforge.net)

•Tracking position and angle of fiducial markers

•Broadcast tracking data via UDP (TUIO protocol)

•C++ library

Perspective Correctioncamera

actual position

perceived position

{

github.com/franchi82/xtrack

Web Editor

Xtext Editors

Orion

freepik / CC BY

Server Browser

Eclipse + Xtext + Orion JavaScript

eclipse.org/orion

XRobots in the Web

• Click the button → server generates a script file

• Browser communicates with server for text coloring, code completion, error markers, etc.

xrobots.itemis.de

Executing Scripts

freepik / CC BY

game server displays token

A6X1

player enters token into browser

token is reserved on server

A6X1

game server requests script for token

The Big Picture

freepik / CC BY

xrobots.itemis.de

github.com/JanKoehnlein/XRobot

Server Display

JavaFX• Now included in the JRE

• Create graphics and sound effects

• Use CSS to specify visual properties

children += new Label => [ text = 'XRobots' styleClass += #['boxed-label', 'logo'] effect = new InnerShadow => [ color = Color.RED width = 3.2 height = 3.2 ]]

.boxed-label { -fx-background-radius: 3.0; -fx-padding: 4.0; -fx-text-fill: white;}

Image generated from camera tracking application

Graphics overlaid by JavaFX

Robots Can TalkText-to-speech library: FreeTTS (freetts.sourceforge.net)

say('Come get some!')

Come get some!

Security

Security?robot 'Dr. Evil'author Miro!BlackMagic { System.exit(0)}

shut down the game server

Security Manager

• Block access to System.exit, file system, network, threads, etc.

• Throws SecurityException ⇒ game over

System.setSecurityManager(...)

thank you