Tools for Programming in the Real World Self Introduction for
MSRA Fellowship 2012 Jun Kato, The University of Tokyo 1
Slide 3
Jun Kato 2 nd - year PhD candidate Igarashi Laboratory, The
University of Tokyo Concentration: Human-Computer Interaction
Especially, Tools for Programming in the Real World 2
Slide 4
Agenda My motivation for the research Introduction of the
projects Ideas for new projects 3
Slide 5
MY MOTIVATION FOR THE RESEARCH Tools for Programming in the
Real World 4
Slide 6
CRISTAL [SIGGRAPH09 E-Tech] Tools for Programming in the Real
World Multi-touch [CHI09 SRC] Roboko [ CHI13 (to be submitted)]
Sharedo [Ro-MAN12 (submitted)] Phybots [DIS12] Surfboard [UIST10
Demo] IntelliViewer [UIST10 SIC] Pressing [UIST09 SIC] DejaVu
[UIST12 (to appear)] 5
Slide 7
Sharedo [Ro-MAN12 (submitted)] Roboko [ CHI13 (to be
submitted)] DejaVu [UIST12 (to appear)] Phybots [DIS12] CRISTAL
[SIGGRAPH09 E-Tech] Multi-touch [CHI09 SRC] Phybots [DIS12]
Surfboard [UIST10 Demo] IntelliViewer [UIST10 SIC] Pressing [UIST09
SIC] CRISTAL [SIGGRAPH09 E-Tech] Multi-touch [CHI09 SRC] In the
Real World From Robot to Human 6
Slide 8
Sharedo [Ro-MAN12 (submitted)] Roboko [ CHI13 (to be
submitted)] Phybots [DIS12] DejaVu [UIST12 (to appear)] CRISTAL
[SIGGRAPH09 E-Tech] Multi-touch [CHI09 SRC] Roboko [ CHI13 (to be
submitted)] Surfboard [UIST10 Demo] IntelliViewer [UIST10 SIC]
Pressing [UIST09 SIC] DejaVu [UIST12 (to appear)] In the Real World
From Robot to Human 7
Slide 9
Surfboard [UIST10 Demo] IntelliViewer [UIST10 SIC] Pressing
[UIST09 SIC] Roboko [ CHI13 (to be submitted)] Phybots [DIS12]
DejaVu [UIST12 (to appear)] Multi-touch [CHI09 SRC] Roboko [ CHI13
(to be submitted)] Phybots [DIS12] DejaVu [UIST12 (to appear)]
CRISTAL [SIGGRAPH09 E-Tech] Multi-touch [CHI09 SRC] Sharedo
[Ro-MAN12 (submitted)] Tools for Programming From Keyboard to IDE
8
Slide 10
Surfboard [UIST10 Demo] IntelliViewer [UIST10 SIC] Pressing
[UIST09 SIC] Roboko [ CHI13 (to be submitted)] Phybots [DIS12]
DejaVu [UIST12 (to appear)] Multi-touch [CHI09 SRC] Surfboard
[UIST10 Demo] IntelliViewer [UIST10 SIC] Pressing [UIST09 SIC]
CRISTAL [SIGGRAPH09 E-Tech] Sharedo [Ro-MAN12 (submitted)] Tools
for Programming From Keyboard to IDE 9
Slide 11
INTRODUCTION OF THE (SELECTED) PROJECTS Tools for Programming
in the Real World 10
Goal of Phybots Add mobility to physical objects In a cheap and
easy way Through high-level and extendible API With support for the
whole prototyping process 14
Slide 16
Prototyping with Phybots 15 Hardware construction Programming
with built-in API Test with runtime debug tool (Extend API when
needed)
Slide 17
Localization and Locomotion API Hardware setup: easy and cheap
Camera PC or Mac Robotic things 16 From our user study: Miniature
drive recorder
Slide 18
Localization and Locomotion API Navigation by global
coordinates Move Push TracePath 17 Task move = new Move( mouseX,
mouseY); move.assign(robot); move.start(); From our user study:
Beach flags with obstacles Y-axis X-axis
Slide 19
Localization and Locomotion API Vector field navigation Easy
design of new behavior ex) Follow another robot Combination of
existing fields ex) Move + collision avoidance 18
User Studies Alpha version deployment To graduate students
Current version deployment To HCI students To robotics students
20
Slide 22
Lessons learned: Mere mobility was not enough. Most apps only
used a single API call. Parameter configuration was painful. Alpha
version deployment 21 Lessons learned: Extendibility is important.
Higher-level task management is desired. Support for testing phase
is needed.
Slide 23
Current version deployment 22
Slide 24
Phybots A toolkit for prototyping robotic things Localization
and Locomotion API Extensible Architecture Runtime Debug Tool
Open-source software available at
http://phybots.comhttp://phybots.com 23
Slide 25
Projects Phybots DejaVu Roboko 24
Slide 26
Motivation Input by the users hands, body, or physical objects
Captured by cameras Processed in real-time Support for
computer-vision- based programming is needed Wang, 2006 Wilson,
2005 Cao, 2003 25
Slide 27
Existing approaches Hardware, Software API Becomes more and
more accessible Webcam, Kinect, OpenCV, etc. Development
Environments Do not provide explicit support Ex) Breakpoint-based
variable monitoring OpenCV Visual Studio Eclipse 26
Slide 28
DejaVu 27
Slide 29
DejaVu Canvas Frame data visualization Freehand sketch 28
Slide 30
DejaVu Canvas Capable of showing variable values at the
different positions in the code 29
Slide 31
DejaVu Timeline 30
Slide 32
Conclusion DejaVu provides integrated support for the
development of interactive camera-based programs that tightly
matches their nature: Visual/Continuous/Non-reproducible the
workflow: Record input/Change code/Replay CanvasWorkflow 31
Slide 33
Projects Phybots DejaVu Roboko 32
Slide 34
Programming robot actions is difficult. 33 DanceClearing
dishesCooking Motivation:
Slide 35
Text-based Programming joint[0].setAngle(0.6);
joint[1].setAngle(0.7); joint[2].setAngle(-0.24);
arm.setPosition(4.3, 2.4); robot1.setPose(pose32); 34 Problem: We
cannot imagine the actual pose of the robot Typical approach:
Slide 36
Visual Editing 35 Problem: We cannot design flexible robot
behavior such as repeat this for 3 times when Typical
approach:
Slide 37
Overview of the IDE 36
Slide 38
Workflow of the Programmer 37 1. Capture2. Drag and Drop Camera
RobotPC
Slide 39
Getting and setting pose 38 Supported APIs:
Slide 40
Execute a sequential action 39 Supported APIs:
Slide 41
Not only a robot but a human (Kinect) 40 Project outcome: new
application This work is an outcome of collaboration with a MSR
researcher, Xiang Cao
Slide 42
Integrating visual representations into text-based programming
environment 41
Slide 43
IDEAS FOR NEW PROJECTS Tools for Programming in the Real World
42
Slide 44
Ideas for New Projects (CENSORED) 43
Slide 45
Thank you for listening Ive designed various tools for
programming in the real world. I am looking forward to further
collaboration with MSRA researchers. 44