Bryan WillimonClemson University 2013

Committee:Stan Birchfield (committee chair)

Ian Walker (co-advisor)Adam Hoover

Damon Woodard

Sensing Highly Non-Rigid Objects with RGBD Sensors for Robotic Systems

Robotic systems are divided into two groups• Industrial

• Recycling• Mail• Food• Laundry

• Domestic• Outdoor

• Lawn care• Pool cleaning• Gutter cleaning• Window cleaning

• Indoor• Floor cleaning• Pet assistance• Picking up objects• Laundry

Why Domestic Robots?• robotic systems are able to accomplish

chores around the house• Domestic robotic systems are getting more

attention in the news

• Domestic• Outdoor

• Lawn care• Pool cleaning• Gutter cleaning• Window cleaning

• Indoor• Floor cleaning• Pet assistance• Picking up objects• Laundry

• Domestic• Outdoor

• Lawn care• Pool cleaning• Gutter cleaning• Window cleaning

• Indoor• Floor cleaning• Pet assistance• Picking up objects• L

Laundry is an Important Problem

Laundry

Laundry is an Important Problem• Industries and research institutes are making

attempts to solve the process

PR2 at UC Berkeley

NEDO Laundry Handling System

Laundry

Difficulties in the Laundry Problem• What are current problems that make a laundry

system difficult to automate?

• Highly deformable objects• Infinitely large number of configurations• Lots of possible grasp points

• The laundry problem is still in a research stage

Clothing ClassificationB. Willimon, S. Birchfield, and I. Walker, “Classification of clothing using interactive perception,” in ICRA 2011.B. Willimon, I. Walker, and S. Birchfield, “A new approach to clothing classification using mid-level layers,” in ICRA 2013.B. Willimon, I. Walker, and S. Birchfield, “Classification of clothing using midlevel layers,” in ISRN Robotics, 2013.

Unfolding ClothingB. Willimon, S. Birchfield, and I. Walker, “Model for Unfolding Laundry using Interactive Perception,” in IROS 2011.

Pose EstimationB. Willimon, S. Hickson, I. Walker, and S. Birchfield, “An energy minimization approach to 3D non-rigid deformable surface estimation using RGBD data,” in IROS 2012.B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation,” Submitted to Autonomous Robotics (AURO). B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation without feature correspondence,” in ICRA 2013.

Research PathFocus of thesis

Kita et al. use a humanoid system to recognize the state of clothes using a cloth model with 22 out of 27 attempts correctly classified

Previous Work on Classification for Robotics

Cusumano-Towner et al. were aimed at classifying the category of seven articles with a success rate of 20 out of 30 trials

Our system does not use predefined cloth models or dual manipulators

IsolationGraph-based SegmentationStereo MatchingDetermining Grasp Point

Classification Hanging Position

Binary Silhouettes Visual-based shape and

appearance informationLying Position

Low level features Characteristics Selection Masks

Classification Framework

Classification in a …

Hanging Position Lying Position

Experimental Results With 6 categories, 5 items per category, and 20 images per item,

the dataset collected by the extraction / isolation procedure consists of 600 images

This dataset was labeled in a supervised learning manner so that the corresponding category of each image was known

Two experiments were conducted:

1) Extraction and isolation process

2) Interaction vs. Non-interaction

Extraction and isolation process:

The image taken by one of the downward-facing stereo cameras

The result of graph-based segmentation

The object found along with its grasp point (red dot)

The image taken by the side-facing camera

The binary silhouettes of the front and side views of the isolated object.

Experimental Results

Interaction vs. Non-interaction:

The process of interacting with each article of clothing provided the system with multiple views using various grasp locations, allowing the system to collect 20 total images of each object.

Experimental Results

Two articles were compared by examining the 400 match scores between their pairs of images (20 images per article).

Classification in a …

Hanging Position Lying Position

Classification in a Lying Position

Experimental Results The proposed L-C-S-H approach is applied to a laundry

scenario.

Two different scenarios involved using: 3 categories {shirts, dresses, socks} 7 categories {shirts, dresses, socks, cloths, pants, shorts,

jackets}

Each scenario is run through three experiments:

1) Baseline System (L-H)

2) L-C-H approach

3) L-C-S-H approach

Experimental Results Scenario 1 → 3 categories {shirts, dresses, socks}

Local Global Both0.00

10.0020.0030.0040.0050.0060.0070.0080.0090.00

100.00

BaselineL-C-HL-C-S-H

Experimental Results Scenario 2 → 7 categories {shirts, dresses, socks, cloths, pants,

shorts, jackets}

Local Global Both0.005.00

10.0015.0020.0025.0030.0035.0040.00

BaselineL-C-HL-C-S-H

Clothing ClassificationB. Willimon, S. Birchfield, and I. Walker, “Classification of clothing using interactive perception,” in ICRA 2011.B. Willimon, I. Walker, and S. Birchfield, “A new approach to clothing classification using mid-level layers,” in ICRA 2013.B. Willimon, I. Walker, and S. Birchfield, “Classification of clothing using midlevel layers,” in ISRN Robotics, 2013.

Unfolding ClothingB. Willimon, S. Birchfield, and I. Walker, “Model for Unfolding Laundry using Interactive Perception,” in IROS 2011.

Pose EstimationB. Willimon, S. Hickson, I. Walker, and S. Birchfield, “An energy minimization approach to 3D non-rigid deformable surface estimation using RGBD data,” in IROS 2012.B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation,” Submitted to Autonomous Robotics (AURO). B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation without feature correspondence,” in ICRA 2013.

Research PathFocus of thesis

Cusano-Towner et al. were aimed at flattening a piece of crumpled clothing by implementing a disambiguation phase and a reconfiguration phase.

Previous Work on Unfolding for Robotics

Our system does not use a dual manipulator or a predefined model of the clothing

First PhaseRemove any major wrinkles

and / or folds Pulling the cloth at individual

corners every d degrees

Second PhaseDefine cloth modelCalculate various components

needed for the cloth model

Model to Unfold Laundry into a Flat Canonical Position

Experimental Results The proposed approach was applied to a 3D simulated cloth to

determine the results of the first and second phase.

Two experiments were conducted:

1) Experimental Test of Algorithm

2) Test to Fully Flatten the Cloth

Experimental Test of Algorithm :

This experiment tested the first phase of the proposed algorithm and monitored the process from eight iterations of pulling the cloth.

The models continually change configurations in a manner that flattens and unfolds larger areas of the cloth as the iterations increase.

Eventually, the cloth is mostly flattened out to a more recognizable shape in the final iteration.

Experimental Results

Test to Fully Flatten the Cloth :

This experiment tested the proposed algorithm in determining if this approach would completely flatten a piece of clothing.

The test used the first and second phase of the algorithm to grasp the cloth at various locations and moved the cloth at various orientations until the cloth obtained a flattened percentage greater than 95%.

Experimental Results

Clothing ClassificationB. Willimon, S. Birchfield, and I. Walker, “Classification of clothing using interactive perception,” in ICRA 2011.B. Willimon, I. Walker, and S. Birchfield, “A new approach to clothing classification using mid-level layers,” in ICRA 2013.B. Willimon, I. Walker, and S. Birchfield, “Classification of clothing using midlevel layers,” in ISRN Robotics, 2013.

Unfolding ClothingB. Willimon, S. Birchfield, and I. Walker, “Model for Unfolding Laundry using Interactive Perception,” in IROS 2011.

Pose EstimationB. Willimon, S. Hickson, I. Walker, and S. Birchfield, “An energy minimization approach to 3D non-rigid deformable surface estimation using RGBD data,” in IROS 2012.B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation,” Submitted to Autonomous Robotics (AURO). B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation without feature correspondence,” in ICRA 2013.

Research PathFocus of thesis

Previous Work on Pose Estimation for Robotics

• Elbrechter et al. (IROS 2011) use a soft-body-physics model with visual tracking to manipulate a piece of paper.

• Bersch et al. (IROS 2011) describe a method to bring a T-shirt into a desired configuration by alternately grasping the item with two hands, using a fold detection algorithm.

Both approaches require predefined fiducial markers.

The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms:

Energy Minimization Approach

Correspondence term

Depth term

Boundary term

Smoothness term

The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms:

Energy Minimization Approach

Energy Minimization Approach

Smoothness term

Correspondence term

Depth term

Boundary term

Smoothness term

Energy Minimization Approach

Energy Minimization ApproachSmoothness term

Correspondence term

Energy Minimization Approach

Depth term

Energy Minimization Approach

Front View Top View

Boundary term

Energy Minimization Approach

Energy Minimization ApproachMinimize energy equation

Segmentation and InitializationForeground / background segmentation

Segmentation and InitializationMesh Model Generator

Segmentation and InitializationReinitialization

Experimental Results

• We captured RGBD video sequences of shirts, pants, shorts, and posters to test our proposed method’s ability to handle different non-rigid objects in a variety of scenarios.

Four experiments were conducted:

1) Estimating pose of clothing

2) Estimating pose of posters

3) Reinitializing mesh after in-plane rotation

4) Quantitative Results

Experimental Results

Shirt partially occluded

Shirt moving in-plane and out-of-plane

Shirt changing scale

Shirt translating from side to side

Shirt moving in-plane

• Estimating pose of clothing•7 shirts•1 pair of pants•2 pairs of shorts

Experimental Results

• Estimating pose of posters

Poster with little texture moving out-of-plane

Poster with a lot of texture moving out-of-plane

Experimental Results

• Reinitializing mesh after in-plane rotation

Experimental Results

• Quantitative Results

Conclusion Clothing Classification

Extraction / Isolation A novel approach in which a pile of laundry is sifted by an

autonomous robot system in order to separate each item. Hanging position

Using interaction to provide multiple views of an object and capture more visual data

The results show that, on average, classification rates using robot interaction are 59% higher than those that do not use interaction.

Lying Position Multi-layer approach involving a mixture of global and local

features Characteristics and selection masks achieve, on average, an

improvement of 27.47% for three categories 17.90% for four categories 10.35% for seven categories

Conclusion Unfolding Clothing

An approach to interactive perception in which a piece of laundry is flattened out into a canonical position by pulling at various locations of the cloth.

The algorithm is shown to flatten a simulated cloth by 95.57% of its total area

Pose Estimation A new and novel algorithm that estimates the 3D configuration of a

deformable object through an RGBD video sequence An energy model is used to create a non-linear energy function and the

information is computed using a semi-implicit scheme Energy terms

Smoothness Feature point correspondence Depth Boundary

References

Copies of: Publications Code Datasets Videos

are located at

www.bryanwillimon.com