Georgia Institute of Technology | Marquette University ... · FPIRC 2016 Implement On Live Hardware...

Georgia Institute of Technology | Marquette University | Milwaukee School of Engineering | North Carolina A&T State University | Purdue University | University of California, Merced | University of Illinois, Urbana-Champaign | University of Minnesota | Vanderbilt University

Generic Presentation Template for CCEFPUpdate this text in the Slide Master

Coordinated Rate Control User Interface and Task Identification of an Excavator

BeauDomingue,GRAGeorgiaIns3tuteofTechnology

Prof.WayneBook

FPIRC2016

Research Goal Makefluidpower:• Moreefficient• Safer• EasiertoUse

ConsolidateresultsonmulDdegreeoffreedominterfacesovertherangeofspeeds,dimensions,numbersofinterfaces,extentofautomaDonandinterfacemodaliDesfoundwithhydraulicactuaDon.

FPIRC2016

GT Excavator Simulator

FPIRC2016

GT Excavator Simulator

FPIRC2016

Implement On Live Hardware •  ExcavatorSimulatorisagreattool,butnotrealthing

•  Verifyresultsonlivemachine•  Doneinpast,wouldliketodoagaininthefuture

FPIRC2016

3D – Stereo on GT Simulator

FPIRC2016

3D – Stereo on GT Simulator

FPIRC2016

Excavator User Interfaces" •  Thecurrentstateoftheart•  TheevoluDonofthetechnologydevelopedforthisproject.

FPIRC2016

Conventional Excavator UI •  JointflowcontrolwithjoysDcks

•  NotintuiDve•  Slowlearningcurve•  RequiresextensivetrainingImprovedUIWould:•  ReducecogniDveload•  Reduceoperatorerrors•  ReducetaskDme&fuelconsumed

FPIRC2016

State of Art Excavator UI

FPIRC2016

Jacobian Breakdown of Control Vectors

FPIRC2016

Jacobian Breakdown of Control Vectors

FPIRC2016

Coordinated Control •  MapsoutputofmulDdegreeoffreedomsystemtocoordinatesystemfamiliartooperator

•  ChangeinputsofexcavatortocontrolendeffectorposiDondirectly,notindirectlythroughjointangles.

•  ControllersolvinginversekinemaDcs,notuser

FPIRC2016

Coordinated Control •  PosiDonorRateControl?

FPIRC2016

Coordinated Rate Control with Phantom

FPIRC2016

Coordinated Position Control with Phantom

FPIRC2016

Position vs. Rate Control on Excavator •  Excavatorsareslower,andfavor

ratecontrol•  ReducedeffecDvenessof

posiDonalcontrolduetofactthattheoperatordoesn’tknowwhatcommandisbeinggiven,leadstoresponseovershootorundershoot

•  EltonconfirmedthisbytesDngposiDoncontrolwithghost,whichperformedsignificantlybeXer

•  ApplicableonremoteoperaDon

FPIRC2016

Conventional Control vs. Coordinated Rate Control with Phantom

FPIRC2016

Phantom Roadblocks

Fragile ExpensiveNotErgonomic(fa3gueissues)

BiodynamicFeedthrough

FPIRC2016

Coordinated Position Control - Kinematically Similar Arm

Fa3gue+biodynamicfeedthroughissues!

FPIRC2016

Coordinated Position - Control Kinematically Similar Arm

RotatecontrolplanesidewaystoaddressfaDgueissues

FPIRC2016

Coordinated Position Control Kinematically Similar Arm

RotatedcontrolplaneinconsequenDaltoperformancemetrics

FPIRC2016

User Interface Adoption Players • Manufacturers

–  Price–  Complexity–  Durability

• Contractors/Owners–  Liability–  FuelEfficiency

• Operators–  Ergonomics–  TimeEfficiency–  LearningCurve

Goal:Implementnewcontroller,consideringallconcerns,usingwhatwe’velearnedfrompreviouswork

FPIRC2016

Proposed Configuration •  NewUIwouldnotrequirenewhandcontrollers•  NewUIshouldreapbenefitsassociatedwiththechangefromjointcontroltocoordinatedcontrolandoffersignificantperformanceimprovements

•  ExcavatorwouldbecapableoftogglingbetweentradiDonalcontrolstyleandnewstyleviaswitch

•  Requirements:excavatormustbeawareofitsjointangles,andhavecomputerordedicatedchipperformthemathemaDcsassociatedwithcontroller

JoysDck HydraulicsFlowCommands JoysDck

Hydraulics

CoordinatedCommands

ExcavatorJointAngles

Controller

FlowCommands

TradiDonalorCoordinatedSwitch

FPIRC2016

New Coordinated Rate Control Mapping?

FPIRC2016

New Coordinated Rate Control Mapping

FPIRC2016

Mapping – Left Joystick

FPIRC2016

Mapping - Right Joystick

FPIRC2016

Conventional Vs. CRC

TradiDonalControl CoordinatedRateControl(CRC)

FPIRC2016

Human Subject Experiment

Session 1 Session 2 Session 3

warm-up | trial1 | trial2 | trial3 warm-up | trial1 | trial2 | trial3 warm-up | trial1 | trial2 | trial3

Group A Conventional Conventional CRC

Group B CRC CRC CRC

FPIRC2016

Time Efficiency

Trialmean 1 2 3 4 5 6

Soil

Rem

oved

(kg)

/ Ti

me

(min

)

0

200

400

600

800

1000

Conv. UICRC UI

FPIRC2016

Group Time Efficiency

Trialmean 1 2 3 4 5 6 7 8 9

Soil

Rem

oved

(kg)

/ Ti

me

(min

)

0

200

400

600

800

1000

Group AGroup B

FPIRC2016

Fuel Efficiency

Trialmean 1 2 3 4 5 6

Soil

Rem

oved

(kg)

/ Fu

el C

onsu

med

(kg) #104

0

0.5

1

1.5

2

2.5

3

Conv. UICRC UI

FPIRC2016

Idle Time

Trialmean 1 2 3 4 5 6

Perc

enta

ge o

f Tim

e Sp

ent I

dle

0

10

20

30

40

50Conv. UICRC UI

FPIRC2016

Survey Results

Conventional CRC No Preference0

2

4

6

8

Num

ber o

f Sub

ject

s

Group A subjects Learning Preference

Expert

Conventional CRC No Preference0

1

2

3

4

5

6

7

Num

ber o

f Sub

ject

s

Group A Subjects Bucket Control Preference

Expert

Conventional CRC No Preference0

1

2

3

4

5

Num

ber o

f Sub

ject

s

Group A Subjects Controller Confidence

Expert

FPIRC2016

Task Identification

FPIRC2016

Standard Trenching Cycle

ReturnDig Unload

FPIRC2016

Task Identification •  Trenchingprocesscanbedelineatedintothreesubtasksorphases:– Digphase– Unloadphase– Returnphase

•  Duetothewelldefinedcyclenatureoftheprocess,asupervisedclassificaDonapproachcanbeused.

FPIRC2016

Artificial Neural Network (2ANN)

Input Layer (2)

Hidden Layer (50)

Output Layer (3)

2x50 Weights

50x3 Weights

Dig

Unload

Return

FPIRC2016

2ANN Classification of seen Training Data

1 2 3 4 5−1

−0.5

0

0.5

1

Time (min)

Nor

mal

ized

Inpu

t

SwingStickDigUnloadReturn

ManualClassification

ANNClassification

FPIRC2016

2ANN Classification of seen Training Data

2−1

−0.5

0

0.5

1

Time (min)

Nor

mal

ized

Inpu

t

SwingStickDigUnloadReturn

ManualClassification

ANNClassification

FPIRC2016

2ANN performance •  TheresulDngclassificaDonwas~96%accurateforthetrainingdatawithclassificaDoncorrecDons.

•  However,theANNwasnotrobusttooperatorstyleduetoitsextremelysimplestructureandresulDngdecisionboundary.

FPIRC2016

2ANN Decision Boundary

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

Swing

Stic

k

DigUnloadReturn

FPIRC2016

2ANN Unseen Data Classification

2 3−1

−0.5

0

0.5

1

Time (min)

Nor

mal

ized

Inpu

t

SwingStickDigUnloadReturn

{ 2 1 3 1 3 1 }

FPIRC2016

2ANN Unseen Data Classification

2 3−1

−0.5

0

0.5

1

Time (min)

Nor

mal

ized

Inpu

t

SwingStickDigUnloadReturn

Premature Transition

{ 2 3 1 }

FPIRC2016

Training Data Generation •  The2ANNgenerallyyieldedpoorclassificaDonsforunseendata.

•  However,withsomepreandpostprocessingofclassificaDons(*thatcouldnotbedoneinrealDme)the2ANNyieldedacceptableresultsforsomedatasets.

•  Thus,the2ANNwasusedtogeneratetrainingdatatotrainanotherANN.

FPIRC2016

Training Data Generation •  5datasets,correspondingto5trialsfromdifferentsubjectsforeachgroup,wereselected.

•  Datawassampledat100Hzeach5minutetrial,correspondingto30,000datapoints,or150,000datapointstotal.

FPIRC2016

Artificial Neural Network (8ANN)

Input Layer (8)

Hidden Layer (50)

Output Layer (3)

8x50 Weights

50x3 Weights

VelocityCommands

JointPositions

Dig

Unload

Return

FPIRC2016

8ANN Classification of seen Training Data

1 2 3 4 5−1

−0.5

0

0.5

1

Time (min)

Nor

mal

ized

Inpu

t

Swing Vel cmdBoom Vel cmdStick Vel cmdBucket Vel cmdSwing PosDigUnloadReturn

ManualClassification

ANNClassification

FPIRC2016

8ANN Classification of seen Training Data

2−1

−0.5

0

0.5

1

Time (min)

Nor

mal

ized

Inpu

t

Swing Vel cmdBoom Vel cmdStick Vel cmdBucket Vel cmdSwing PosDigUnloadReturn

ManualClassification

ANNClassification

FPIRC2016

8ANN Classification of unseen Data

2 3−1

−0.5

0

0.5

1

Time (min)

Nor

mal

ized

Inpu

t

Swing Vel cmdBoom Vel cmdStick Vel cmdBucket Vel cmdSwing PosDigUnloadReturn

Swing Right Swing LeftRobust to Oscillation/Glitches

FPIRC2016

Visualization of subspace of 8 Dimensional Decision Boundary

1-1-1

Boom

0

0

Stic

k

Swing

0

1

-11

DigUnloadReturnBucket ClosingBucket nullBucket Opening

FPIRC2016

Visualization of subspace of 8 Dimensional Decision Boundary

FPIRC2016

Thanks to Contributors and Partners

FPIRC2016

Contact •  BeauDomingue–bdomingue3@gatech.edu

•  WayneBook–wayne.book@me.gatech.edu