Automated Tissue Scanning

Brady KingArchie KinneyLuke Reisner

Advisor:Dr. Abhilash

Pandya

Problem Statement

� Intraoperative tissue classification is slow and difficult� Human-performed biopsy (20+ minutes)

� Surgeon makes final decisions

� Tissue classification could allow faster, more accurate resections� Better treatment of cancer or other procedures

� Faster recovery time, lower cost

Our Proposal

� Develop a robotic surgery system capable of classifying tissue in near real-time� Use Raman spectroscopy

� Automate the process ofsample collection

� Interface with currentimage-guided surgerysystems

� Develop novel interfaces for configuring scans and presenting results to the surgeon

The Aesop 3000 (bloop-bloop!)

� “Automated Endoscopic System for Optimal Positioning”

� 7 degrees of freedom (4 active joints)� Control schemes: voice, hand,

foot, touch screen, manual, andserial� Moves end-effector through a

pivot point

� Only moves up/down/left/rightand in/out

Objectives

� Objective 1� Integrate a rigid robotic arm, the Aesop 3000, into

our current image-guided surgery system

� Objective 2� Integrate a Raman probe with the robotic system

� Objective 3� Modify the Aesop 3000 to facilitate automated

tissue scans

� Objective 4� Develop a novel interface for both the selection of

scan parameters and presentation of scan data

Objective 1

� Integrate a rigid robotic arm, the Aesop 3000, into our current image-guided surgery system� Reverse engineer the Aesop to

find control pins and voltages� Breakout box

� Develop a system to relay the tracking information to a PC

� Integrate the tracking information with our current image-guided surgery system

Breakout Box

� Fearlessly sliced a $2,500 Aesopcontrol cable in half

� Connected 2 x 55 wires to a PCB� Made a ribbon cable for routing signals� Mounted on a sturdy metal platform� Big thanks to David Sant!

Breakout Box Picture

Pin Determination

� Tested all 55 pins to determine their functions and working values� Power, motor control,

potentiometer feedback, encoder feedback, etc.

9.48 V DC10

…11

Shoulder/elbow CW control9

Shoulder/elbow CCW control8

Ground7

Elbow potentiometer6

Shoulder potentiometer5

Linear potentiometer4

Linear down control3

Linear up control2

Ground1

UsePin

Potentiometer Feedback

� Developed a system to read potentiometer feedback of 5 most important joints� Some joints have broken or no pots

� Used a 12-bit USB A/D converter to send the feedback voltages to any PC� Can be logged to a file

� Had to buffer linear motor feedback to prevent automatic shutdown

Potentiometer System

DH Parameters

� Measured Aesop’s link lengths, joint limits, and other parameters

� Derived the Aesop’s kinematics model using the standard DH notation

0 + (θ7)180°-d707

90° + (θ6)-90°0a66

180° + (θ5)90°d505

90° + (θ4)90°004

0 + (θ3)90°0a33

0 + (θ2)00a22

000 + (d1)01

00d000

θiαidiaii

Kinematics Model

Note: Diagram is more complicated than it looks.

Tracking in Matlab

� Robotics toolbox for Matlab used to implement the Aesop’s DH model

� Program reads logged pot voltages� Function converts voltages to joint

angles/position� Modified plot function displays the robot’s

motion in 3D using forward kinematics

Tracking Demo!

� Enjoy the demo of Matlab tracking the Aesop’s movements

(Note from Luke: If it doesn’t work, it’s Brady’s fault.)

Objective 1 Challenges

� Aesop 3000 is difficult to work with� No documentation

� Disassembly, automatic shutdown, etc.

� Some joints have missing or broken potentiometers� Will use encoders

� Connectors made by different companies� Breakout box was more difficult to make

� David Sant helped us with this

Objective 1 Changes

� Haven’t integrated with image-guided surgery system (yet)�Decided to use encoders�Waiting for motion controller

� Getting motion controller now will save significant time in Objective 3� Integrating potentiometer feedback would

be a waste of time

Intermission

(I spent way too much time on this slide)

Objective 2

� Integrate a Raman probe with the robotic system� Physically attach the Raman probe

� Integrate Raman classification software with our image-guided surgery system

� Perform a human factors study

� Further develop classification software

� At this point, the system will be ready to perform simple point classification

Objective 2 System Diagram

Request Raman

Point

Send Raman Point

Request A

rm

Location

Send Arm Location

Portable Raman Probe

� Acquired portable Raman probe (finally!)

� Needs to be tested on actual tissue� Eventually will be mounted on the

Aesop

Neural Network Classification

� Part of our Raman data classification algorithm to identify cancer, etc.

� Completed C++ implementation of neural network forward pass� Supports a variable

number of neurons andactivation functions

� Will be integrated with theRaman server application

Raman Server Application

� Acquires end-effector location from a robot server (for the MicroScribe)

� Queues and retrieves data points� Sends data points to 3D Slicer clients� Need to communicate with Raman probe� Need to implement Raman data processing

� Pre-processing (noise filtering, background fluorescence subtraction, normalization)

� Peak extraction, neural network identification

Raman Server Diagram

Objective 2 Challenges

� Raman probe still needs more testing� Compare with previous

Raman data

� Certain algorithmsdifficult to implementoutside of Matlab� Bundled Raman probe software may help

Objective 3

� Modify the Aesop 3000 to facilitate automated scanning motions� Physically modify the Aesop 3000 to better

facilitate movement in vivo

�New sensors, motors, etc.�May switch to Zeus if insufficient

� Develop software to control scanning motion

�Model robot dynamics�Order/build motor controller

Motor Research

� Determined what motors are used in the Aesop 3000� Various brush DC motors from MicroMo

� Researched motor specifications� Voltage and power requirements, etc.

� Determined what encoders are used and their specifications� Rotary encoders of various

resolutions from US Digital

Motion Controller

� Ordered motion controller (we think) from Galil that can handle our motors/encoders� Standalone Ethernet device

� 24 V, 12 A power supply� 4-axis, 200 W amplifier to

drive the motors� 16-bit A/D daughterboard for reading pot

feedback of passive joints

Objective 3 Challenges

� Aesop has only 4 active joints (3 useful)� Could enhance Aesop or switch to Zeus

�Current work will transfer over

� Complete control of the robot arm is difficult� People at Intuitive Surgical are jerks

� Have to set up a standalone motion controller

� Automatic shutdown may be an issue

� Positioning accuracy hard to predict

Revised Budget

Free ($120)System for Aesop tracking3.

$20Breakout box for reverse engineering Aesop2.

Free ($50k)Raman probe4.

$2,905Total:

NegotiableTwo graduate students’ tuition, benefits, and stipend (3 years)

8.

$200Robot modifications7.

$2,635Motion controller (with supply, amplifier, A/D)6.

$50Raman probe mounting hardware5.

Free ($60k-1M)Aesop 3000 (or Zeus)1.

CostItem#

Current Progress

7%

0%

10%

20%

90%

0% 20% 40% 60% 80% 100%

Objective 1

Objective 2

Objective 3

Objective 4

Taking Over theWorld

Future Timeline

January ’09Develop novel interfaces for automated scans, human factors

Obj. 4

November ’07Modify the Aesop 3000 to facilitate automated scans

Obj. 3

March ’07Integrate the Raman probe with the robotic system, human factors

Obj. 2

May ’06Track the end-effector of the Aesop 3000

Obj. 1

DateDescriptionTask

Feedback

Any questions, comments, or suggestions?

(Other than the obvious question)