Embed Size (px)
Citation preview
Robotic Grasper based on an End-to-End NeuralArchitecture using Raspberry PiPranav B Sreedhar
Dept of Computer Science and Engineering,Amrita School of Engineering, Bengaluru
Amrita Vishwa VidyapeethamIndia
Email: [email protected]
Suja Palaniswamy*Dept of Computer Science and Engineering,
Amrita School of Engineering, BengaluruAmrita Vishwa Vidyapeetham
IndiaEmail: [email protected]
Abstract—Reinforcement learning coupled with Neural Net-works has been demonstrated to perform well in robotic ma-nipulation tasks. Yet they require large volume of sample datawhich are trained using huge amount of computing resources.We propose an End-to-End Neural Network architecture basedrobotic system that can be deployed on embedded platforms suchas a Raspberry Pi to perform robotic grasping. The proposedmethod potentially solves the exploration-exploitation dilemmaeven under undecidable scenarios.
Index Terms—Reinforcement Learning, Neural Networks,Robotics, Artifical Intelligence
I. INTRODUCTION
Robotic grasping can be described as using a robotic systemto perform physical manipulation of an object or set of objects.There exists a difficulty in robotic grasping due to the varietyof input scenarios and the lack of general solutions that canbe mapped easily from the input. Learning algorithms seem tohold promise in solving such hard problems. Learning throughdemonstrations can be considered a way to ensure safety ofthe robotic system but the training times can prove to belarge and considerable bias of the demonstrator is alwaysinduced into the system. Although many approaches to theproblem have made significant contributions they are yet to berealized effectively on an embedded hardware platform suchas Raspberry Pi.
Lebedev et al. [1] showed that a macaque monkey can learnto adapt, percieve and control a robotic arm as its own limbto achieve a visual target shown on a screen. As monkeys aremotivated to perform a task for a reward, a sip of fruit juiceis given as reward for every successful attempt. This providesus the inspiration to develop a neural network based systemthat can perform such tasks using a reward based system.
Reinforcement learning has emerged as a successful ap-proach to solving such problems in the robotic domain. Mnihet al. [2] proposed to use neural networks as opposed to state-action tables in reinforcement learning to map screen imagesdirectly to corresponding actions in video games. They wereable to play at superhuman levels using an enhancement ofQ-learning they developed called Deep Q-learning.
Bojarski et al. [3] proposed to perform robotic controlfor self-driving tasks using an End-to-End approach to map
sensory input to robot actions with high degree of success.Due to the high dimensionality in the input space and actionspace, an End-to-End approach has the capability of achievingsuccess in complex tasks.
Levine et al. [4] claimed to be the first to map pixelobservations directly to motor torque control using DeepConvolutional Neural Networks (DCNN) containing 92,000parameters. They used a guided policy search algorithm tobring the policy search problem to the supervised learningdomain. They demonstrated complex learning tasks such asplacing a block into a shape sorting cube, tightening a bottlecap and placing a coat hanger on a clothes rack. Their pro-posed method is a proof of concept that End-to-End learningworks. They used a PR2 robot and a large neural network fortraining thereby making it impractical to use on an embeddedplatform.
Gu et al.[5] demonstrated an asynchronous Deep Reinforce-ment Learning (DRL) using Normalized Advantage Function(NAF) to perform door opening task on real robotic platforms.They propose to use multiple robots to performing the dooropening task in parallel and then updating a centralized threadduring the learning process at the end of each episode. Theirresults show that increasing parallelism has diminishing effectson the learning time. In practicality it may be infeasible touse multiple robots to learn a task due to high cost and smallreturns.
Levine et al. [6] used a CNN to predict the success ofthe grasp trajectory based on the current camera image asinput. This was done without any manual annotation by usinga sensor on the gripper to report success or failure of agrasp. This feedback was used as label to train the CNN inthe supervised learning domain. They also varied the cameraposition relative to robot configuration to achieve a level ofgeneralization. This approach allowed them to collect data ata large scale (800,000 attempts) with minimal human effortand led to a very high success rates.
Zhang et al. [7] followed a different approach to pick andplace by using a two-dimensional(2D) simulation environmentfor a robotic arm that has 3 joints with 9 discrete actions.They follow a reinforcement learning approach with the re-ward function defined as positive if the end effector of the
robotic arm is near the target and negative otherwise. Theyattempted to introduce noise into the environment to achievegeneralization but the results on simulated test data were poor.They had zero success rate when tested in real environmentwith simulated training.
Finn et al. [8] proposed a general approach that implementsmeta learning to improve training sample efficiency and canbe used by any model that uses gradient descent. Their methodcan be applied to reinforcement or supervised learning domainand shown to compete with state of the art accuracy in pick andplace tasks with significantly less training samples. They showmarked improvement in accuracy after one gradient update.Their method focuses on good initialization of parameters toachieve faster convergence of the model.
Balaji et al. [9] approached the problem of universalgripping by implementing granular jamming. It works byhardening into the shape of the target object when air pressureis decreased thereby ensuring a firm grip.
Sanju et al. [10] proposed a method to implement anautonomous robotic arm that works by performing randompick and place and can address user interrupt instantly.
Pereira et al. [11] proposed a method to perform objectsorting at a low cost using a Raspberry Pi, a robotic arm andimage processing using the Pi Camera module. The roboticarm is coupled with an Arduino which then communicates tothe Raspberry Pi.
Siagian et al. [12] proposed a method for web basedmonitoring and control of a custom built robotic arm usingthe Raspberry Pi as the intermediary. This method shows thata robotic arm can be sufficiently controlled with a RaspberryPi even over the Internet.
Some of the factors in applying reinforcement learningbased on neural networks to robotic perception-based con-trol include the image resolution, sample size and sparsityof rewards. All of the aforementioned factors are severelyconstrained on embedded platforms such as Raspberry Pi.
This paper aims to answer the following question: - cana perception based robotic control adapt under undecidablescenarios and converge to a desired solution? Many of therelated works attempt to bring the reinforcement learningdomain into the supervised learning domain to train the neuralnetwork since there are established ways for training in thesupervised learning domain. In such scenarios rewards areconsequently considered as labels.
The contributions of this paper are: - 1) solve theexploration-exploitation dilemma from the perspective of thesystem. 2) Perform End-to-End learning for robotic arm onthe Raspberry Pi platform.
In summary, previous work have focused on accuracy andcomplexity as the prime factor and hence cannot be usedalong with a low compute platform such as a Raspberry Pi.Most of the work need the model to stop learning at somepoint and then deployed and as such cannot be adaptableto new scenarios. Such models have the downside that theycannot decide when a new input is given, it can be called
as an undecidable scenario from the model’s perspective. Weproposed a method in this paper to overcome these challenges.
The remainder of this paper is organized as follows. SectionII discusses theoretical background and Section III narratesproblem insight. Proposed method and End-to-End architec-ture are explained in section IV and V respectively. Section VIand Section VII describes results and conclusion respectively.
II. THEORETICAL BACKGROUND
Reinforcement learning is a machine learning method thatis inspired by animal behavior wherein an agent (animal)acts within an environment (ecosystem) with the motive tomaximize the cumulative reward received from the environ-ment. Reinforcement learning assumes the system to be aMarkov Decision Process (MDP).Each unique scenario of theenvironment and the agent is encapsulated in mathematicalform as a state. The state s is referred from the agent’sperspective. In a MDP, at any given time step the agent isin some s which is a subset of all possible states referred as Sof the environment. Every agent has a set of actions named Ait can perform within its environment. The agent can transitioninto the next state s’ from s by performing an action a from Aat any given time step. During its lifetime, the agent transitionsfrom one state to another with respect to a discrete time stepwhich is recorded as a sequence of states. Given s and a thenext state s’ can be found from the state transition functionF(s, a) and it is conditionally independent of all previous statesand actions i.e. states prior to s and their corresponding actionsa need not be kept in memory to compute F(s, a).
Reinforcement learning can be done as model-based ormodel-free i.e. either the model of the environment is learntand every action can be planned based on the model (model-based) or the model of the environment is not known and onlythe value or utility of each state can be learnt (model-free).Q-learning is a well-known technique to perform model-freereinforcement learning. The Q-Learning algorithm estimatethe Q-values or utility for each action in each state. Inother words, Q-Learning gives information (Q-value) aboutthe cumulative reward for taking each action. We can use theBellman Equation as in (1) to perform Q-learning where theaction which gives maximum cumulative reward is chosen atevery time step.This is given in Equation
Q (s, a) = R+ γ(maxa′
Q (s′, a′))
(1)
Where Q(s, a) is the Q-Value of the state-action pair of states and action chosen a. R represents the immediate rewardreceived from the environment for performing the action a.g represents the discount factor varying between 0 < g < 1which quantifies the importance of the future rewards receivedrelative to the present reward. Discount factor g > 0.9 givesimportance to recent rewards distantly into the future.
The policy that is followed is defined as given in Equation(2).
π ∗ (s) = argmaxa′
Q ∗ (s′, a′)∀s ∈ S (2)
Where p*(s) is the final policy followed by the algorithm tofind the optimal solution. Q* is the final Q-function and S isthe set of all possible states. The policy signifies that the actionwhich accumulates maximum rewards is always chosen.
Equation (3) is used to update the Q function.
Q′ (s, a) = Q (s, a) +α(R+ γ
(maxa′
Q (s′, a′))−Q (s, a)
)(3)
where a is the learning rate in the range of 0 < a < 1. TheBellman equation is used to assess the current utility of thataction a in the state s. The difference of the new value andcurrent value is counted as the update with a determining theimportance of the update. Q’ is the updated Q-function.
In traditional Q-Learning we can use state-action tables totabulate the value of each action while in a state and followa greedy approach of choosing the most rewarding action atevery time step. As the number of states increases it becomesinfeasible to maintain a table for every state-action pair andtherefore an approximation becomes necessary. Neural Net-works are considered to be universal function approximatorsand it has been proven to perform very well when used in thereinforcement learning domain.
When using Neural Networks for Q-Learning, a techniquecalled Experience Replay as introduced by Mnih et al. [2]has been demonstrated to improve the learning efficiency andaccuracy of the learning algorithm. Experience Replay collectsthe information of the state, the action taken and the rewardreceived in that time step and stores it in a buffer. Usingthe Bellman Equation as the target value for the action takenhelps to train the Neural Network as in the supervised learningdomain. This can then be called the Q-network. Using this Q-network we can then map raw pixel inputs to agent actions,where the agent can be a robotic arm.
III. PROBLEM INSIGHT
Traditionally, controlling a robotic arm requires special soft-ware that is catered to the needs of the robot i.e. the softwareneed to know the exact details of the robot under differentconditions before it can be used. A Personal Computer istypically used to run the software and an input file containingthe set of coordinates that the robot must traverse toward,needs to be given beforehand. For pick and place tasks theexact location of the object is generally given or a sensoris used to detect the object and then the task is completed.The problem with this approach is that it works only in acompletely known environment and preferably static in nature.
A more considerate approach to a pick and place task wouldbe to first use a camera to detect objects and classify themusing a popular deep learning based model. Then localizethe position of the object with respect to the camera andthe end effector of the robotic arm. The location can thenbe converted into coordinates of the robotic system and givento the controlling software to perform the pick and also place.A similar approach can also be followed for placing the sameobject in a different location.
There are downsides to the aforementioned approach. Theobjects are assumed to be known to the system. The dynamicsof the object and the environment is known to the designerof the system, based on which the specific algorithm isdevised. There is a large gap in the communication betweenthe perceptual system (object detection and classification) andthe robotic arm control system (coordinate system and controlsoftware). Given that the approach was for a pick and placetask it may require significant efforts to tweak the systemfor other robotic grasping tasks such as reaching, pushing,manipulation etc.
An End-to-End Learning approach towards this problemwas proposed by many of the recent works with the view thatit can alleviate the limitations of the other approaches andalso add to their advantages. The gap between the perceptionsystem and the control system is reduced as they are tightlyintegrated in an End-to-End approach. The provision to add tothe set of tasks that the robot is capable of performing throughlearning reduces the necessary human effort. The conceptof trial and error through reinforcement learning allows forexploring the dynamic capabilities of the robotic system whichmay not be easily conceivable or requires a lot of humaneffort. The fact that the software required remains almost sameregardless of the type and configuration of robotic arm albeitwith an additional learning phase makes this approach portableand compatible with generic robotic systems.
Most of the work on End-to-End approach to robotic grasp-ing consider it from a software based perspective and thereforehave relaxed constraints on hardware computing resources.The primary objectives prevalent in recent works are graspingaccuracy i.e successful grasps, mastering of tasks, minimizingtime to acceptable performance.
From a practical standpoint there is a need for an integratedsystem that has balanced provision for the primary objectivesand also considers limited hardware resources. The computinghardware for a robotic system needs to be limited in terms ofphysical size, weight and cost to meet the goals of the roboticsystem. Technologies such as Artificial Neural Networks areyet to be fully utilized in the embedded space although therehas been prolific work being done in recent times.
One of the most challenging aspects of building and de-ploying a robotic system is incorporating decision mechanismsfor unforeseen scenarios that may be encountered by therobotic system. While using a learning approach every newstate of the system can be viewed as an unforeseen scenarioand therefore the decision taken in that state carries a valuethat may not be apparent until a state in the distant future.Upon close inspection it can be noted that there arises somecircumstances such as absence of an object in the visible scenedespite its presence in the environment which are inevitablyundecidable. Extending from this line of thought, a searchalgorithm is always confined to the known information of theenvironment and therefore never capable of adapting to theunknown environment.
IV. THE PROPOSED SYSTEM
This section describes the hardware and software basedcontributions of proposed method.
Fig. 1. The proposed articulated robotic arm
A. Hardware
An articulated robotic arm was custom built for this work. 3Degrees of Freedom (3 DOF) for the robotic arm is proposedto meet the requirement of basic pick and place tasks. Therobot requires movement in multiple directions. Namely inthe X,Y and Z axes of a Cartesian coordinate system. A polarconfiguration robot that can follow a polar coordinate systempossesses sufficient freedom in all the axes as seen in Fig. 1 .
The end effector chosen is a 4 fingered gripper which isrotary actuated. The fingers are integrated with a pressureswitch tuned to a particular pressure that suits most objects.A set of counter weights are used to balance the torque at theother end of each arm. Although it adds to the weight of therobot the total torque on the motors at each joint is negligiblewhich facilitates the use of relatively less torque motors andalso allows for a good payload of about 150gms. Varying thecounter weight allows for accommodating different payloadranges.
The hip joint is a swiveling base which uses a pulley systemto connect with a 1.8 degree/step stepper motor. Since it hasroughly 1:7 ratio very little torque is required to swivel andalso achieves over 7 times the accuracy of the stepper motor.Jerk free operation of each joint is desired to keep a smoothoperation of the robot. Due to many repeated operations themotors are required to be of high durability. As a result it wasfound that off-the-shelf servo motors were not suited to thetask.
The shoulder and elbow joints are directly driven usingturbo worm DC geared motors with a specially built servofeedback mechanism using high precision potentiometers.APulse Width Modulation (PWM) based control signal along-side the feedback input from the potentiometer is used as input
to a window comparator. It accepts duty cycles as low as10% and as high as 90% which is mapped to the maximumallowable limits of each joint in the range of 90 degrees. Theoutput of the window comparator is given to a H-Bridge circuitthat powers the DC motors.
Such a design was preferred as it offered two distinctadvantages. First, it allowed for a greater range of PWM to beused as opposed to off-the-shelf traditional servoing PWM andtherefore the PWM generating hardware has a reduced burdenof accuracy. This decomposition of the servo system allows forindividual replacement of either the motor, potentiometer or H-Bridge to meet different requirements. For example, a largermotor can used to handle higher load without replacing thepotentiometer or even the H-Bridge if its capable of deliveringthe required power.
The Raspberry Pi 3 was chosen as the computing platformfor the proposed system majorly due its popularity in the em-bedded and Do-It-Yourself community and consequently theirwidespread support on troubleshooting forums. Also, its cost isrelatively lesser than other comparable single board computers.It has a well matured and maintained software ecosystem thatextends support to many desktop based software packages. Ithas a Quad-Core ARM Cortex A-53 CPU with 1GB RAM,built-in Wi-Fi Radio,USB and 40 GPIO pins. An RGB USBwebcam is used as the camera sensor.
B. Software
Raspbian Stretch is the OS used on the Raspberry Pi 3. Thecode is written in Python due to its ease of use, flexibilityand availability of deep learning frameworks and extensivelibrary support for GPIO programming. RPi python library isused for GPIO programming. Open CV is the software librarypackage chosen for Image Processing and Computer Visiontasks. Keras is used as the high level API for Neural Networkprogramming. TensorFlow is the default choice as the backenddeep learning framework although Theano can also be usedinterchangeably.
V. PROPOSED END-TO-END NEURAL NETWORKARCHITECTURE
This section describes the architecture based contributionsof this work.
A. Neural Network
The input image is from RGB camera frame converted to 10x 10 pixel resolution and 8-bit grayscale. The neural networktakes the input image as the input layer and contains twofully connected hidden layers each containing 40 neurons withReLU activation. The output layer contains 8 neurons withoutactivations for the three motors with one output per direction(clockwise (CW) or counterclockwise (CCW)). Another outputto switch on the gripper and the final output for the “I don’tknow” signal as depicted in Fig. 2. The outputs of the motorsare given to the corresponding motor of the robotic arm witha fixed step size of 1 degree in that particular direction.
Raspberry Pi 3
Gripper Fully Connected
Hidden Layer 2
10 x 10 8-bit Grayscale Image
From Webcam
Feedback
Articulated Robotic Arm
Input Layer
Fully connected Artificial Neural Network
Motor 1 (CCW)
Motor 2 (CW)
Motor 2 (CCW)
“I Don’t Know”
Fully Connected
Hidden Layer 1
Output Layer
Motor 1 (CW)
Motor 3 (CW)
Motor 3 (CCW)
Fig. 2. End-to-End Neural Network Architecture for robotic grasping
The weights are randomly initialized. Stochastic GradientDescent is used as the optimizer with a learning rate of 0.3and batch size of 30. Every input frame is stored along withthe corresponding action taken during that frame and thereward received in that frame in an experience replay bufferof maximum size 1000, similar to method suggested by [2].Training is done after every action by sampling a random batchfrom the experience replay buffer. A reward of -1 is given atthe end of the action if the object has not been picked else+1 is given. It was found through experimentation that givinga reward of 0 instead of -1 generally results in more numberof steps taken to reach the goal. Presumably because givingthe reward as 0 doesn’t result in the accumulation of rewardsat every step whereas giving a reward of -1 results in fasternegative accumulation of rewards at every step.
B. “I Don’t Know” signal
1) Exploration-Exploitation dilemma: While performingreinforcement learning in an environment, how an action ischosen forms the crux of the system. In Q-Learning, the valueof an action is embodied in the form of cumulative rewards. Inother words, the model describes the cumulative reward thatcan be received for taking a particular action. Consequently, agreedy policy can always be followed that chooses the actionthat maximizes the positive reward.
From a practical perspective, the value of an action fora given state can only be found by taking that action andcalculating the total reward received after the end of theattempt. The end of an attempt is when a terminating conditionhas been met, in our case this can be a successful grasp or say
a maximum number of steps. Given the large number of statesand a sizable set of actions, exhaustively finding the value ofeach action in every state becomes a hard problem. This searchfor value of an action in a state can be called as Exploration.Once the values of all actions in every state becomes apparentthen it becomes worthy to exploit this information in everystate by choosing the action that gives maximum rewards. Thiscan be called as Exploitation.
The Exploration-Exploitation dilemma emerges from thefact that it is difficult to determine if a state needs to beexplored further or exploit the existing information. Even littleinformation about the value of a state can be exploited but itmay lead to less favorable results. Exploration can producemore favorable results in the long term but suffer in theshort term. The downside to exploitation is that the resultstend to be deterministic and hence never improves withoutexploration. The value of exploration can only be brought outby exploitation. In terms of performance, exploration has highvariance and low mean whereas exploitation has high meanand low variance.
2) Proposed solution: For any given state a random actionis chosen during exploration and a specific action duringexploitation. It can be assumed that the choice of choosinga random action (exploration) also has a value. This valueencapsulates a multitude of information regarding the learningprogress of a model, value of a state and even if it isundecidable to chose an appropriate action. The questions thatremain are:- how to determine the value and how to exploitit?
If there is no prior information about a state then a random
action has more apparent value than any particular action. Ifthere is a solution from the given state then over multipleattempts (approximately the number of actions) choosing arandom action will give that solution. Conversely, if necessaryinformation regarding a state is available then there exists atleast one action that has more value than choice of choosinga random action. It is possible to harness this necessaryinformation from exploration i.e. the choice of choosing arandom action.
Consider that the Q-learning model can chose to leave thechoice of action to randomness apart from the other possibleset of actions. If the model finds that choosing a random actionhas more value than all other actions then it will execute thatrandom action. At a later stage during training, the chosenaction can be revealed as if it had been chosen by the modelwhile in that state. It can also be viewed as performing thatparticular action gives a particular reward. An additional factoris also introduced to learn from the possibility that choosinga random action is in fact the best action. This can be calledthe “revelation” factor which determines whether it should berevealed as the “to be chosen” action or the choice of randomaction.
There may exist a state which has neither a best action nora worst action i.e. all actions can potentially lead to the goal.There also exists a possibility that there is a state for which anyaction chosen is relevant only for that time period and becomesirrelevant for any other time period, where time is consideredas the only independent variable. The action to be taken inthese states cannot be determined or decided. These states canbe called as undecidable states. One such example is a statewhich has no information. Learning from past experiences forthese states always leads to a diminishing value. Thus the valuefor the choice of random action is always going to be higherthan any individual action. Therefore using this choice is thebest way to reliably transition from an undecidable state to amore decidable state.
This choice to chose a random action is called as the “Idon’t know” signal because it signifies that the model hasinsufficient information to take a reliable decision i.e. it doesnot know which action to take and wants to learn more aboutthe state and the value of different actions i.e. it wants toexplore. It is using this signal the model transitions from notknowing anything regarding a state to the knowing everythingabout a state.
As a summary, we propose to use Raspberry Pi as theplatform to both control the robotic arm as well as to performreinforcement learning using the proposed method,whereasearlier work used a seperate controller such as a microcon-troller for the robotic arm and larger computing platformsfor performing reinforcement learning. The “I don’t know”signal introduced in this work essentially determines whetherto explore or to exploit at any given state with regards tothe learning progress, previous experience and the qualityof information present in a state. In essence, giving thecontrol over to the model effectively solves the exploration-exploitation dilemma.
VI. RESULTS
(a) epsilon starts at 0.7 and decays by0.05
(b) epsilon starts at 0.5 with slow decayof 0.01
(c) epsilon starts at 1 and decays by 0.05 (d) epsilon starts at 1 and decays by 0.05
(e) epsilon starts at 1 and decays by 0.05
Fig. 3. Plots for the no. of steps taken to reach the termination state (goal orfailure) vs the no. of epochs. Epsilon value is decayed every 10 epochs. Reddenotes the performance of the epsilon greedy method. Blue represents theproposed method. Green signifies the no. of steps where the “I don’t know”signal is chosen as the best action and a random action is taken in the proposedmethod. Yellow denotes the no. of steps that were random actions determinedby the value of epsilon in that epoch. 3a and 3b shows the performance withdifferent epsilon and have different decay for a grid size of 10. 3c and 3drepresents performance for a grid size of 20 when the proposed method is runtwo separate times for the same set of initial positions from the epsilon greedymethod. The experiments are run for 500 epochs.3e shows the performance ofboth methods under undecidable scenarios. The undecidable scenario occurswhen the object is allowed to go outside the boundaries of the grid by 5 gridblocks for grid size of 10 and is run for 100 epochs.
To test the proposed solution for the exploration-exploitationdilemma, a simulation was created where the agent hadto bring an object to the center and zoom in on it. Theenvironment grid size (pixel resolution) can be configured andgenerally the object is not allowed to go outside the grid area.A grid size of 10x10 pixels and 20x20 pixels were tested. As acontrol setting, epsilon-greedy method was used. To keep thedifficulty on par, every epoch was initialized with the objectat a random position within a random edge of the grid. Alsothe initial position of the object in the control was used as theinitial position in the experimental setup for correspondingepochs.
Plots in Fig. 3 compare the performance of the control i.e.epsilon-greedy method from Mnih et al. [2] and the proposedmethod. It can be noted that the proposed method outperformsthe epsilon-greedy method regardless of the epsilon valuesand the difference is much more prominent for a grid size of20x20 pixels. As can be noted from Fig. 3e under undecidablescenarios the epsilon-greedy method is able to successfullycomplete the task only 58% of the time whereas the proposedmethod is successful 91 % of the time and taking far lessernumber of steps by utilizing the “I don’t know” signal effec-tively. The proposed architecture was deployed on a RaspberryPi 3 coupled with a custom built articulated robotic arm andexecuted successfully.
VII. CONCLUSION AND FUTURE WORK
We proposed an End-to-End Neural Architecture for roboticgrasping that can be deployed on a Raspberry Pi and demon-strated that it outperformed the epsilon-greedy method undernormal settings as well as in undecidable scenarios.
There are limitations in our work that we wish to addressin the future. A higher image resolution is desirable for morecomplex tasks but without compromising other requirementssuch as training time, similar computing resources etc. Infuture, we would like to use continuous control instead ofdiscrete control with fixed step size as it may allow higherimage resolution with lesser number of steps and lessertraining time that makes the system much more scalable.
ACKNOWLEDGMENT
I would like to thank my parents for their unconditionalsupport throughout the course of this work. I especially thankmy father, Mr. B. Sreedhar for his timeless effort in contribut-ing to all facets of this work, without whom constructing therobotic arm would be unfathomable. - Pranav B Sreedhar
REFERENCES
[1] M. A. Lebedev, J. M. Carmena, J. E. O’Doherty, M. Zacksenhouse,C. S. Henriquez, J. C. Principe, and M. A. L. Nicolelis, “Corticalensemble adaptation to represent velocity of an artificial actuatorcontrolled by a brain-machine interface,” Journal of Neuroscience,vol. 25, no. 19, pp. 4681–4693, 2005. [Online]. Available: http://www.jneurosci.org/content/25/19/4681
[2] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G.Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovskiet al., “Human-level control through deep reinforcement learning,”Nature, vol. 518, no. 7540, p. 529, 2015.
[3] M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner, B. Flepp,P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang,J. Zhao, and K. Zieba, “End to end learning for self-drivingcars.” CoRR, vol. abs/1604.07316, 2016. [Online]. Available: http://dblp.uni-trier.de/db/journals/corr/corr1604.html#BojarskiTDFFGJM16
[4] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training ofdeep visuomotor policies,” The Journal of Machine Learning Research,vol. 17, no. 1, pp. 1334–1373, 2016.
[5] S. Gu, E. Holly, T. Lillicrap, and S. Levine, “Deep reinforcement learn-ing for robotic manipulation with asynchronous off-policy updates,” inRobotics and Automation (ICRA), 2017 IEEE International Conferenceon. IEEE, 2017, pp. 3389–3396.
[6] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, “Learninghand-eye coordination for robotic grasping with deep learning and large-scale data collection,” The International Journal of Robotics Research,p. 0278364917710318, 2017.
[7] F. Zhang, J. Leitner, M. Milford, B. Upcroft, and P. Corke, “Towardsvision-based deep reinforcement learning for robotic motion control,”arXiv preprint arXiv:1511.03791, 2015.
[8] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learningfor fast adaptation of deep networks,” in Proceedings of the 34thInternational Conference on Machine Learning, ser. Proceedingsof Machine Learning Research, D. Precup and Y. W. Teh,Eds., vol. 70. International Convention Centre, Sydney, Australia:PMLR, 06–11 Aug 2017, pp. 1126–1135. [Online]. Available:http://proceedings.mlr.press/v70/finn17a.html
[9] A. Balaji, N. K. AA, and T. Sudarshan, “3 axis scara robot with universalgripper,” in Trends in Automation, Communications and ComputingTechnology (I-TACT-15), 2015 International Conference on. IEEE,2015, pp. 1–5.
[10] S. Sanju, N. K. A. A, and T. S. B. Sudarshan, “Autonomous roboticarm with enhanced intelligence,” International Journal of AdvancedComputational Engineering and Networking, vol. 2, pp. 12–17, 2014.
[11] V. Pereira, V. A. Fernandes, and J. Sequeira, “Low cost object sortingrobotic arm using raspberry pi,” in Global Humanitarian TechnologyConference-South Asia Satellite (GHTC-SAS), 2014 IEEE. IEEE, 2014,pp. 1–6.
[12] P. Siagian and K. Shinoda, “Web based monitoring and control ofrobotic arm using raspberry pi,” in Science in Information Technology(ICSITech), 2015 International Conference on. IEEE, 2015, pp. 192–196.
