Contact Localization and Force Estimation of Soft Tactile Sensors usingArtificial Intelligence

DongWook Kim and Yong-Lae Park

Abstract— Soft artificial skin sensors that can detect contactforces as well as their locations are attractive in various softrobotics applications. However, soft sensors made of polymermaterials have inherent limitations of hysteresis and nonlinear-ity in response, which makes it highly difficult to implementtraditional calibration techniques and yields poor estimationperformance. In this paper, we propose intelligent algorithmsbased on machine learning and logics that can improve theperformance of soft sensors. The proposed methods in thispaper could be solutions to the aforementioned long-standingproblems. They can also be used to simplify the systemcomplexity by reducing the number of signal wires. Threemachine learning techniques are discussed in this paper: anartificial neural network (ANN), the k-nearest neighbors (k-NN) algorithm, and a recurrent neural network (RNN). ThePreisach model of hysteresis and simple logics were used tosupport these algorithms. We proved that classifying contactlocations on a soft sensor is possible using simple algorithms inreal time. Also, force estimation of a single contact was possibleusing an ANN with the Preisach method. Finally, we successfullyestimated forces of multiple contact locations by predicting theoutputs of mixed RNN results.

I. INTRODUCTION

Recent advancement of robotics has made significant im-provements in various technologies. In particular, implemen-tation of soft robot technologies to force and tactile sensing,haptics, and wearable devices has rapidly expanded theirapplication areas. Highly stretchable artificial skin sensor thatcan detect contact pressures is one of core technologies insoft robotics, since soft tactile sensors are essential to buildsystems that are largely involved with physical interactionsand activities of human users.

Various research groups have demonstrated methods offabricating artificial skin sensors using highly deformablematerials. Their sensing mechanisms have a similarity ina sense that changes in electrical resistance of the sensormaterial is used to estimate the mechanical stimuli, suchas compression, stretching, and bending. For example, softsensors with embedded microfluidic channels filled with aconductive liquid can detect strains and contact pressuresbased on the resistance change of the microchannels causedby their geometric change [1]. Electrical impedance to-mography has been implemented to an elastomer matrix

This work was supported in part by the National Research Foundationof Korea (Grant NRF-2016R1A5A1938472) funded by the Korean Govern-ment (MSIT), and in part by the Technology Innovation Program (Grant20001881) funded By the Ministry of Trade, Industry & Energy (MOTIE,Korea). (Corresponding author: Yong-Lae Park)

DongWook Kim and Yong-Lae Park are with the Department of Mechan-ical and Aerospace Engineering, Seoul National University, Seoul 08826,Republic of Korea. (e-mail: shigumchis, ylpark@snu.ac.kr)

Fig. 1. Example of nonlinear and hysteretic force response from a softsensor. Loading and unloading phases make a large gap in response.

Fig. 2. Example of post-processing module for as a single element sensor,composed of amplifier, noise filter, and analog-digital converter (ADC).

embedded with conductive microfluidic channels for multi-location pressure sensing [2]. Soft sensors made of a con-ductive rubber [3],[4] and an elastomer embedded with asodium chloride solution [5] have also been proposed. Othermethods, using metal oxide semiconductor transistors [6] andcarbon particles [7], have been developed for detecting strainand pressure changes as well.

Although the aforementioned methods have shown thecapability of detecting different modes of mechanical defor-mation, these sensors have two major limitations in common.First, many of this type of sensors show highly nonlinear andhysteretic responses to input stimuli, such as pressure andstrain, due to the viscoelastic property of the polymeric basematerials. These behaviors cause difficulty in calibrating thesensor signals and may provide inaccurate output estimation.Fig. 1 shows an example of a nonlinear force response with ahysteresis loop of soft sensors. In this response, the unloadingphase generates a new trajectory rather than retracing that ofthe loading phase.

2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)Madrid, Spain, October 1-5, 2018

978-1-5386-8093-3/18/$31.00 ©2018 IEEE 7480

The second limitation is related with a space efficiencyof the sensor system. In many sensor systems, the rawoutput signals are typically low and noisy, requiring a post-processing module for signal conditioning, such as filteringand amplification, as shown in Fig. 2. For this reason, it is noteasy to make the entire sensor system compact even thoughthe sensing element itself can be miniaturized. In general,the size of the post-processing modules grows in proportionto the number of sensing elements, which means a tactilesensor with a large sensing area or a high spatial resolutionleads to a loss of space and cost efficiency.

Several trials have been made to directly characterize oralleviate the hysteresis and nonlinearity effect of soft sensors.Lacasse et al. have proposed to model the time response ofsoft sensor signals based on the Burgers model [7], whichis a well-known model to interpret the creep effect of apolymer structure. This model enables a time response to aunit step input to express an overshooting and exponentiallydecreasing output. Park et al. used linear elastic fracturemechanics (LEFM) to asymptotically plot the non-linearrelationship of the pressure-resistance curve [8]. Shin et al.embedded microbeads in the microfluidic channels in a softsensor to improve the linearity of the pressure response [9].Park et al. have demonstrated that hysteresis of a soft sensorcan be significantly reduced by changing the cross-sectionalgeometries of the embedded microfluidic channels [10].

To address the above two limitations at the same time,Han et al. have investigated the time dependency of the hys-teresis effect. They introduced a machine learning approachemploying a recurrent neural network (RNN) to analyze thesensor signal [11]. Using a deep learning technique, theywere not only able to characterize the hysteretic behaviorof the sensor but also reduce the number of signal wires.The RNN seems a proper choice in this approach, sinceRNNs are effective to deal with time dependent sequences.Although the RNN worked for characterizing the nonlinearand viscoelastic behavior of the soft sensor in this case, itshowed a limitation that it could only work during post-processing, which is not practical in real-time control.

In this paper, we demonstrate four case studies that usedifferent intelligent algorithms to solve the above two prob-lems. In the first case, we tried to estimate a highly nonlinearand hysteretic pressure response of a microfluidic soft sensorusing an artificial neural network (ANN) and the Preisachmethod. The second case was about localization of a singlecontact force using the k-nearest neighbor (k-NN) method.Multiple contacts can be also localized as shown in the thirdcase. In this case a simple logic combination approach wasused for increased computation efficiency, which enabledreal-time estimation. Finally, both locations and magnitudesof multiple contact forces were estimated using a recurrentneural network (RNN) in the last case.

II. MATERIALS AND PREPARATION

We prepared four different soft sensor samples for theexperiments of the four case studies. The overall fabricationof the sensor samples are almost the same except that a

Fig. 3. The sensor is fabricated by three steps: (a) Silicone rubber is pouredat the 3D-printed mold. (b) Cured silicone is spin-coated. (c) Liquid metaleGaIn is injected.

few specific patterning steps are slightly different for eachsample. The sensor bodies were made using 3D-printed(Object 30, Stratasys) molds. Each mold contains a uniquepattern of a microchannel into which conductive liquid isinjected. When the mold is prepared, a liquid state siliconerubber is poured in the mold and cured for about fourhours in an oven (60C), making an open-top microchannellayer (Fig. 3-a). The microchannel is sealed by bonding thetop surface of the cured layer to a thin flat silicone layermade by spin-coating (Fig. 3-b). After the silicone structureis cured for about two hours in the same oven again forcomplete bonding of the two layers, a liquid metal (eutecticgallium-Indium, eGaIn) is injected to make the microchannelelectrically conductive (Fig. 3-c). Finally, the two ends ofthe microchannel are connected to signal wires and sealedfor preventing leaking of eGaIn.

Along with sensor fabrication, a sensor readout circuitwas constructed. Each sensor sample was connected to aglobal ground and a current source made of a p-type metal-oxide-semiconductor field-effect transistor (MOSFET) regu-lator. The total resistance of the sensor was approximately5.8 Ω at the unloaded phase, which neglects channel-lengthmodulation effect. A drain current was measured between70 mA and 80 mA. The voltage signal was amplified andlow-pass filtered using an instrumentation amplifier and apassive low pass filter, respectively. A commercial load cellwas used to obtain reference force data through universalasynchronous receiver/transmitter (UART) connection to alaptop. The outputs from the soft sensor were read by amicro controller (Arduino Mega) connected to a laptop. Oncethe data from both the soft sensor and the load cell wereobtained, MATLAB (Mathworks) was used to process thedata. In addition, Tensorflow, an open-source software libraryfor data flow programming, was used to train and test thedata for machine learning.

III. METHODS

A. Force Estimation of a Single Contact: Hysteresis Com-pensation by ANN with Preisach Method

The first problem we decided to tackle was compensatinghysteresis for single contact force estimation. A RNN has

7481

Fig. 4. (a) A relay operator. (b) Illustration of the staircase memory of thePreisach Plane.

been previously implemented to characterize the hysteresiseffect by reflecting the time-dependency in the sensor data[11]. However, it is not practical for real time control due tothe low signal processing speed. We found that there are sev-eral ways to mathematically describe hysteresis phenomena,and one of them we employed in this work is the Preisachmethod [12]. This method can be easily implemented bycombining with an artificial neural network (ANN) [13].An ANN has a merit in the processing speed, since itcan be operated in real-time in a single central processingunit (CPU) level where a RNN cannot be implemented.The Preisach method uses a relay operator. The governingequation is given:

y =

∫ ∫a≥b

µ(β, α)R(β, α)[µ, ξ]dβdα (1)

F (α, β) =

∫ ∫T

R(α, β)µ(β, α)dβdα (2)

where α and β are Preisach domain variables, µ is the weightin the each location of Preisach domain, and R is the relayfunction. Fig. 4 represents a relay operator and Preisachplane. The function of two variables α and β is a one-to-onefunction to the output signal, which describes Equation 1.The output y is a weighted area in the Preisach plane, whichis illustrated in green space at Fig. 4-b. Using the Preisachtransformation, we do not have to use a RNN because thetransformed signal is not history-dependent anymore. Wemake F function which is a function of R and µ which isdescribed in Equation 2. This function is trained in ANN andused as a integral function. Instead, we can use an ANN totrain the one-to-one function of the transformed dataset.

A fully connected neural network was configured byusing Tensorflow. The network was composed of two fullyconnected layers of 25 and 12 hidden units. The rectifiedlinear unit (ReLU) was used as an activation function. TheAdam gradient optimizer and the batch regularization ofsize 64 was used to avoid overfitting. To obtain the trainingdata, a tabletop computer numerical control (CNC) millingmachine was used to compress the soft sensor repeatedlywith randomly-generated forces and rates. We collected thetotal number of 56,708 training data. 10% of them were

Fig. 5. Double-layered soft sensor design with two sensor layers (top) andactual prototype showing front and back of the sensor (bottom). Each layercontains a single microchannel with five sections with different channelwidths with increasing and decreasing orders for the 1st and 2nd layers,respectively. A, B, C, D, and E represent locations for contact localization.

used to validate the performance. Four random loading andunloading tests were done to test the performance.

B. Single Contact Localization: k-NN

In the second case, we focused on reducing the numberof wires in a soft sensor with multiple sensing elements.Previously, at least two wires were needed to operate asingle sensor: one for a current source, and the other fora ground. For localization test of a contact force, we builta soft sensor sample by connecting multiple sensing ele-ments (i.e. microchannels) in series in a soft sensor withunique dimensions for each sensing element. To amplify theuniqueness of each element, another sensor layer with thesame microchannel patterns but in an opposite direction wasprepared and stacked in parallel, as shown in Fig. 5. Whenone of the five sensing locations is compressed, a sensoroutput from each layer can be obtained. Then, a combinationof the two sensor signals from the two layers is processedand classified with the location of compression. To acquirethe training data, we compressed the five different locationsmultiple times. The output voltages and the correspondinglocations of compression were considered as a single trainingdata set in this approach. A total of 1,592 data were collectedand processed by supervised-learning based on k-NN. Thismethod is light enough to be implemented in real time.To evaluate the performance, we directly implemented thetrained structure as a MATLAB function and showed theresult in real time. During training and testing, the sensor wascompressed by a human finger instead of a CNC machine inthis case, to demonstrate the simplicity and the robustnessof our method, since compressions by a human finger makemuch larger variations during testing than those generatedby a CNC machine.

C. Multi-Contact Localization: Logic Combinations

In this method, we demonstrate that multiple contacts canbe also identified by adding a few more signal wires to thesame sensor used in the previous case. Although more signal

7482

Fig. 6. Results of force estimation of a single contact using an ANN and the Preisach method for four different loading-unloading loops.

wires were required, we tried to keep the post-processingmodule as simple as possible by employing a grid-patternsensor configuration similar to the wire configuration usedin a soft keypad [14]. In this approach, we did not use ma-chine learning but constructed a simple logical relationshipbetween the two sensor outputs to localize multiple contacts.For simplicity of testing, only locations A, C, and E inFig. 5 were used in the experiments. This method has aunique logic depending on the number and the locations ofthe contacts. Table I summarizes the entire logics in thisapproach. Numbers written in each cases V1 and V2 arethe relative amplitude of the voltage level. 1 indicates thelowest voltage amplitude, and 3 indicates the highest voltageamplitude. All the classification cases have their own uniquelogics. We tested the performance of the proposed methodby directly collecting the sensor data and by comparingthem with the corresponding logics. We applied forces tothe sensor by compressing it with human fingers in thiscase as well. This method has an advantage of extremelysimple processing and is useful when the number of sensingelements is relatively small. However, it will not be verypractical if the number of sensing elements increases, sincethe number of logic combinations will rapidly increase.

D. Multi-Contact Localization and Force Estimation: RNN

In the last case, we show the feasibility of overcomingthe two limitations (i.e. hysteresis/nonlinearity and wirecomplexity) of soft sensors at the same time. It has beenpreviously shown that both localization and estimation of asingle contact force is possible using a RNN [11]. In ourapproach, either a single or multiple contact forces wereapplied to locations B and D in the same sensor sample(Fig. 5), and the location(s) and the magnitude of the force(s)were simultaneously estimated using a RNN.

TABLE ILOGIC TABLE FOR MULTIPLE LOCATION CLASSIFICATION

Contact Locations V1 V2A 3 1C 2 2E 1 3

A, C 3+2 1+2A, E 3+1 1+3C, E 2+1 2+3

A, C, E 3+2+1 1+2+3

In this method, we first constructed a RNN that made arelationship between the force data and the sensor outputdata. The two locations were compressed multiple timeswith different speeds and forces. The proposed sequencelength of the network was 20.A total of 3,960 data and atotal 3,651 data were collected from the locations B and D,respectively. Each network was constructed with three longshort-term memory (LSTM) layers and a single fully con-nected layer. The dropout regularization was implementedwith a probability 0.5 at third LSTM layer. All the layershad 70 hidden units, and the ReLU was used as an activationfunction. The final fully-connected layer had 25 hidden units.The mean squared error was used to calculate the lossfunction. The Adam optimizer and batch analysis was usedto avoid overfitting. A total of 1,452 test data was collectedby randomly compressing locations B and D with a finger.

Using the constructed RNN, we can estimate the appliedforce by minimizing the loss function of the RNN. Theprocess of learning starts with a 1 × 10 preoccupied sequencematrix that corresponds to each location of B or D with all-zero initial values. The two sequences were considered as atime response of the estimated force value. The range of thenext sequence of the estimated force values are then deter-mined between the last value±0.4 N that was experimentallyselected. Once the network provides the new outputs, theywere examined by the loss functions to find the outputvalue with the smallest loss. The MSE between the selectedvalue’s RNN output and the voltage data is considered asa loss function. We consider the smallest value as the firstdata of the new sequence, which corresponding output valuebecomes the estimated force value of each location. Fortesting after training, we first compressed locations B andD individually to check whether the network works wellwith a single force estimation. Then, we tested simultaneouscompressions on B and D with random forces. We checkedthe peak and the tendency of the estimated sequences.

IV. RESULTS

A. Force Estimation of a Single Contact: Hysteresis Com-pensation by ANN with Preisach Method

The ANN with the Preisach method was tested using fourtesting hysteresis loops. The results are plotted in Fig. 6.The network was trained for a total of 1,000 epochs within5 minutes and 31 seconds. The MSE of the training network

7483

Fig. 7. Sensor output distribution from two sensor layers showing classifiedsensor outputs for different contact locations.

was converged to 0.003 at the final epoch, starting from20.07. The test results showed that the relationship of theforce and the voltage sequences can be expressed with afully connected layer even though it is a history dependentdata.

B. Single Contact Localization: k-NN

We compressed each location of the sensor 20 times withrandom forces. The output distribution is plotted in Fig. 7,and the result is shown in Table II. There were no estimationerrors in localizing the contacts on locations A, C and E,however, small errors on locations B and D were observed.The table shows that there were twice and five times offalse estimation on locations B and D, respectively. The totalaccuracy was 93%.

C. Multi-Contact Localization: Logic Combinations

The three locations A, C, and E were randomly com-pressed three times for each combination. The result is shownin Table III indicating that there was no false estimation forany combinations of locations.

D. Multi-Contact Localization and Force Estimation: RNN

The test data and the estimation are plotted together inFig. 8. The reference and the estimation showed reasonableagreements for both locations B and D following the peaktendency. We made eight force peaks for each location (i.e. Xor Y) at the reference data. A total of 16 peaks were detectedcorrectly. leading errors are observed in the sequences of

TABLE IITEST RESULT FOR K-NEAREST NEIGHBOR METHOD. REFER FIG. 5 TO

SEE THE SENSOR LOCATION.

Ref\Est. A B C D EA 20B 18 2C 20D 15 5E 20

150-200 and 230-240 for location B, and lagging errorsare shown in the sequences of 80-100 and 190-210 forlocation D. In addition, there are mismatching peaks in thesequence of 0-100, which is due to the sampling delay fromthe reference and the sensor data however, not from thealgorithm.

The result shows that it is possible to localize multiplecontacts with rough estimation of the applied forces althoughaccurate force estimation is not easy in this method.

V. CONCLUSION

In this paper, we discussed different approaches to estimat-ing and localizing contact forces applied to soft artificial skinsensors using artificial intelligence. To show the feasibilityof the proposed methods, we demonstrated four case studiesfocusing on overcoming the limitations of nonlinear andhysteric responses and wire complexity, which are commonlyshown in soft sensors.

In the first case, the nonlinear and hysteric force responsewas estimated with an accuracy using an ANN that is lighterin processing and more desirable for real-time control thana RNN. The second and the third cases were focused onlocalization of contact forces for a single contact and multi-contact applications, respectively. Although it was possibleto reliably identify the location of the contact using k-NN,a simple logic-based algorithm can be used if the numberof the sensing locations is not large for a high efficiencyin computation. These localization schemes are effective onreducing the signal wires and the size of the post-processingmodule. The last case study showed the feasibility of esti-mating and localizing multiple contact forces at the sametime, which was the most challenging problem. A RNN wasconstructed and implemented in this case. The result showeda reliable localization performance of multi-contacts withreasonable force estimation. Although there were some errorson force estimation, we were able to confirm that hardwarefor a soft sensor system can be considerably simplified ifan intelligent algorithm is properly combined with signalprocessing in this case.

In the future, we plan to increase the accuracy of forceestimation in the last case by implementing various signalprocessing techniques. One example would be the analysison independent components that decouples a mixed signalinto separate independent components, as discussed in [15].Another possibility would be use of capacitive-type softsensors [16],[17] combined with intelligent algorithms, since

TABLE IIITEST RESULT FOR LOGIC METHOD

Ref\Est. A C E A+C C+E E+A A+C+EA 3C 3E 3

A+C 3C+E 3E+A 3

A+C+E 3

7484

Fig. 8. Reference and estimated force data with random compressions of soft sensor. The force resolution in this method is 0.1 N.

capacitive sensing may provide a higher resolution thanresistive sensing. Another area of future work would beimplementation of the ANN-Preisach method to a real-timerobot control. In this case, the estimated force will be thefeedback input to correct errors. In this research, we showedthat various intelligent learning and logic algorithms canbe used to to analyze the history-dependent behavior ofsoft sensors. The proposed techniques for localizing andestimating contact forces in soft sensors will create a newspace for developing a new type of soft sensors for highlyinteractive soft robotic environments.

REFERENCES

[1] Y.-L. Park, B.-R. Chen, and R. J. Wood, “Design and fabrication of softartificial skin using embedded microchannels and liquid conductors,”IEEE Sensors Journal, vol. 12, no. 8, pp. 2711–2718, 2012.

[2] J.-B. Chossat, H.-S. Shin, Y.-L. Park, and V. Duchaine, “Soft tactileskin using an embedded ionic liquid and tomographic imaging,”Journal of Mechanisms and Robotics, vol. 7, no. 2, p. 021008, 2015.

[3] A. Nagakubo, H. Alirezaei, and Y. Kuniyoshi, “A deformable anddeformation sensitive tactile distribution sensor,” in Robotics andBiomimetics, 2007. ROBIO 2007. IEEE International Conference on.IEEE, 2007, pp. 1301–1308.

[4] Y. Kato and T. Mukai, “Tactile sensor without wire and sensingelement in the tactile region using new rubber material,” in Sensors:Focus on Tactile Force and Stress Sensors. InTech, 2008.

[5] S. Russo, T. Ranzani, H. Liu, S. Nefti-Meziani, K. Althoefer, andA. Menciassi, “Soft and stretchable sensor using biocompatible elec-trodes and liquid for medical applications,” Soft robotics, vol. 2, no. 4,pp. 146–154, 2015.

[6] K. Takei, T. Takahashi, J. C. Ho, H. Ko, A. G. Gillies, P. W. Leu,R. S. Fearing, and A. Javey, “Nanowire active-matrix circuitry for low-voltage macroscale artificial skin,” Nature materials, vol. 9, no. 10, p.821, 2010.

[7] M.-A. Lacasse, V. Duchaine, and C. Gosselin, “Characterization ofthe electrical resistance of carbon-black-filled silicone: Application toa flexible and stretchable robot skin,” in Robotics and Automation(ICRA), 2010 IEEE International Conference on. IEEE, 2010, pp.4842–4848.

[8] Y.-L. Park, C. Majidi, R. Kramer, P. Berard, and R. J. Wood,“Hyperelastic pressure sensing with a liquid-embedded elastomer,”Journal of Micromechanics and Microengineering, vol. 20, no. 12,p. 125029, 2010.

[9] H.-S. Shin, J. Ryu, C. Majidi, and Y.-L. Park, “Enhanced performanceof microfluidic soft pressure sensors with embedded solid micro-spheres,” Journal of Micromechanics and Microengineering, vol. 26,no. 2, p. 025011, 2016.

[10] Y.-L. Park, D. Tepayotl-Ramirez, R. J. Wood, and C. Majidi, “Influ-ence of cross-sectional geometry on the sensitivity and hysteresis ofliquid-phase electronic pressure sensors,” Applied Physics Letters, vol.101, no. 19, p. 191904, 2012.

[11] S. Han, T. Kim, D. Kim, Y.-L. Park, and S. Jo, “Use of deep learningfor characterization of microfluidic soft sensors,” IEEE Robotics andAutomation Letters, vol. 3, no. 2, pp. 873–880, 2018.

[12] J. A. Stakvik, M. R. Ragazzon, A. A. Eielsen, and J. T. Gravdahl, “Onimplementation of the preisach model: identification and inversion forhysteresis compensation,” 2015.

[13] M. R. Zakerzadeh, M. Firouzi, H. Sayyaadi, and S. B. Shouraki,“Hysteresis nonlinearity identification using new preisach model-basedartificial neural network approach,” Journal of Applied Mathematics,vol. 2011, 2011.

[14] R. K. Kramer, C. Majidi, and R. J. Wood, “Wearable tactile keypadwith stretchable artificial skin,” in Robotics and Automation (ICRA),2011 IEEE International Conference on. IEEE, 2011, pp. 1103–1107.

[15] A. Hyvarinen, J. Karhunen, and E. Oja, “Independent componentanalysis,” Studies in informatics and control, vol. 11, no. 2, pp. 205–207, 2002.

[16] A. Frutiger, J. T. Muth, D. M. Vogt, Y. Menguc, A. Campo, A. D.Valentine, C. J. Walsh, and J. A. Lewis, “Capacitive soft strain sen-sors via multicore–shell fiber printing,” Advanced Materials, vol. 27,no. 15, pp. 2440–2446, 2015.

[17] S. Laflamme, M. Kollosche, J. Connor, and G. Kofod, “Soft capac-itive sensor for structural health monitoring of large-scale systems,”Structural Control and Health Monitoring, vol. 19, no. 1, pp. 70–81,2012.

7485