MIN-Fakultät

Fachbereich Informatik

Intelligent ProsthesisFrom Sensor Data to Motion Generation

Irina Barykina

Universität HamburgFakultät für Mathematik, Informatik und NaturwissenschaftenFachbereich Informatik

Technische Aspekte Multimodaler Systeme

12. December 2016

Irina � Intelligent Prosthesis 1 / 21

OutlineMotivation Sensors EMG control systems Feedback Nice video References

1. Motivation

2. SensorsTypesMyoelectric sensor

3. EMG control systemsSystem overviewPreprocessing of dataNon-pattern recognition systemsClassi�cation with ANNLimitations

4. Feedback

5. Nice video

6. References

Irina � Intelligent Prosthesis 2 / 21

MotivationMotivation Sensors EMG control systems Feedback Nice video References

I Limb prosthesis for amputeesrehabilitation (approx. 10millions of amputees over theworld in 2008 [4]).

I Limb orthosis forrehabilitation of people witha violation of musculoskeletalsystem.

I Exoskeleton systems forassisting people in dailyroutines.

[www.touchbionics.com]

Irina � Intelligent Prosthesis 3 / 21

MotivationMotivation Sensors EMG control systems Feedback Nice video References

I Limb prosthesis for amputeesrehabilitation (approx. 10millions of amputees over theworld in 2008 [4]).

I Limb orthosis forrehabilitation of people witha violation of musculoskeletalsystem.

I Exoskeleton systems forassisting people in dailyroutines.

[www.touchbionics.com]

Irina � Intelligent Prosthesis 3 / 21

MotivationMotivation Sensors EMG control systems Feedback Nice video References

I Limb prosthesis for amputeesrehabilitation (approx. 10millions of amputees over theworld in 2008 [4]).

I Limb orthosis forrehabilitation of people witha violation of musculoskeletalsystem.

I Exoskeleton systems forassisting people in dailyroutines.

[www.touchbionics.com]

Irina � Intelligent Prosthesis 3 / 21

SensorsType of Input SignalsMotivation Sensors EMG control systems Feedback Nice video References

I Electromyography(EMG)I Mechanomyogram (MMG) or sound myogramI Electroencephalography (EEG)I Electrooculography (EOG)I Electrocorticogram (EcoG)

[www.science.education.nih.gov]Irina � Intelligent Prosthesis 4 / 21

SensorsType of Input SignalsMotivation Sensors EMG control systems Feedback Nice video References

I Electromyography(EMG)I Mechanomyogram (MMG) or sound myogramI Electroencephalography (EEG)I Electrooculography (EOG)I Electrocorticogram (EcoG)

[www.science.education.nih.gov]Irina � Intelligent Prosthesis 4 / 21

SensorsType of Input SignalsMotivation Sensors EMG control systems Feedback Nice video References

I Electromyography(EMG)I Mechanomyogram (MMG) or sound myogramI Electroencephalography (EEG)I Electrooculography (EOG)I Electrocorticogram (EcoG)

[www.science.education.nih.gov]Irina � Intelligent Prosthesis 4 / 21

SensorsType of Input SignalsMotivation Sensors EMG control systems Feedback Nice video References

I Electromyography(EMG)I Mechanomyogram (MMG) or sound myogramI Electroencephalography (EEG)I Electrooculography (EOG)I Electrocorticogram (EcoG)

[www.science.education.nih.gov]Irina � Intelligent Prosthesis 4 / 21

SensorsType of Input SignalsMotivation Sensors EMG control systems Feedback Nice video References

I Electromyography(EMG)I Mechanomyogram (MMG) or sound myogramI Electroencephalography (EEG)I Electrooculography (EOG)I Electrocorticogram (EcoG)

[www.science.education.nih.gov]Irina � Intelligent Prosthesis 4 / 21

SensorsMyoelectric SensorMotivation Sensors EMG control systems Feedback Nice video References

[MyoWare Muscle Sensor DevKit,www.sparkfun.com/products/13772]

[www.mananatomy.com/basic-anatomy]

Irina � Intelligent Prosthesis 5 / 21

EMG control systemOverviewMotivation Sensors EMG control systems Feedback Nice video References

Irina � Intelligent Prosthesis 6 / 21

Data SegmentationConstraintsMotivation Sensors EMG control systems Feedback Nice video References

De�nition

Segment is a time slot for acquiring myoelectric data considered forfeature extraction. [2]

Time constraints:

I Real time constraint: segmentlength + processing time ≤300ms

I Lower bound: 32ms

Signal state constraints:

I Transient state

I Steady state (preferred) [2]

Irina � Intelligent Prosthesis 7 / 21

Data Segmentation: WindowingWindowingMotivation Sensors EMG control systems Feedback Nice video References

I Adjacent Windowing

I Overlapped Windowing

I Continuous Segmentation(plus majority votingtechnique)

[6]

Irina � Intelligent Prosthesis 8 / 21

EMG control systemMajority VotingMotivation Sensors EMG control systems Feedback Nice video References

I Post-processing technique.

I Smoothes class decisions.

I Class at point t = most frequent class at points [t-m; t+m].

m ∗ Tprocess ≤ Tdelay ,where

Tprocess = time consumed during feature extraction,projection and classi�cation;

Tdelay = acceptable response time of the control system.

Irina � Intelligent Prosthesis 9 / 21

EMG control systemNon-pattern recognition systemMotivation Sensors EMG control systems Feedback Nice video References

T.Sono et al.[7] upper limb prosthesis with simple threshold control.

I Biceps deliver �exion, triceps -extension.

I Threshold T = 20% of maxcontraction.

I Open if closing EMG < T &&opening EMG > T

I Close if closing EMG > T &&opening EMG < T

I Otherwise => do nothing (noneed to keep musclescontracted).

[7]

Irina � Intelligent Prosthesis 10 / 21

EMG control systemNon-pattern recognition systemMotivation Sensors EMG control systems Feedback Nice video References

T.Sono et al.[7] upper limb prosthesis with simple threshold control.

I Biceps deliver �exion, triceps -extension.

I Threshold T = 20% of maxcontraction.

I Open if closing EMG < T &&opening EMG > T

I Close if closing EMG > T &&opening EMG < T

I Otherwise => do nothing (noneed to keep musclescontracted).

[7]

Irina � Intelligent Prosthesis 10 / 21

EMG control systemNon-pattern recognition system (cont.)Motivation Sensors EMG control systems Feedback Nice video References

Grasping obeject with 2 �ngers (upper line)and 1 �nger (bottom line). [7]

Irina � Intelligent Prosthesis 11 / 21

EMG control systemFeature ExtractionMotivation Sensors EMG control systems Feedback Nice video References

Red lines = mean values with applied low-pass �lter.Red dashed lines = onset of signal.

[3]

Irina � Intelligent Prosthesis 12 / 21

EMG control systemClassi�cation with ANNMotivation Sensors EMG control systems Feedback Nice video References

Vectorization of data in experiment of M.Gandolla et al. [3]

[3]

Irina � Intelligent Prosthesis 13 / 21

EMG control systemClassi�cation with ANNMotivation Sensors EMG control systems Feedback Nice video References

Calibration procedure in experiment of M.Gandolla et al. [3]

[3]

Irina � Intelligent Prosthesis 14 / 21

EMG control systemLimitationsMotivation Sensors EMG control systems Feedback Nice video References

I No tactile feedback. Visual feedback is not enough to providesubconcious control.

I No individual control on some of the muscles (true for innatelimbs as well).

I Necessity to concentrate and physically react during operation.

Irina � Intelligent Prosthesis 15 / 21

EMG control systemLimitationsMotivation Sensors EMG control systems Feedback Nice video References

I No tactile feedback. Visual feedback is not enough to providesubconcious control.

I No individual control on some of the muscles (true for innatelimbs as well).

I Necessity to concentrate and physically react during operation.

Irina � Intelligent Prosthesis 15 / 21

EMG control systemLimitationsMotivation Sensors EMG control systems Feedback Nice video References

I No tactile feedback. Visual feedback is not enough to providesubconcious control.

I No individual control on some of the muscles (true for innatelimbs as well).

I Necessity to concentrate and physically react during operation.

Irina � Intelligent Prosthesis 15 / 21

FeedbackCould device be included into user's body image?Motivation Sensors EMG control systems Feedback Nice video References

Experiment of Hernandez A. Alejandro et al. [1].

I Uses neural plasticity toassociate an unique event withnew stimuli.

I Substitutes tactile feedbackwith an electrical stimulation inhealthy arm.

I Results: brain shows activationof the sensory area related tothe amputated arm.

[1]

Irina � Intelligent Prosthesis 16 / 21

ConclusionMotivation Sensors EMG control systems Feedback Nice video References

I Patients are able to return to simple functional activities ofdaily life.

I Enhancement with biofeedback helps to prevent phantom painand neuro reconstraction.

I Unfortunately, contemporary solutions are still inconvenient

Irina � Intelligent Prosthesis 17 / 21

ConclusionMotivation Sensors EMG control systems Feedback Nice video References

I Patients are able to return to simple functional activities ofdaily life.

I Enhancement with biofeedback helps to prevent phantom painand neuro reconstraction.

I Unfortunately, contemporary solutions are still inconvenient

Irina � Intelligent Prosthesis 17 / 21

ConclusionMotivation Sensors EMG control systems Feedback Nice video References

I Patients are able to return to simple functional activities ofdaily life.

I Enhancement with biofeedback helps to prevent phantom painand neuro reconstraction.

I Unfortunately, contemporary solutions are still inconvenient

Irina � Intelligent Prosthesis 17 / 21

Nice VideoMotivation Sensors EMG control systems Feedback Nice video References

Upper limb prosthesis from John Hopkins University, AppliedPhysics Laboratory:https://www.youtube.com/watch?v=-0srXvOQlu0

[www.bloomberg.com]Irina � Intelligent Prosthesis 18 / 21

ReferencesMotivation Sensors EMG control systems Feedback Nice video References

[1] H. A. Alejandro, R. Kato, H. Yokoi, T. Arai, and T. Ohnishi.An fMRI Study on the E�ects of Electrical Stimulation asBiofeedback. In 2006 IEEE/RSJ International Conference onIntelligent Robots and Systems, pages 4336�4342, Oct. 2006.DOI:10.1109/IROS.2006.282006.

[2] M. Asghari Oskoei and H. Hu. Myoelectric control systems�Asurvey. Biomedical Signal Processing and Control, 2(4):275�294, Oct. 2007. ISSN 1746-8094.DOI:10.1016/j.bspc.2007.07.009. URLhttp://www.sciencedirect.com/science/article/pii/

S1746809407000547.

Irina � Intelligent Prosthesis 19 / 21

References (cont.)Motivation Sensors EMG control systems Feedback Nice video References

[3] M. Gandolla, S. Ferrante, D. Baldassini, M. Cotti Cottini,C. Seneci, and A. Pedrocchi. Arti�cial Neural-Network EMGClassi�er for Hand Movements Prediction, pages 640�643.Springer International Publishing, Cham, 2016. ISBN978-3-319-32703-7. DOI:10.1007/978-3-319-32703-7_123.URLhttp://dx.doi.org/10.1007/978-3-319-32703-7_123.

[4] M. LeBlanc. Give hope - give a hand, 2011. URLhttps://web.stanford.edu/class/engr110/2011/

LeBlanc-03a.pdf.

[5] C. J. D. Luca, A. Adam, R. Wotiz, L. D. Gilmore, and S. H.Nawab. Decomposition of Surface EMG Signals. Journal ofNeurophysiology, 96(3):1646�1657, Sept. 2006. ISSN0022-3077, 1522-1598. DOI:10.1152/jn.00009.2006. URLhttp://jn.physiology.org/content/96/3/1646.

Irina � Intelligent Prosthesis 20 / 21

References (cont.)Motivation Sensors EMG control systems Feedback Nice video References

[6] E. C. Orosco, N. M. Lopez, and F. di Sciascio.Bispectrum-based features classi�cation for myoelectric control.Biomedical Signal Processing and Control, 8(2):153 � 168,2013. ISSN 1746-8094.DOI:http://dx.doi.org/10.1016/j.bspc.2012.08.008.URL http://www.sciencedirect.com/science/article/

pii/S1746809412000900.

[7] T. Sono, L. Menegaldo, and M. Pinotti. Hand prosthesisprototype controlled by EMG and vibrotactile force feedback. In5th IEEE RAS/EMBS International Conference on BiomedicalRobotics and Biomechatronics, pages 91�95, Aug. 2014.DOI:10.1109/BIOROB.2014.6913758.

Irina � Intelligent Prosthesis 21 / 21