Presentation: Finding Common Ground: A Survey of Capacitive …€¦ · Frequency Capacitive...

Finding Common Ground: A Survey of Capacitive Sensing in Human-Computer InteractionTobias Grosse-Puppendahl, Christian Holz, Gabe Cohn, Raphael Wimmer, Oskar Bechtold, Steve Hodges, Matthew S. Reynolds, Joshua R. Smith

Activ

ePa

ssiv

e

Shunt Mode Transmit Mode Loading ModeReceive Mode

Not applicable

Frank Beck and Bent Stumpe: Two devices for operator interaction in the central control of the new CERN accelerator

capacitive sensor capacitive sensing mutual capacitance sensing Capacitively coupled body mmunication capacitive touch sensor capacitive touch sensing Electric Potential Sensing PowerLinsitioning Loading mode electric field sensing Capacitive Proximity Sensing Electrostatic Inductio ctric Field Imaging n/a Measurement of Electric Field bioimpedance Passive Sensing of Changes an Body Electric Potential capacitive multi-touch sensing electro-magnetic sensing Ambient elect

eld sensing TouchNet Intrabody Communication Intrabody communications Mutual-Capacitance sing AC hum detection combined inductive-conductive proximity sensing carrier-frequency metho

ear field imaging Body-coupled communications capacitive coupling capacitance proximity sensing body area communication touch-identification tokens Self-Capacitance capacitive sensor arrays capacitance measuring compressive sensing on capacitive touch screens active capacitive sensing apacitive positioning capacitance sensor intra-body communication human body communication pacitive finger sensing 2.5D capacitive touch sensor capacitive sensor array capacitive multitouchensor electric field ranging projected capacitance capacitive touch panel Electrostatic Potential ensing capacitive touch communication time-multiplexed loading mode capacitive sensing Swept Frequency Capacitive Sensing Capacitive widgets indirect capacitive sensing using electromagnetic rference single capacitive touch sensor capacitive touch-sensing EM sensing capacitive coupling

low-frequency signals electromagnetic noise sensing electromagnetic interference capacitive teractions capacitive charging time passive measurement of static electric field of the environment

acitive and resistive touch sensing static electric field sensing capacitive position measurements sk potential level Swept Frequency Capacitive Sensing Capacitive Near Field Communication

capacitive sensor capacitive sensing mutual capacitance sensing Capacitively coupled body mmunication capacitive touch sensor capacitive touch sensing Electric Potential Sensing PowerLinsitioning Loading mode electric field sensing Capacitive Proximity Sensing Electrostatic Inductio ctric Field Imaging n/a Measurement of Electric Field bioimpedance Passive Sensing of Changes an Body Electric Potential capacitive multi-touch sensing electro-magnetic sensing Ambient elect

eld sensing TouchNet Intrabody Communication Intrabody communications Mutual-Capacitance sing AC hum detection combined inductive-conductive proximity sensing carrier-frequency metho

ear field imaging Body-coupled communications capacitive coupling capacitance proximity sensing body area communication touch-identification tokens Self-Capacitance capacitive sensor arrays capacitance measuring compressive sensing on capacitive touch screens active capacitive sensingapacitive positioning capacitance sensor intra-body communication human body communication pacitive finger sensing 2.5D capacitive touch sensor capacitive sensor array capacitive multitouchensor electric field ranging projected capacitance capacitive touch panel Electrostatic Potential ensing capacitive touch communication time-multiplexed loading mode capacitive sensing Swept Frequency Capacitive Sensing Capacitive widgets indirect capacitive sensing using electromagnetic rference single capacitive touch sensor capacitive touch-sensing EM sensing capacitive coupling

low-frequency signals electromagnetic noise sensing electromagnetic interference capacitive teractions capacitive charging time passive measurement of static electric field of the environment

acitive and resistive touch sensing static electric field sensing capacitive position measurements sk potential level Swept Frequency Capacitive Sensing Capacitive Near Field Communication

capacitive sensor capacitive sensing mutual capacitance sensing Capacitively coupled body mmunication capacitive touch sensor capacitive touch sensing Electric Potential Sensing PowerLinsitioning Loading mode electric field sensing Capacitive Proximity Sensing Electrostatic Inductio ctric Field Imaging n/a Measurement of Electric Field bioimpedance Passive Sensing of Changes an Body Electric Potential capacitive multi-touch sensing electro-magnetic sensing Ambient elect

eld sensing TouchNet Intrabody Communication Intrabody communications Mutual-Capacitance sing AC hum detection combined inductive-conductive proximity sensing carrier-frequency metho

ear field imaging Body-coupled communications capacitive coupling capacitance proximity sensing body area communication touch-identification tokens Self-Capacitance capacitive sensor arrays capacitance measuring compressive sensing on capacitive touch screens active capacitive sensing apacitive positioning capacitance sensor intra-body communication human body communication pacitive finger sensing 2.5D capacitive touch sensor capacitive sensor array capacitive multitouchensor electric field ranging projected capacitance capacitive touch panel Electrostatic Potential ensing capacitive touch communication time-multiplexed loading mode capacitive sensing Swept Frequency Capacitive Sensing Capacitive widgets indirect capacitive sensing using electromagnetic rference single capacitive touch sensor capacitive touch-sensing EM sensing capacitive coupling

low-frequency signals electromagnetic noise sensing electromagnetic interference capacitive teractions capacitive charging time passive measurement of static electric field of the environment

acitive and resistive touch sensing static electric field sensing capacitive position measurements sk potential level Swept Frequency Capacitive Sensing Capacitive Near Field Communication

A New Taxonomy

Research Challenges

Activ

ePa

ssiv

e

Shunt Mode Transmit Mode Loading ModeReceive Mode

Not applicable

Proposed Taxonomy

Activ

ePa

ssiv

e

Shunt Mode Transmit Mode Loading ModeReceive Mode

Not applicable

Proposed Taxonomy

Activ

ePa

ssiv

e

Shunt Mode Transmit Mode Loading ModeReceive Mode

Not applicable

Proposed Taxonomy

capacitive sensor capacitive sensing mutual capacitance sensing Capacitively coupled body mmunication capacitive touch sensor capacitive touch sensing Electric Potential Sensing PowerLinsitioning Loading mode electric field sensing Capacitive Proximity Sensing Electrostatic Inductio ctric Field Imaging n/a Measurement of Electric Field bioimpedance Passive Sensing of Changes an Body Electric Potential capacitive multi-touch sensing electro-magnetic sensing Ambient elect

eld sensing TouchNet Intrabody Communication Intrabody communications Mutual-Capacitance sing AC hum detection combined inductive-conductive proximity sensing carrier-frequency metho

ear field imaging Body-coupled communications capacitive coupling capacitance proximity sensing body area communication touch-identification tokens Self-Capacitance capacitive sensor arrays capacitance measuring compressive sensing on capacitive touch screens active capacitive sensing apacitive positioning capacitance sensor intra-body communication human body communication pacitive finger sensing 2.5D capacitive touch sensor capacitive sensor array capacitive multitouchensor electric field ranging projected capacitance capacitive touch panel Electrostatic Potential ensing capacitive touch communication time-multiplexed loading mode capacitive sensing Swept Frequency Capacitive Sensing Capacitive widgets indirect capacitive sensing using electromagnetic rference single capacitive touch sensor capacitive touch-sensing EM sensing capacitive coupling

low-frequency signals electromagnetic noise sensing electromagnetic interference capacitive teractions capacitive charging time passive measurement of static electric field of the environment

acitive and resistive touch sensing static electric field sensing capacitive position measurements sk potential level Swept Frequency Capacitive Sensing Capacitive Near Field Communication

A New Taxonomy

Research Challenges

Crawling > 5900 Papers indexed by IEEE and ACM 1.

Crawling > 5900 Papers indexed by IEEE and ACM 1.

Review of each paper: Does it match our criteria? 2.

Tagging of 193 relevant papers by > 2 authors3.

Crawling > 5900 Papers indexed by IEEE and ACM 1.

Review of each paper: Does it match our criteria? 2.

[Video of Tagging Process]

0

20

40

60

80

100

120

140

NovelApplications of

CapacitiveSensing

HardwareContribution

QuantitativeStudy

Algorithm Model ofElectrical

Properties

Toolkits forCapacitive

Sensing

SimulationResults

Type of Contribution (193 Papers)Read our paper: aka.ms/capsense

capacitive sensor capacitive sensing mutual capacitance sensing Capacitively coupled body mmunication capacitive touch sensor capacitive touch sensing Electric Potential Sensing PowerLinsitioning Loading mode electric field sensing Capacitive Proximity Sensing Electrostatic Inductio ctric Field Imaging n/a Measurement of Electric Field bioimpedance Passive Sensing of Changes an Body Electric Potential capacitive multi-touch sensing electro-magnetic sensing Ambient elect

eld sensing TouchNet Intrabody Communication Intrabody communications Mutual-Capacitance sing AC hum detection combined inductive-conductive proximity sensing carrier-frequency metho

ear field imaging Body-coupled communications capacitive coupling capacitance proximity sensing body area communication touch-identification tokens Self-Capacitance capacitive sensor arrays capacitance measuring compressive sensing on capacitive touch screens active capacitive sensing apacitive positioning capacitance sensor intra-body communication human body communication pacitive finger sensing 2.5D capacitive touch sensor capacitive sensor array capacitive multitouchensor electric field ranging projected capacitance capacitive touch panel Electrostatic Potential ensing capacitive touch communication time-multiplexed loading mode capacitive sensing Swept Frequency Capacitive Sensing Capacitive widgets indirect capacitive sensing using electromagnetic rference single capacitive touch sensor capacitive touch-sensing EM sensing capacitive coupling

low-frequency signals electromagnetic noise sensing electromagnetic interference capacitive teractions capacitive charging time passive measurement of static electric field of the environment

acitive and resistive touch sensing static electric field sensing capacitive position measurements sk potential level Swept Frequency Capacitive Sensing Capacitive Near Field Communication

A New Taxonomy

Research Challenges

Activ

e Ca

paci

tive

Sens

ing

Shunt Mode

A Taxonomy for Capacitive Sensing

Thomas G. Zimmerman, Joshua R. Smith, Joseph A. Paradiso, David Allport, and Neil Gershenfeld. 1995. Applying Electric Field Sensing to Human-computer Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’95).

Activ

e Ca

paci

tive

Sens

ing

Shunt Mode

A Taxonomy for Capacitive Sensing

Thomas G. Zimmerman, Joshua R. Smith, Joseph A. Paradiso, David Allport, and Neil Gershenfeld. 1995. Applying Electric Field Sensing to Human-computer Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’95).

Jun Rekimoto: SmartSkin: An Infrastructure for Freehand Manipulation on Interactive Surfaces. CHI ‘02.

SK Lee, William Buxton, and K. C. Smith. 1985. A multi-touch three dimensional touch-sensitive tablet. CHI’85.

Activ

e Ca

paci

tive

Sens

ing

Shunt Mode

A Taxonomy for Capacitive Sensing

Thomas G. Zimmerman, Joshua R. Smith, Joseph A. Paradiso, David Allport, and Neil Gershenfeld. 1995. Applying Electric Field Sensing to Human-computer Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’95).

Joshua R. Smith: Electric Field Imaging (PhD Thesis). 1999.

Jun Rekimoto: SmartSkin: An Infrastructure for Freehand Manipulation on Interactive Surfaces. CHI ‘02.

SK Lee, William Buxton, and K. C. Smith. 1985. A multi-touch three dimensional touch-sensitive tablet. CHI’85.

Activ

e

Shunt Mode

A Taxonomy for Capacitive Sensing

Transmit Mode Receive Mode

Activ

e

Shunt Mode

A Taxonomy for Capacitive Sensing

Transmit Mode Receive Mode

Paul Dietz and Darren Leigh. 2001. DiamondTouch: a multi-user touch technology. UIST ’01.

Activ

e

Shunt Mode

A Taxonomy for Capacitive Sensing

Transmit Mode Loading ModeReceive Mode

Activ

e

Shunt Mode

A Taxonomy for Capacitive Sensing

Transmit Mode Loading ModeReceive Mode

Jun Rekimoto, Takaaki Ishizawa, Carsten Schwesig, and Haruo Oba. 2003. PreSense: interaction techniques for finger sensing input devices. UIST ‘03.

Chia-Hsun Lee and Ted Selker. 2004. iSphere: a proximity-based 3D input device. In SIGGRAPH ‘04

Activ

ePa

ssiv

e

Shunt Mode Transmit Mode Loading ModeReceive Mode

A Taxonomy for Capacitive Sensing

Activ

ePa

ssiv

e

Shunt Mode Transmit Mode Loading ModeReceive Mode

A Taxonomy for Capacitive Sensing

H. Prance, P. Watson, R. J. Prance, and S. T. Beardsmore-Rust. 2012. Position and movement sensing at metre standoff distances using ambient electric field. Measurement Science and Technology 23, 11 (2012), 115101.

Activ

ePa

ssiv

e

Shunt Mode Transmit Mode Loading ModeReceive Mode

A Taxonomy for Capacitive Sensing

Activ

ePa

ssiv

e

Shunt Mode Transmit Mode Loading ModeReceive Mode

A Taxonomy for Capacitive Sensing

Rekimoto et al.: Sensing GamePad: electrostatic potential sensing for enhancing entertainment oriented interactions. CHI ‘04

Activ

ePa

ssiv

e

Shunt Mode Transmit Mode Loading ModeReceive Mode

A Taxonomy for Capacitive Sensing

Activ

ePa

ssiv

e

Shunt Mode Transmit Mode Loading ModeReceive Mode

A Taxonomy for Capacitive Sensing

Gabe Cohn, Daniel Morris, Shwetak Patel, and Desney Tan. Humantenna: Sensing Gestures Using the Body as an Antenna. In CHI ’12.

Activ

ePa

ssiv

e

Shunt Mode Transmit Mode Loading ModeReceive Mode

Not applicable

A Taxonomy for Capacitive Sensing Read our paper: aka.ms/capsense

capacitive sensor capacitive sensing mutual capacitance sensing Capacitively coupled body mmunication capacitive touch sensor capacitive touch sensing Electric Potential Sensing PowerLinsitioning Loading mode electric field sensing Capacitive Proximity Sensing Electrostatic Inductio ctric Field Imaging n/a Measurement of Electric Field bioimpedance Passive Sensing of Changes an Body Electric Potential capacitive multi-touch sensing electro-magnetic sensing Ambient elect

eld sensing TouchNet Intrabody Communication Intrabody communications Mutual-Capacitance sing AC hum detection combined inductive-conductive proximity sensing carrier-frequency metho

ear field imaging Body-coupled communications capacitive coupling capacitance proximity sensing body area communication touch-identification tokens Self-Capacitance capacitive sensor arrays capacitance measuring compressive sensing on capacitive touch screens active capacitive sensing apacitive positioning capacitance sensor intra-body communication human body communication pacitive finger sensing 2.5D capacitive touch sensor capacitive sensor array capacitive multitouchensor electric field ranging projected capacitance capacitive touch panel Electrostatic Potential ensing capacitive touch communication time-multiplexed loading mode capacitive sensing Swept Frequency Capacitive Sensing Capacitive widgets indirect capacitive sensing using electromagnetic rference single capacitive touch sensor capacitive touch-sensing EM sensing capacitive coupling

low-frequency signals electromagnetic noise sensing electromagnetic interference capacitive teractions capacitive charging time passive measurement of static electric field of the environment

acitive and resistive touch sensing static electric field sensing capacitive position measurements sk potential level Swept Frequency Capacitive Sensing Capacitive Near Field Communication

A New Taxonomy

Research Challenges

Research Challenge #1: Sensitivity to Grounding

How can we bring these powerful interactions to devices that only have floating references for practical applications?

Research Challenge #1: Sensitivity to Grounding

Research Challenge #1: Sensitivity to Grounding

Masaaki Fukumoto and Mitsuru Shinagawa.CarpetLAN: A Novel Indoor Wireless(-like)Networking and Positioning System. Ubicomp ‘05

Electro-optic crystals:

A. Pouryazdan, R. J. Prance, H. Prance, and D. Roggen. 2016. Wearable electric potential sensing: a new modality sensing hair touch and restless leg movement. UbiComp ‘16

Ultra-high input impedance sensors:

Valkyrie Savage et al.: Midas: fabricating custom capacitive touch sensors to prototype interactive objects. UIST ’12.

Cutting Ink-based Printing

Nan-Wei Gong et al.: PrintSense: a versatile sensing technique to support multimodal flexible surface interaction. CHI ’14.

3D Printing

Martin Schmitz et al.: Capricate: A Fabrication Pipeline to Design and 3D Print Capacitive Touch Sensors for Interactive Objects. UIST ’15

Research Challenge #2: Support for End-to-End Prototyping

Valkyrie Savage et al.: Midas: fabricating custom capacitive touch sensors to prototype interactive objects. UIST ’12.

Cutting Ink-based Printing

Nan-Wei Gong et al.: PrintSense: a versatile sensing technique to support multimodal flexible surface interaction. CHI ’14.

3D Printing

Martin Schmitz et al.: Capricate: A Fabrication Pipeline to Design and 3D Print Capacitive Touch Sensors for Interactive Objects. UIST ’15

We are making great advances with prototyping electrodes….But real end-to-end systems are still hard to prototype.

Research Challenge #2: Support for End-to-End Prototyping

Hsin-Liu (Cindy) Kao, Artem Dementyev, Joseph A.Paradiso, and Chris Schmandt. 2015. NailO: NailO: Fingernails as an Input Surface. CHI ’15.

Research Challenge #2: Support for End-to-End Prototyping

Hsin-Liu (Cindy) Kao, Artem Dementyev, Joseph A.Paradiso, and Chris Schmandt. 2015. NailO: NailO: Fingernails as an Input Surface. CHI ’15.

Super small and thin prototyping electronicsfor capacitive sensing are needed.

Research Challenge #2: Support for End-to-End Prototyping

Research Challenge #2: Support for End-to-End Prototyping

1. Reduce the size of prototypingelectronics

2. Reduce the size of the battery by usinghybrid active / passive capacitive sensing

Steve Hodges, Nicolas Villar, Nicholas Chen, Tushar Chugh, Jie Qi, Diana Nowacka, and Yoshihiro Kawahara. Circuit stickers: peel-and-stick construction of interactive electronic prototypes. CHI ’14.

Nan-Wei Gong, Steve Hodges, and Joseph A. Paradiso. Leveraging conductive inkjet technology to build a scalable and versatile surface for ubiquitous sensing. UbiComp ’11.

Ivan Poupyrev, Nan-Wei Gong, Shiho Fukuhara, Mustafa Emre Karagozler, Carsten Schwesig, and Karen E. Robinson. Project Jacquard: Interactive Digital Textiles at Scale. CHI ‘16.

Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jürgen Steimle. iSkin: Flexible, Stretchable and Visually Customizable On-Body Touch Sensors for Mobile Computing.

Research Challenge #3: Enabling Flexible and Stretchable Applications

Ivan Poupyrev, Nan-Wei Gong, Shiho Fukuhara, Mustafa Emre Karagozler, Carsten Schwesig, and Karen E. Robinson. Project Jacquard: Interactive Digital Textiles at Scale. CHI ‘16.

Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jürgen Steimle. iSkin: Flexible, Stretchable and Visually Customizable On-Body Touch Sensors for Mobile Computing.

How can we make touch and gesture recognition robust?

Research Challenge #3: Enabling Flexible and Stretchable Applications

Liwei Chan, Stefanie Mueller, Anne Roudaut, and Patrick Baudisch. CapStones and ZebraWidgets. CHI ’12.

Tam Vu, Akash Baid, Simon Gao, Marco Gruteser, Richard Howard, Janne Lindqvist, Predrag Spasojevic, and Jeffrey Walling. Distinguishing users with capacitive touch communication. MobiCom 2012.

Passive Tags Active Tags

Research Challenge #4: Enriching Sensing with Communications

Liwei Chan, Stefanie Mueller, Anne Roudaut, and Patrick Baudisch. CapStones and ZebraWidgets. CHI ’12.

Tam Vu, Akash Baid, Simon Gao, Marco Gruteser, Richard Howard, Janne Lindqvist, Predrag Spasojevic, and Jeffrey Walling. Distinguishing users with capacitive touch communication. MobiCom 2012.

Passive Tags Active Tags

Higher spatial and temporal resolution touch-screens would make a difference.

Research Challenge #4: Enriching Sensing with Communications

Mehrdad Hessar, Vikram Iyer, and Shyamnath Gollakota. 2016. Enabling On-body Transmissions with Commodity Devices. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’16).

Christian Holz and Marius Knaust. 2015. Biometric Touch Sensing. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology - UIST ’15.

Receive Transmit

Research Challenge #4: Enriching Sensing with Communications

Mehrdad Hessar, Vikram Iyer, and Shyamnath Gollakota. 2016. Enabling On-body Transmissions with Commodity Devices. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’16).

Christian Holz and Marius Knaust. 2015. Biometric Touch Sensing. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology - UIST ’15.

Receive Transmit

Commercially available touchscreens impose limits on communication mechanisms

Research Challenge #4: Enriching Sensing with Communications

Research Challenges: Touch Screen Research @ CHI

Jun Rekimoto. SmartSkin: an infrastructure forfreehand manipulation on interactive surfaces. CHI ’02.

Darren Leigh, Clifton Forlines, Ricardo Jota, Steven Sanders, and Daniel Wigdor. High Rate, Low-latency Multi-touch Sensing with Simultaneous Orthogonal Multiplexing. UIST ‘14

Research Challenges: Touch Screen Research @ CHI

Jun Rekimoto. SmartSkin: an infrastructure forfreehand manipulation on interactive surfaces. CHI ’02.

Darren Leigh, Clifton Forlines, Ricardo Jota, Steven Sanders, and Daniel Wigdor. High Rate, Low-latency Multi-touch Sensing with Simultaneous Orthogonal Multiplexing. UIST ‘14

Reviving touch screen research can enable great new tangible and wearable experiences.

Activ

ePa

ssiv

e

Shunt Mode Transmit Mode Loading ModeReceive Mode

Not applicable

Conclusion Read our paper: aka.ms/capsense

Capacitance is Everywhere!

Research Challenges

• Towards Better Reproducibility• Sensitivity to Grounding• Implications for Real-World Deployments• Support for End-to-End Prototyping• Reducing Form Factor & Instrumentation• Enabling Flexible and Stretchable Applications• Unifying Approaches to Interpret Electric Fields• Enriching Sensing with Communications