SelfSustainableCHI: Self-PoweredSustainable Interfaces and Interactions

Yogesh Kumar MeenaComputational FoundrySwansea University, UKy.k.meena@swansea.ac.uk

Niels HenzeUniversity of RegensburgRegensburg, Germanyniels.henze@ur.de

Xing-Dong YangDartmouth CollegeHanover, USAxing-dong.yang@dartmouth.edu

Steve HodgesMicrosoft ResearchCambridge, UKsteve.hodges@microsoft.com

Markus LöchtefeldAalborg UniversityAalborg, Denmarkmloc@create.aau.dk

Matt JonesComputational FoundrySwansea University, UKmatt.jones@swansea.ac.uk

Matt CarnieCollege of EngineeringSwansea University, UKm.j.carnie@swansea.ac.uk

Nivedita AroraGregory AbowdGeorgia Institute of TechnologyGeorgia, USAnivedita.arora@gatech.eduabowd@gatech.edu

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).

© 2020 Copyright held by the owner/author(s).CHI’20 Extended Abstracts, April 25–30, 2020, Honolulu, HI, USAACM 978-1-4503-6819-3/20/04.10.1145/3334480.3375167

AbstractThe continued proliferation of computing devices comeswith an ever-increasing energy requirement, both duringproduction and use. As awareness of the global climateemergency increases, self-powered and sustainable (Self-Sustainable) interactive devices are likely to play a valuablerole. In this workshop we bring together researchers andpractitioners from design, computer science, materials sci-ence, engineering and manufacturing industries working onthis new area of endeavour. The workshop will provide aplatform for participants to review and discuss challengesand opportunities associated with self-powered and sus-tainable interfaces and interactions, develop a design spaceand identify opportunities for future research.

Author KeywordsInterfaces and Interactions; Internet-of-Things (IoT); Internet-of-Materials (IoM); Photovoltaics (PV); Electrochromic (EC)display; e-ink displays.

BackgroundComputational devices such as Internet-of-Things (IoT)connected sensors, actuators, smart and interactive dis-plays, digital signboards, smartwatches, voice controllers,etc. have all made tremendous progress over the last 10years. However these systems must typically either be

plugged into a constant source of energy or rely on bat-teries.

The growing significance of climate change suggests thatwe cannot rely on these same sources of energy forever.With demand for devices such as those listed above grow-ing globally as standards of living increase, the devices andservices we use, plus the energy sources that power themmust be more sustainable. A two-pronged strategy natu-rally emerges – powering our technologies from renewablesources while also making more intelligent devices that aremore energy efficient.

Energy harvesting is a promising solution. Many researchersand industries are investigating novel solutions to har-vest and store energy to achieve self-sustainable IoT de-vices. Research into ambient energy harvesting and ul-tralow power electronics has produced some interestingdevices which are capable of harvesting energy from a va-riety of sources, including ambient indoor lighting [2], ther-mal sources [14], mechanical vibrations [6], and ambientRF [12].

The amount of energy harvested clearly governs what’spossible–sensing, actuation, communication or storage areall possible and have been demonstrated in various self-sustainable applications. For instance, photovoltaic (PV)materials have been combined with ultra-low power displays(see Fig 1 (top)) [8].

To allow self-sustainable devices to have more functionality,energy harvesters can be used not just as power sourcesbut also as self-sustainable sensors. For example, photo-voltaic sensors that harvest energy from light can be usedto detect changes in ambient light intensity, and also to de-tect the motion of hand gestures. Watches and eye-glasseshave been

Figure 1: Top:Ultra-low powerdisplays combine an e-paperdisplay with photovoltaic energyharvesting [8], Middle two: flexibleand transparent PVmaterials [7], [15], Bottom:transparent and flexible displaysbased on electrochromics [10].

prototyped using light to both detect hand gestures with ahigh degree of accuracy and provide self-sustainable op-eration [13]. Another example is to use mechanical energyharvesters to both sense mechanical vibration and com-municate that information. SATURN [4, 3] is a thin flex-ible material which can sense tiny mechanical vibrationcaused by human voice or touch and then transmit it usingRF backscatter technology (see Fig 2). Ubiquitous pres-ence of such self-sustainable, or very low energy, requiringsurfaces can provide enormous contextual understanding,interactive and surveillance capabilities.

Since power is a major design constraint for the functional-ity a self-sustainable device can provide, researchers haveworked on leveraging in-situ power sources to design ap-plications. For example explicit interaction performed on adevice by a user can be used to power it. Interactive Gen-erator (InGen) [5] and Peppermill [16] are examples ofhuman-powered user interface devices which scavenge en-ergy from the physical effort required to operate them. Forexample, InGen supports both haptic feedback and wire-less communication based entirely on energy provided bythe user during interaction. Another example of an in-situpower source is the wireless infrastructure present in theenvironment, be it phone or RF transmitters. RFTouchPads[9] and Tip-Tap [11] leveraged radio-frequency identification(RFID) technology nearby to allow for 2D touch sensing anddiscrete 2D fingertip input respectively. RF fields have beenshown to allow interaction with devices with 3D gestures [1].Mobile phones carried by users can also be leveraged aswireless power source (NFC [18]) or communication (e.gFM backscatter [17]) method.

Another trend in self-sustainable devices is to build themin a form factor that they can be easily instrumented ontoeveryday objects and surfaces to support more natural user

interaction. Self-sustainable displays that are flexible, trans-parent and coloured have been made using patterned PVmaterials [7], [15] (see the example of flexible and trans-parent conducting adhesive of PV materials in Fig 1 (mid-dle)). Customisable electrochromic (EC) displays [10] (seeexample of display in Fig 1 (bottom)) enable unique devicesthat show promise of being practically incorporated into anydomestic object. SATURN [4, 3] is a wireless mechanicalsensing material which looks and feels so like paper thatit can be embedded into everyday objects, yet it behaveslike a wireless microphone (see Fig 2). Finally, advances inadditive manufacturing, flexible electronics and energy har-vesting techniques allow us to build novel self-sustainablecomputational materials which in the words of Mark Weiser,“can weave themselves into the fabric of our everyday livesuntil they are indistinguishable from it”, leading to the rise ofa new generation of computing, Internet of Materials (IoM).

Figure 2: Top: SATURN - thin andflexible self-poweredmicrophone [4] , Bottom: ZEUSSS: Zero-energy Ubiquitous SoundSensing Surface [3].

Workshop Topics of Interest: The topics of interest for theworkshop include, but are not limited to the following:

• Methods, materials, and technologies for self-poweredinterfaces and interactions.

• Novel interfaces and interaction styles afforded byenergy harvesting and low-power materials.

• Scenarios and contexts of use for sustainable inter-faces and interactions.

Goal of the workshop: In this workshop, we will discussthe challenges and opportunities associated with self-poweredand sustainable interfaces and interactions. It will bring to-gether an international and cross-disciplinary group of re-searchers from academia and industry to collaborate andexplore the the topic.

OrganisersThe organising team combines expertise and interests frommaterial science, HCI, ubiquitous computing and systemengineering.

Yogesh Kumar Meena is a Research Officer in the FutureInteraction Technology Lab at Swansea University. His re-search focuses on Self-Powered Internet-of-Things (IoT)devices (photovoltaic cells, digital displays) and Human-Machine Interface (multimodal interactions, brain-computerinterfaces, gesture detection).

Nivedita Arora is PhD student in Ubiquitous ComputingLab at Georgia Institute of Technology. She combines learn-ing from HCI, material science, chemical, electrical and me-chanical engineering to develop computational material thatcan self-sustainably sense, compute, actuate and communi-cate.

Xing-Dong Yang is an Assistant Professor of ComputerScience at Dartmouth College. His research contributesnew concepts and working prototypes that advance thetechnology of interactive smart “things” through (1) betterinput devices and techniques; (2) more expressive non-visual output mechanisms; and (3) interactive systems thatconsume less power.

Markus Löchtefeld is an Associate Professor in the De-partment of Architecture, Design and Media Technologyat Aalborg University, Denmark. His research focuses onwearable and tangible computing as well as novel prototyp-ing and fabrication techniques.

Matt Carnie is Associate Professor in Materials Engineer-ing, at the College of Engineering, Swansea University. Heworks on a number of photovoltaic technologies, not leastperovskites which have shown exceptional ambient light

harvesting.

Niels Henze is professor for Media Informatics at the Uni-versity of Regensburg. His research interests are mobilehuman-computer interaction and pervasive computing. Hisrecent work focuses on what makes mobile devices socialacceptability and how we will interact with them in the fu-ture.

Steve Hodges joined Microsoft’s Cambridge research labin 2004 with the ambition of building hardware systems thatchange people’s perceptions of technology and how it canbe used. His technical expertise spans interactive systems,connected devices, wireless communications, novel sensingand displays, embedded camera systems, location sys-tems, energy management, security, wearable technologiesand rapid prototyping.

Matt Jones is a research professor at the ComputationalFoundry, Swansea University. His work considers bothemerging technologies and "emergent" users in rural andurban settings notably in India and sub-Saharan Africa.

Gregory Abowd is Regents’ Professor and J.Z. Liang Chairin School of Interactive Computing at Georgia Institute ofTechnology. His research interests concern how the ad-vanced information technologies of ubiquitous comput-ing (or ubicomp) impact our everyday lives when they areseamlessly integrated into our living spaces.

WebsiteThe workshop web pages can be found at (http://cs.swansea.ac.uk/~SelfSustainableCHI/).

Pre-Workshop PlansWe will distribute a call for position papers in design, com-puter science, materials science, engineering and manu-

facturing communities. We will announce the Call for Par-ticipation in mailing lists and calendars (e.g., ACM, CHI-announcements, Interaction-Design.org, WikiCFP) and so-cial media (e.g., Twitter, Facebook). The calls will also beposted on the workshop website, along with other detailsabout the workshop. Furthermore, we will directly contactresearchers and practitioners who are likely to be interestedin the workshop and write to relevant institutions, projectsand companies. We will continue our efforts of promotingthe workshop and getting in touch with potential participantsduring the period leading up to the position paper deadline.We have already made a list of 40 potential collaboratorsand attendees from industry, startups, design/HCI, materialscience, computer science and engineering.

Workshop StructureThe one-day workshop will be organized in three phases.The workshop will start with an introductory round and anoverview of the grand challenges and opportunities in self-powered sustainable interfaces and interactions research.Then three phases will be used to drive an agenda for fu-ture research and innovation. These phases will be seededby the participants’ position papers and prototypes.

Phase 1: Technological possibilities

• This phase will concentrate on deepening under-standing of the materials, methods, technologiesavailable for self-powered interfaces and interactions.

Phase 2: Interface and Interaction Paradigm Shift

• Given the emerging materials, methods, and tech-nologies, this phase will explore the features of thenew interaction paradigms that they afford. We will,for example, consider granularity of input/ output andinterface dynamics (e.g. speed of response). We will

also consider possible applications afforded by thisparadigm shift, and highlight potential timescales foradoption.

Phase 3: Setting an agenda for transformation

• The outcome of this phase will be a coherent set ofresearch directions that will enable other researchersto pursue the goal of creating technological futuresthat will enable rich digital interactions through re-sponsible and sustainability-orientated innovation.

Post-Workshop PlansThe results of the workshop will be communicated to thelarger HCI community via a magazine article (e.g., ACMInteractions). In this article, we will define future designspaces and opportunities for future research.

We will also invite all workshop participants to submit an ex-tended article of their submission for a special journal issueon Self-Powered Sustainable Interfaces and Interactions(e.g., in Personal Ubiquitous Computing). This informationand call for papers will also be shared on the website andwill be open to other interested researchers in the commu-nity.

Call for ParticipationWe invite position papers for the CHI 2020 Workshop onSelf-powered Sustainable Interfaces and Interactions. Thisone-day workshop will offer a cross-disciplinary forum ofdiscussion and knowledge exchange for both academicsand practitioners.

This workshop focuses on three grand challenges in self-powered sustainable interfaces and interactions research:

• Methods, materials, and technologies for self-poweredinterfaces and interactions.

• Novel interfaces and interaction styles afforded byenergy harvesting and low-power materials.

• Effective and delightful scenarios and contexts of usefor sustainable interfaces and interactions.

Researchers from both academia and industry with an in-terest in research into self-powered sustainable interfacesand interactions are invited to submit a position paper. Thisposition paper should address one or more of the work-shop’s three grand challenges or suggest another (we en-courage diversity and provocation). All submissions willbe reviewed by the workshop organizers. Accompanyingdemos are encouraged, and will be allocated time in theworkshop agenda.

Participants will be selected on the basis of the quality oftheir position paper and on the basis of background andperspective; we are seeking to bring together a cross-disciplinary mix of perspectives. At least one author ofeach accepted paper must register for the workshop andfor one day of the conference itself. Participants will be in-vited to present a position statement at the workshop andwill actively engage in a discourse on the meaningful de-sign space for sustainability of self-powered interfaces andinteractions in HCI.

For further details or to submit a position paper, please seethe workshop website: http://cs.swansea.ac.uk/~SelfSustainableCHI/

AcknowledgmentsWe gratefully acknowledge the support of Engineering andPhysical Sciences Research Council grant EP/R032750/1(http://deliot.me/).

References[1] Alejandro Alanis, Trang Thai, Gerald Dejean, Ran

Gilad-Bachrach, and Dimitrios Lymberopoulos. 2014.3D gesture recognition through RF sensing. Tech.Rep. MSR-TR-2014-81 (2014).

[2] Ryota Arai, Seiichi Furukawa, Narumi Sato, andTakuma Yasuda. 2019. Organic energy-harvestingdevices achieving power conversion efficiencies over20% under ambient indoor lighting. Journal ofMaterials Chemistry A 7, 35 (2019), 20187–20192.

[3] Nivedita Arora and Gregory D. Abowd. 2018.ZEUSSS: Zero Energy Ubiquitous Sound SensingSurface Leveraging Triboelectric Nanogenerator andAnalog Backscatter Communication. In The 31stAnnual ACM Symposium on User Interface Softwareand Technology Adjunct Proceedings (UIST ’18Adjunct). 81–83.

[4] Nivedita Arora, Steven L. Zhang, Fereshteh Shahmiri,Diego Osorio, Yi-Cheng Wang, Mohit Gupta, ZhengjunWang, Thad Starner, Zhong Lin Wang, and Gregory D.Abowd. 2018. SATURN: A Thin and FlexibleSelf-powered Microphone Leveraging TriboelectricNanogenerator. Proc. ACM Interact. Mob. WearableUbiquitous Technol. 2, 2, Article 60 (July 2018), 28pages.

[5] Akash Badshah, Sidhant Gupta, Gabe Cohn, NicolasVillar, Steve Hodges, and Shwetak N Patel. 2011.Interactive generator: a self-powered haptic feedbackdevice. In Proceedings of the SIGCHI Conference onHuman Factors in Computing Systems. ACM,2051–2054.

[6] S P Beeby, M J Tudor, and NM White. 2006. Energyharvesting vibration sources for microsystems

applications. Measurement science and technology17, 12 (2006), R175.

[7] Daniel Bryant, Peter Greenwood, Joel Troughton,Maarten Wijdekop, Mathew Carnie, Matthew Davies,Konrad Wojciechowski, Henry J Snaith, TrystanWatson, and David Worsley. 2014. A transparentconductive adhesive laminate electrode forhigh-efficiency organic-inorganic lead halide perovskitesolar cells. Advanced Materials 26, 44 (2014),7499–7504.

[8] Tobias Grosse-Puppendahl, Steve Hodges, NicholasChen, John Helmes, Stuart Taylor, James Scott, JoshFromm, and David Sweeney. 2016. Exploring theDesign Space for Energy-Harvesting SituatedDisplays. In Proceedings of the 29th AnnualSymposium on User Interface Software andTechnology (UIST ’16). ACM, New York, NY, USA,41–48. DOI:http://dx.doi.org/10.1145/2984511.2984513

[9] Meng-Ju Hsieh, Jr-Ling Guo, Chin-Yuan Lu, Han-WeiHsieh, Rong-Hao Liang, and Bing-Yu Chen. 2019.RFTouchPads: Batteryless and Wireless ModularTouch Sensor Pads Based on RFID. In Proceedings ofthe 32Nd Annual ACM Symposium on User InterfaceSoftware and Technology (UIST ’19). ACM, New York,NY, USA, 999–1011.

[10] Walther Jensen, Ashley Colley, Jonna Häkkilä, CarlosPinheiro, and Markus Löchtefeld. 2019. TransPrint: AMethod for Fabricating Flexible Transparent Free-FormDisplays. Advances in Human-Computer Interaction2019 (2019).

[11] Keiko Katsuragawa, Ju Wang, Ziyang Shan, NingshanOuyang, Omid Abari, and Daniel Vogel. 2019. Tip-Tap:Battery-free Discrete 2D Fingertip Input. InProceedings of the 32Nd Annual ACM Symposium onUser Interface Software and Technology (UIST ’19).ACM, New York, NY, USA, 1045–1057. DOI:http://dx.doi.org/10.1145/3332165.3347907

[12] Miao Li, George Daniel, Bruce E Kahn, Liam H Ohara,Bernard DF Casse, Nathan Pretorius, Brent Krusor,Ping Mei, Gregory L Whiting, Chip Tonkin, and others.2018a. All Printed Large Area E-field Antenna UtilizingPrinted Organic Rectifying Diodes for RF EnergyHarvesting. In 2018 IEEE 18th InternationalConference on Nanotechnology (IEEE-NANO). IEEE,1–5.

[13] Yichen Li, Tianxing Li, Ruchir A. Patel, Xing-DongYang, and Xia Zhou. 2018b. Self-Powered GestureRecognition with Ambient Light. In Proceedings of the31st Annual ACM Symposium on User InterfaceSoftware and Technology (UIST ’18). ACM, New York,NY, USA, 595–608. DOI:http://dx.doi.org/10.1145/3242587.3242635

[14] Ayesha Sultana, Md Mehebub Alam, Tapas RanjanMiddya, and Dipankar Mandal. 2018. A pyroelectricgenerator as a self-powered temperature sensor forsustainable thermal energy harvesting from waste heat

and human body heat. Applied Energy 221 (2018),299–307.

[15] Joel Troughton, Daniel Bryant, Konrad Wojciechowski,Matthew J Carnie, Henry Snaith, David A Worsley, andTrystan M Watson. 2015. Highly efficient, flexible,indium-free perovskite solar cells employing metallicsubstrates. Journal of Materials Chemistry A 3, 17(2015), 9141–9145.

[16] Nicolas Villar and Steve Hodges. 2010. Thepeppermill: a human-powered user interface device. InProceedings of the fourth international conference onTangible, embedded, and embodied interaction. ACM,29–32.

[17] Anran Wang, Vikram Iyer, Vamsi Talla, Joshua RSmith, and Shyamnath Gollakota. 2017. FMBackscatter: Enabling Connected Cities and SmartFabrics. In 14th {USENIX} Symposium on NetworkedSystems Design and Implementation ({NSDI} 17).243–258.

[18] Yi Zhao, Joshua R Smith, and Alanson Sample. 2015.NFC-WISP: an open source software defined nearfield RFID sensing platform. In Adjunct Proceedings ofthe 2015 ACM International Joint Conference onPervasive and Ubiquitous Computing and Proceedingsof the 2015 ACM International Symposium onWearable Computers. ACM, 369–372.