Wearable Computing and Human Computer Interfaces

Jeffrey Funk

Division of Engineering and Technology Management

National University of Singapore

Human-Computer Interfaces and Wearable Computing

Thanks to Karthik Nandakumar for the first drafts of these slides in 2012For information on other technologies, see http://www.slideshare.net/Funk98/presentations

What do the Following Trends Mean for Human-Computer Interfaces?

Better ICs, MEMS, wireless transceiversMoore’s LawMore than Moore

Better camerasHigher resolution and sensitivity

Better displaysCheaper, flexible and more durableMore sensitive touchNew forms of touch

Better combinations of these interfacesBetter neural interfaces

A New Generation of Input Interfaces

Neural

Speech

Touch

GestureAugmented Reality

Wearable

That Can Change the Way We Live and Work

Not just consumer applications Assemblers can see drawings Construction workers can see through walls,

wires, and pipes Prospectors can see through the ground Architects can see entire 3D image of building Students can see 3D representation of anatomy,

materials, universe Similar systems could be useful for tourists,

shopping, soldiers, and artists

HCI

Human-Computer Interface (HCI)

• HCI is the technology that connects man and machine

• Robust HCIs are needed to enable ubiquitous computing

We primarily focus on input interfaces

Human Computer

Thoughts

ActionInput

InterfaceAction

Recognition

Task Execution

Understanding

Output Interface

Sensory Perception

Rendering

Traditional Input Interfacesare Disappearing..

Command Line Interfaces (CLI), i.e., keyboard

Batch Interfaces

Graphical User Interfaces

(GUI)

Even God is Interested!

Session Technology

1 Objectives and overview of course

2 How/when do new technologies become economically feasible?

3 Two types of improvements: 1) Creating materials that better exploit physical phenomena; 2) Geometrical scaling

4 Semiconductors, ICs, electronic systems

5 Sensors, MEMS and the Internet of Things

6 Bio-electronics, Health Care, DNA Sequencing

7 Lighting, Lasers, and Displays

8 Human-Computer Interfaces and Wearable Computing

9 Information Technology and Land Transportation

10 Nano-technology and Superconductivity

This is Eighth Session of MT5009

Outline

• Overview

• Speech Interfaces

• Touch Interfaces

• Gesture Interfaces

• Augmented Reality (various combinations)

• Wearable Computing

• Neural Interfaces

Performance of Input Interfaces

• Accuracy: Precision in recognizing the action

• Throughput: Information that can be processed per unit time

• Affordability: Inversely proportional to the cost

• Ease of use: Ease in learning to use the interface

• Sociability: Multi-person interactivity

• Mobility: Size and mass of device, power consumption

• User experience: Subjective perceptions of utility & efficiency

Comparison of Input Interfaces

Key Components of Input Interfaces Human-Computer Input Interfaces

Neural Interfaces

Natural UI

Speech

Micro phone

Neural electrodes/

sensors

GestureTouch

3D Camera

Touch sensor

Tracking & Recognition

Software

Materials/Nanotechnology

Signal Processing Hardware (Semicon-ductors)

Outline

• Overview







Speech Interfaces

Key Components

Microphone

Automated Speech Recognition (ASR) and Natural Language Understanding (NLU) Software

Possible methods of improvement are

• Increase Signal to Noise Ratio (SNR) from microphone

• Achieve human-level performance on ASR/NLU tasks

Key dimension that needs improvement is Accuracy

Video on Siri (iPhone 4S)

http://www.youtube.com/watch?gl=SG&hl=en-GB&v=L4D4kRbEdJw

Evolution of Microphone Technology

Electret Condenser Microphone (ECM)

Signal to Noise

Ratio (SNR): 55-58 dB

MEMS Digital Microphone

SNR: 61 dBFlatter frequency response

Smaller size (CMOS fabrication)Can be reflow soldered

Analog Devices, MEMS Microphone Technology, October 2010

MEMS Microphone Technology

Signal to Noise Ratio (SNR) is directly proportional to

• Area of back-plate• Air gap spacing• Applied bias voltage• Mechanical compliance factor (inversely proportional to

stiffness of the diaphragm)

Air gap spacing

Sound wave

Elko et al., Capacitive MEMS Microphones, Bell Labs Technical Journal, 2005

Can MEMS Microphones be Improved?

• Size of microphone depends on area of backplate & gap spacing

• Reducing size will also reduce the quality (SNR), unless it is compensated by other factors

• Wavelength of audible sound waves is > 17 mm, while size of a MEMS microphone is only a few mm; noise level in a microphone is already close to the thermal noise limit

• Dramatic reductions in size or increase in SNR of single MEMS microphones does not appear to be possible

S. Beus, “MEMS mic enables thinner phone designs”, 2005

Microphone Array

Microphone array can mitigate background noise & interference

Courtesy: Audience Inc.

Accuracy based on Microphone Array

1. LOUD project from MIT Computer Science and Artificial Intelligence Laboratory, 2005

2. Microphone Array project in MSR: Approach and Results, Microsoft Research, June 2004

Noise suppression algorithms can increase SNR by 18dB with just 4 microphones in an array1,2

Improvement in SNR Corresponding decrease in Word Error Rate

Automated Speech Recognition (ASR)

“Increase in vocabulary sizes needs exponential increase in computing power due to potential combinatorial explosions”

L. Rabiner, “Challenges in Speech Recognition”, NSF Symp. on Next Gen. ASR, 2003

Automated Speech Recognition (ASR)

“Increase in vocabulary sizes needs exponential increase in computing power due to potential combinatorial explosions”

L. Rabiner, “Challenges in Speech Recognition”, NSF Symp. on Next Gen. ASR, 2003

Only Acceptable in Some Niches

Wor

d er

ror

rate

ASR Accuracy in Text Dictation

• Stand-alone speech interfaces may be useful for tasks like dictation

• Speech as an important modality in multimodal user interfaces (e.g., Microsoft Kinect) may be the future

* http://blogs.msdn.com/b/sprague/archive/2004/10/22/246506.aspx

*

?

For Online TranslationsGoogle’s service translates documents with

inference models that were developed from analysis of a trillion words or 95 billion English sentences.

By 2010, its dataset covered more than 60 languages and could accept voice input in 14 languages and English is sometimes used as a bridge between two different languages when direct translations don’t exist

Siri

Outline

• Overview







Touch ScreensMany kindsBut most are

variations of eitherResistive Capacitive (iPhone)

Depend on new materials that are deposited on top of an LCD display

Processors interpret the datahttp://www.youtube.com/watch?

v=FyCE2h_yjxI&src_vid=5fOI-EQCOOQ&feature=iv&annotation_id=annotation_558874

What are the limitations with these touch screens?

Which technologies might contribute towards overcoming these limitations?

CHALLENGE: SENSITIVITY

Source for next 15 slides: Group presentation in Fall 2014

Challenges – Sensitivity

0 1 2 3 4 5 60.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

140.00%

160.00%

180.00%

200.00%

f(x) = 0.138313454391182 x^-1.08178288123264

Sensitivity Level vs Overlay Thickness

Ratiotrendline

Overlay Thickness (mm)

Percentage Change in Raw

Count

*Note: Microprocessor doesn’t recognize capacitance domain but rather it register the change in raw count

Source: FYP Industrial Collaboration (Fischer-Tech and NUS) Project Report 2013 – An Empirical Approach towards Capacitive Touch-Sensing in Functional Plastics

APPLE IPHONEModel Type Overlay Thickness

3GS, 4, 4S Gorilla Glass 1 1.0mm5, 5S Gorilla Glass 2 0.8mm

SAMSUNG GALAXYModel Type Overlay Thickness

S1, S2 Gorilla Glass 1 1.0mmS3 Gorilla Glass 2 0.8mmS4, S5, Note 3 Gorilla Glass 3 0.4mm

1.0mm

0.8mm

0.4mm

Thinner Glass Increases Sensitivity and Use of Smart Phones in Cold Countries

Source: [Spec sheet download for glass 1, 2 and 3] http://www.corninggorillaglass.com and http://www.corning.com/WorkArea/showcontent.aspx?id=63819

http://www.corninggorillaglass.com/

http://www.corninggorillaglass.com/

http://www.corning.com/WorkArea/showcontent.aspx?id=63819

http://www.corning.com/WorkArea/showcontent.aspx?id=63819

CHALLENGE: DURABILITY

Low Damage ResistanceLow Bending StrengthLow Critical Load Bearing

New Generations of Gorilla Glass: Higher Loads

Through IOX: ion-exchanged glass and new processhttps://www.youtube.com/watch?v=q4ZU7zUxdM8

Source: [Spec sheet download for glass 1, 2 and 3] http://www.corninggorillaglass.com

*Note: Critical Load is the min. amount of load before radial cracks start to form and propagate

4500

15000

7500

New Generations of Gorilla Glass Have Tighter Strength Distributions

(low probabilities have been eliminated)


Gorilla Glass 1 Gorilla Glass 2 Gorilla Glass 3

4 MPa 6.25 MPa 6.75 MPa

Greater Strength at all Thicknesses


65

75

85

Gorilla Glass 1

Gorilla Glass 2

Gorilla Glass 3

CHALLENGE: FLEXIBILITY AND CONFORMITY

Achieving Flexibility and Conformity

OLEDs are more flexible than are LCDsBut other parts of the display are not flexible

Transparent conductorExisting Glass

Indium-tin oxide is inflexible and must be replaced with new materialSilver Nano wires?Carbon nanotubes, graphene?

Glassneed thinner glass, which is becoming feasible

Carbon Nano Tubes are More Flexible than Indium-Tin Oxide

Source: http://iopscience.iop.org/1347-4065/53/5S1/05FD04/article

Thinner Glass Leads to Greater Flexibility

Source: www.corning.com/WorkArea/downloadasset.aspx?id=48957

Willow Glass

Thinner Glass Leads to Lower Bend Stress and Failure Probability

Source: www.corning.com/WorkArea/downloadasset.aspx?id=48957 and http://www.corning.com/WorkArea/showcontent.aspx?id=63819

CHALLENGE: LIGHT TRANSMISSION

Read and Input on Both Sides

GamesSurgical Discussions through Glass

LG announceda transparenttelevision (30%)

Do we need one?

Or is that anoxymoron?http://www.extremetech.com/computing/186241-lgs-flexible-and-transparent-oled-displays-are-the-beginning-of-the-e-paper-revolution

Need New Forms of Transparent Conductors

Source: http://iopscience.iop.org/1347-4065/53/5S1/05FD04/article , http://www.beilstein-journals.org/bjnano/single/articleFullText.htm?publicId=2190-4286-4-12

ITO is expensiveand inflexible

Can we use:

Silver Nanowires?

CNTs?

Graphene?

Are Silver Nano-Wires the Best?

At 100 Ohm per Area, Light Transmission of:

• ITO – 91%• Cambrios ClearOhm

(Ag Nanowire) – 98%

OTHER CHALLENGESExisting touch screens require one to look

carefully at screen while touching a specific place

Fingers can easily touch wrong placesTactus offers an overlay to existing

touch screens that facilitates proper location of finger“Bubbles” rise out of the display when fingers touch

the display thus helping fingers find the “right spot”These bubbles are formed using MEMS (micro-

electronic mechanical systems)Studies have found that faster and more accurate

typing are achieved with the Tactus overlay

How the Tactus System Works

Micro-channels are filled with fluid whose refractive index matches that of top polymer layer. Thus, transparency is even across surface. http://www.youtube.com/watch?v=wrSKbTzc4BI 0:40-1:30

Texture Touch DisplaysSensation of texture can provide more information for

usersThis can be done using changes in vibration with

small motors or transparent electrodes (Senseg) that provide

information about texture, etc. www.youtube.com/watch?v=FiCqlYKRlAA (from 0:30-2:00 minute mark)

Another one from Disney: www.washingtonpost.com/

blogs/the-switch/wp/2013/10/08/disney-invents-touchscreen-that-lets-you-feel-textures/

Early applications: 3D modeling or remote surgery can benefit from data on texture of materials or organs

Sensing Force of TouchNext generation Apple (this fall) will detect

how hard user is pressing on screenCan distinguish between light tap and deep

pressNew applications?

Piano-playing app?New types of games?

Applications for Touch ScreensSmart phones for purchasing?

http://www.youtube.com/watch?v=Gg3tmZrwbDs

Smart displays in laboratoriesAlso

Advertising displays at bus stops or MRT stations

Mall information displaysSelf checkout in storesInformation counter in storesFor example, Sony’s AtracTable is

being developed for these applications

Outline

• Overview







Components of Gesture Interfaces

Key Components

2D/3D Camera (image sensor)

Tracking, Recognition &

Gesture Understanding

Software

Key dimensions that need improvement are Accuracy, Throughput and Affordability

Working of a 2D Image Sensor

http://www.cameratechnica.com/2011/11/30/five-reasons-you-may-soon-be-shooting-at-iso-50000/

Image Sensor Characteristics• Spatial resolution: Number of pixels

• Temporal resolution: Frames per second

• Image sensor area: Size of the image sensor – area is proportional to no. of pixels and pixel size

• Photometric exposure: light gathering ability of the sensor – depends on the properties of the lens

• “Light available per pixel”: No. of photons incident on a pixel – proportional to photometric exposure and pixel size

• Pixel sensitivity: is proportional to “light available per pixel”, quantum efficiency of photodiodes, and optical efficiency

• Dark Limit & Dynamic range: Ability to detect dim details & bright details in one image – depends on pixel sensitivity and capacity

Improvements in Image SensorsAccuracy

• Higher spatial resolution (no. of pixels)

• Robustness to lighting changes (high pixel sensitivity, low dark limit, and high dynamic range)

• More accurate depth sensing (lower depth error)

Throughput

• Higher frame rate

Affordability

• Smaller pixel size reduces price per pixel

Improvements in Spatial Resolution

T. Suzuki, “Challenges of Image-Sensor Development”, ISSCC, 2010

Number of pixels (resolution) has increased, but image sensor size has not increased because of reduction in pixel size

Year

Pixel Size vs. Sensitivity Tradeoff

CMOS-based image sensors are also expected to follow Moore’s Law in size and cost scaling


As pixel size decreases, “light available per pixel” will become less, so sensitivity decreases

Back illuminated CMOS technology provides better trade-off between pixel size and sensitivity than traditional charge coupled device (CCD)-based image sensorsL

igh

t av

aila

ble

per

pix

el (

arb

. un

it)

Camera Technology Improvements


Reducing pixel-size (green square) and improving sensitivity (Yellow circle ) miniaturized cameras without reducing quality

Image Sensors vs. Human Eye

Number of frames per second

Spati

al re

solu

tion (

cycl

es

per

degre

e)

Human Eye

Better than Human Eye

Modern cameras are close to human eye in terms of resolution

Skorka & Joseph, “Toward a digital camera to rival the human eye”, J of Electronic Imaging, 2011

Image Sensors vs. Human Eye

Dynamic Range (dB)

Dark

Lim

it (

cd/s

q.m

)

Human Eye

Better than Human Eye

But improvement can be achieved in terms of sensitivity

Pixel sensitivity determines the dark limit and dynamic range of an image sensor

Further Improvements in Sensitivity• Increase quantum efficiency through the design of better

photosensitive materials using nanotechnology

• (e.g., Single Carrier Modulation Photo Detector)

• Increase optical efficiency through a “vertically integrated” layered arrangement as in the human retina

http://www.future-fab.com/documents.asp?d_ID=4926

Cost per pixel of Camera Chips has fallen dramatically

3D Depth Sensing Technologies

* R. Lange, “3D Time-of-flight distance measurement with custom solid-state image sensors in CMOS/CCD-technology”, PhD Thesis, 2000

Comparison of 3D Sensing Technologies

Application Range (m)

Depth

Reso

luti

on (

m)

Usable Range for Gesture Interfaces

Microsoft Kinect

• Cost-effective 3D image sensors are now becoming available (e.g., Microsoft Kinect ~ 150 USD)

• Such cameras will further improve the accuracy of gesture UIs

3D Depth Sensing: Interferometry

• Most accurate depth sensing technology (accuracy depends only the wavelength of light)

• Low miniaturization potential and very limited range

3D Depth Sensing: Time of Flight

• Time of flight (ToF) cameras requires processors with high clock speed (3 GHz speed can provide only 4.5 cm depth resolution)

• High miniaturization potential and large range

• Improvements in CMOS technology are likely to very beneficial

3D Depth Sensing: Triangulation

Passive TriangulationActive Triangulation

• Limited range

• Low miniaturization potential

• Depth accuracy decreases with square of the distance

Leap has Generated Excitement

http://www.youtube.com/watch?v=_d6KuiuteIA

Leap uses multiple camera sensors to recognize gesturesWorkspace is about 3 cubic meters. Better sensors will enable larger work spaces. $70 control system that plugs into any computer. MIT’s Technology Review calls Leap, “The most important new technology since the smart phone…”

How about Microsoft’s Kinect? http://www.youtube.com/watch?v=o4U1pzVf9hY

Or wearable ring (each position represents different number)? http://www.youtube.com/watch?v=Gx3zWHS8amA

Replace Cameras with MEMS-based wrist band from Thalmic Labs, called MYO

Gestures are recognized before movement

Muscle activityis monitored with9-axis inertial measurement unit (MEMS)

Other Applications for 3D CamerasUse cameras to

track eye movementsMonitor drivers or

other operators of machines

Help paralyzed people use computers

As cost of cameras fallEye tracking might

become user interface for non-paralyzed

Eye tracking can also be used with Google Glasses (see below)

Source: http://www.economist.com/news/technology-quarterly/21567195-computer-interfaces-ability-determine-location-persons-gaze

3D Cameras and Virtual Reality Can Improve Video Conferencing

Might this finally be the technology that reduces need for travel?

Intel introduced RealSense technology, which gives the cameras in laptops the ability to see and understand depthNotebooks this yearTablets and phones next year?

Eliminates the background, enables better communication with hands

Virtual reality makes the video conferencing even betterVR discussed in 7th sessionAltspaceVR is designed for video conferencing

Outline

• Overview







Types of Augmented Reality

Glasses

Phones

How Different from Virtual Reality?

Augmented Reality Supports our understanding of the real world while virtual reality immerses us in a new type of world

Recently Oculus VR was acquired by FacebookImprovements in 3D displays, accelerometers,

gyroscopes, compasses (MEMS) and graphic processors are enabling rapid improvements in VR

How might Oculus VR help Facebook?Will users want further immersion in social

networking……

Returning to Augmented Reality

What do you see through the glassesor Lens?

Handheld devices may be sufficient, particularly if the images are easily integrated with your surroundings

What about a or a virtual oneSupermarket? at a subway station?

http://www.youtube.com/watch?v=yKNSOwLcrkE

What about superimposing the images on a car’s windshield?

Outline

• Overview




• Augmented Reality (various combinations)• Wearable Computing Main Source: A Rajaraman, B Madhumita,

Mayank Tewari, D Nelson, S K Rao, Wearable Technology Design, Spring 2015


“Moore’s Law” and “More than Moore” are Making Wearable Computing Economically Feasible

But where will the devices be attached?

Wrist/Arm?

Leg?

Head?

Body?

Where will the Devices be Attached?

Different data can be collected from different parts of the body

Data can be viewed better on some parts of the body

What does this mean for the products and services that will succeed?

Services and software will be an important part of wearable computingNot just hardware!

Lets look at current products and where they are attached

Future projects should probe deeper, providing a better idea about the best places to attach devices

Skull

Google’s Project Glass

Image and information are displayed on the glassesUsers choose which information to display on the glassesChoice controlled by voice, remote control, or maybe

thoughts in futurehttp://www.youtube.com/watch?v=9c6W4CCU9M4 Improvements in ICs, displays, other components are

leading to better performance and cost of glassesBut maybe they won’t help men find a girlfriend

http://www.youtube.com/watch?v=8UjcqCx1BvgUnless you can access data without talking (e.g.,

tilting one’s head, touching the device, or blinking your eyes)

Google Glass in Factories (1)Replace stacks of wiring

instructions with Google Glass display

Test programs are being conducted at Boeing, Daimler, United Parcel, and others

Boeing workers use them to assemble wire-harnesses

When assembler reads out loud coding on wire, correct hole on the electronic version lights up and flashes, providing easy to follow guide

Error rates and assembly time have fallen

http://www.wsj.com/articles/smart-glasses-get-new-look-on-factory-floor-1433301177

Google Glass in Factories (2)Deutsche Post is working with DHL and Ricoh

Warehouse workers process orders ofr parts and equipment by scanning bar codes on cartons with their glasses

Eliminates need for hand-held bar-code scanners and paper invoices

Instructions are relayed through glassesReplace stacks of wiring instructions with Google Glass

displayDaimler uses them on assembly lines

Glasses provide checklists so workers don’t have to memorize or hold the paper lists

Discovered defects are immediately reported through voice-recorded report with photos

Safety and security are issueshttp://www.wsj.com/articles/smart-glasses-get-new-look-on-factory-floor-1433301177

Other Examples

Microsoft’s HoloLensCan be used to play games Or to interact with the world https://

www.youtube.com/watch?v=Qm2gnnyyvEg

Other exampleshttp://www.youtube.com/watch?v=t-m6

YL64lkUhttps://www.youtube.com/watch?

v=fJI8tNG1rbQ

https://www.youtube.com/watch?v=Qm2gnnyyvEg



http://www.youtube.com/watch?v=t-m6YL64lkU

http://www.youtube.com/watch?v=t-m6YL64lkU

Whole ARM

WHOLE LEG

Can this Analysis be taken Further?What are the costs/prices?

How much will they come done and how fast?What are the benefits and where are they the

largest?Can we quantify the potential benefits from

specific wearables? Do specific sensors (earlier tonight) work with

specific wearables? Which sensors are getting cheaper and better?Can we identify where the largest benefits might

be?Can we use this information to design better

wearable devices

Perhaps More ImportantlyNew forms of software are needed for

wearable computingOperating systemsCloud computingOpen sourceWhat startups will do this and be valued at more

than $1 BillionNew forms of software services are needed

Big dataConsumer InternetWhat startups will do this and be valued at more

than $1 Billion

One example of New Software

Will new forms of passwords be needed for wearable equipment

Fingerprints and iris scans require special equipment, thus increasing the weight of wearable equipment

Ballisocardography is study of body’s movements in response to the activity of the heartBody shifts slightly as heart beatsUnique to individualsShifting Identity, Economist, June 20, 2015, p.

76

Outline

• Overview







Neural Interfaces

Key Component

Brain scanning device

Key dimensions that need improvement are Accuracy, Throughput and Affordability

Required improvements in brain scanning technology

Accuracy & Throughput – Higher spatial and temporal resolution

Affordability – Size and better materials

Current State of ArtBest systems enable a person to control a

robot or cursor or type one letter a minute with their “mind”For paralyzed, very expensivehttp://www.youtube.com/watch?v=C7H_M8-dBHc

(0-1 minute)http://www.youtube.com/watch?

v=cDiWFcA0gaw&playnext=1&list=PL7FD931F8953A0F87&feature=results_main (0-2:45 minute)

Accessory for your iPhone$99 device measures your

brain waves. App displays data on phone

http://singularityhub.com/2011/01/07/iphone-accessory-from-xwave-channels-your-brain-waves-to-the-iphone/

SPECT

EEG

1936 1950 1972 19751968

CT Scan

1983

MEG

1991

fMRINIRS

1973

MRI PET

US$2.9M

US$1M-1.5M

US$250K US$2.4M US$0.5M-3M

US$180K- 250K>US$30K

Non-Invasive Brain Scanning

Electro Encephalo Graphy

Magneto Encephalo Graphy

Near-Infra Red Spectroscopy

functional Magnetic Resonance Imaging

EEG & MEG directly measure neuronal activity, NIRS & fMRI measure blood activity

What do EEG & MEG Measure?

Localization of EEG vs. MEGEEGMEG

Where we Are for Resolution (1)

Ideally, a non-invasive technology with high spatial resolution and high temporal resolution is required

Additionally, the technology must be affordable and portable in order to be useful in HCI applications

Gerven, M. v., et al., “The Brain-Computer Interface Cycle”, J. Neural Eng, 2009

Non-invasive

Neuron can fire ~0.1mm (spatial) & ~10 ms (temporal)

Invasive

Where we are, and where we want to be (2)

(in 60 years?)

http://singularityhub.com/2011/01/07/iphone-accessory-from-xwave-channels-your-brain-waves-to-the-iphone/

Invasivetechniques

Spatial Resolution Improvement

While spatial resolution is important for accuracy, high temporal resolution is also critical for user interfaces

R. Kurzweil, “The Singularity is Near”, 2005

fMRI

EEG Challenges

Key limitation: Poor spatial resolution

Increasing number of EEG electrodes may provide limited improvement in spatial resolution and higher SNR

J. Malmivuo, “Comparison of the Properties of EEG and MEG”, Intl J of Bioelectromagnetism , 6 (1), 2004

MEG Challenges

Baranga, A. B.-A. (2010). "Brain's Magnetic Field: a Narrow Window to Brain's Activity". Electromagnetic field and the human body workshop, (pp. 12).

fT

MEG: Improvements in Millimeter-scale Atomic Magnetometer

Target: ~100fT and <100Hz

J. Kitching, et al., “Uncooled, Millimeter-Scale Atomic Magnetometers”, IEEE Sensors 2009 Conference (pp. 1844-1846)

A: 2004B: 2007C: 2007D: 2009

Invasive Techniques

Chips are implanted intoa person’s brain

Source: Stevensen I, Kording K 2011, How Advances in Neural Recording Affect Data Analysis,Nature Neuroscience 14 (2): 139-142

Improvements in Invasive Techniques

Source: Stevensen I, Kording K 2011, How Advances in Neural Recording Affect Data Analysis,Nature Neuroscience 14 (2): 139-142

But the improvements are probably not occurring fast enough

At current rates of improvement, in 220 years we will be able to simultaneously record all 100 billion neurons

On the other hand, maybe we don’t need to record all of

them simultaneously because we will find ways to interpret the data

Or maybe we will find better non-invasive techniques

Conclusions

• Rapid improvements are occurring in HCI

• Most of these improvements are being driven by improvements in ICs or MEMS (reductions in scale)• Speech recognition (microphones, processors)

• Gesture interfaces (cameras)• Neural interfaces (electrodes)

• Other improvements are being driven by creation of new materials• Touch screens

Pictorial Representation of the Drivers

Microphone

Multitouch Sensors

3D camera & motion sensors

Haptic Devices

Neural Sensors

Speaking

Touch Events/

Gestures

Body/head movements

Free hand gestures

Facial expression

Eye gaze

Hand pressure

Brain activity

Speech Recognition &

Language Understanding

Touch/Gesture Recognition & Understanding

Neural Signal Analysis &

Understanding

User Interface System Design & Integration

Multimodal Fusion

Human Factors

Engineering

Context Aware

Services

Conclusions

• Ubiquitous computing requires new HCI paradigms

• Natural and Neural Interfaces are the future of human-computer input interfaces

• Touch interfaces have already diffused into the mainstream; speech and gesture interfaces are becoming more accurate and affordable

• Neural interfaces requires development of more accurate, cheap, and portable sensors

• Numerous entrepreneurial opportunities are available both in technology development & customization

One-Page Write-ups4 one-page write-ups on topics related to technologies covered

in sessions 4 through 10 (20% of your grade)If you had two months to investigate one of these topics, what would

you do? What types of data would you gather and how would you gather the

data? Who might you interview? Topics are listed at end of these slides, of each session, and in

assessment section of IVLE

2 one-page write-ups on topics related to technologies covered in group presentations (10% of your grade)Topic should be main points of presentationpropose a different method of analysis than the group did

It should not take you longer than 2 hours to do each assignment

Be Careful!The one-page write-ups on topics (related to technologies covered

in sessions 4 through 10) are different than the 2 one-page write-ups on (group presentations)They require very different answers

Even for one-page write-ups on topics related to technologies covered in sessions 4 through 10Each question is different because cost and performance depends on

different things for each technologyEach question is also different because amount of information is different

If technology has been commercialized, there is more info about technology – costs, performance, benefits, needs

If it has not been commercialized, there is less informationSome technologies depend more on improvements in components than do othersThus, the answers will be different

Session 8 Topics for Write-upsGoogle glass How would you assess the costs and benefits of

google glass and when it will become widely used in Singapore (>10% of Internet users)?

Gesture interface: How would you assess the costs and benefits of gesture interfaces and when they will become widely used in Singapore (>10% of internet users)?

Health data recorded with wrist device: How would you assess the costs and benefits of recording health data with a wrist device and when it will become widely done in Singapore (>10% of Internet users)?

Augmented reality with cameras and phones: How would you assess the costs and benefits of augmented reality with cameras and phones and when it will become widely used in Singapore (>10% of Internet users)?

Business

Wearable Computing and Human Computer Interfaces