HH QUALCOMM ‘hear’ & now qualcomm® snapdragon™ audio

1 ©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.

‘Hear’ & Now Qualcomm® Snapdragon™ Audio

Ravi Satyanarayanan, Director Product Management, Qualcomm Technologies, Inc. Tony Rodriguez, CTO, Digimarc Richard Kjerstadius, Director, DSP Software, Opalum

Qualcomm Snapdragon is a product of Qualcomm Technologies, Inc.


‘Hear’ & Now: Snapdragon Audio

User Experience

1 Ecosystem Enablement

2 Digimarc

Watermark Detection

3 Opalum

Speaker Protection

4


Audio user experience Enhancing and expanding the phone experience

Camcorder

Entertainment

Natural user interaction

Voice call

Music playback


Voice call. Hear. And be heard. Voice enhancement technology for clearer conversations

©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.

• Fluence & Fluence Pro noise cancellation technology echo cancellation and noise suppression

• Freedom to hold the phone how you want

− Landscape or portrait speakerphone

− Handset

• Hear your conversation & not the background sound

− Active Noise Cancellation to suppress noise around you

− Automatically amplify speech relative to background sound

• HD Voice in all calls

• Reduce noise on the incoming calls


Music playback Hear music as the musicians intended

6

• Hi-Fi audio at the same quality as studio masters

- High quality WCD codec

- End-to-end 24bit / 192kHz

• Listen all day with low power playback on the DSP

• Active Noise Cancellation to hear only what you want to hear



Home theatre on-the-go Enjoy the same surround sound available in home theaters wherever you are

7

• Virtual surround sound over headphones to bring the immersive experience with you

• Big booming sound

- Bigger bass, clearer vocals, louder sounds

• Collaborations with industry leaders: Dolby, DTS



Camcorder Capture what you want to hear

• Capture what you see with Sound Focus

- Suppress unwanted noise from outside of the field of view

• Audio zoom when the camera zooms in

• Surround Sound Record for the Full HD / 4K camcorder experience

• High quality audio, with high dynamic range



Voice natural user interface

• Always on, low power voice activation

- No need to press a button to interact with your device

- Launch applications and give commands quickly

• Activate & Unlock in one step

- No need to enter a PIN

• Hands free operation with Fluence

- Drive mode




User Experience

1 Ecosystem

Enablement

2 Digimarc

Watermark Detection

3 Opalum

Speaker Protection

4


Audio ecosystem

• Added on Snapdragon 810, 808

• Great for low latency, timing sensitive algorithms

• Speaker Protection by Opalum

Audio front-end Voice

Audio

3rd-party relationships for OEM differentiation

• End-to-end high-fidelity audio

– Native 24bit/192kHz framework

• Low power decode on Qualcomm® Hexagon™ DSP

• Snapdragon Voice Activation

• ISVs can enable similar solutions using the framework

Always on, low power

• DSP-accelerated voice codecs and ECNS for VoIP

• Hexagon Access Program support to add voice codecs and processing modules

Qualcomm Hexagon is a product of Qualcomm Technologies, Inc.


How others add value, and find success



User Experience


2 Digimarc

Watermark Detection

3 Opalum

Speaker Protection

4

Qualcomm® Snapdragon SenseTM Audio + Digimarc Discover

Enabling Low-Power Always-On Contextual Awareness

Tony Rodriguez

Qualcomm Snapdragon Sense is a product of Qualcomm Technologies, Inc.

— Slide 15 — ©2014 Digimarc Corporation

Digimarc

Vision and Mission • Enable mobile devices to see, hear, and understand by providing

digital identities for all media objects (Internet of Everything) Approach

• Lessons learned from the last decade of deployment (started w/DoJa & BrewTM):

• ‘Seeing & Hearing’ is a function of how fast, accurate and reliable the underlying object/media recognition algorithms are.

• Deterministic object recognition is significantly quicker, more accurate and reliable.

• Developers are best served when fast, reliable and accurate object recognition is available as a system service that minimizes bandwidth consumption and is always-on & low-power.

• 2013: Delivered Intuitive Computing Platform (ICP) to our retail, publishing and brand partners who have deployed apps to enable engagement with print, packaging, radio, etc.

BREW is a product of Qualcomm Technologies, Inc.


Digimarc Discover built on ICP: Enabling Shoppers Journey

• Digimarc® Discover app & SDK’s built on Intuitive Computing Platform.

• Enables object and media

recognition through all phases of Shopper’s Journey

• Discovery & transaction via mobile devices across all touch points of the Journey

• Digimarc® Barcode: Imperceptible

barcodes for CPG that can double throughput at front-of-store.


Examples: Publications, TV, Retail

Print to Digital Contextual Second-Screen Shopping


Deterministic Identification Powered by Digimarc ID’s

We can’t see or hear Digimarc ID’s, but mobile and wearable devices can

Digimarc ID

Original Digital Media

Digimarc ID Imperceptibly Embedded

Enabled Content

Print

Music, Movies or Images

Television and Radio


Comparison of App Experience in Noisy Environment

• Digimarc Audio ID’s

effective in hostile environments.

• Accuracy maintained, no missed matches (deterministic)

• Developers can assign multiple ID’s, creating richer, contextually aware experience.


Digimarc Audio ID: Details

• Signal Construction • Weak signal in 120 Hz to 8 kHz

range, communicated via host audio (channel)

• Extensible & key based payload protocols

• Embedding

• Psychoacoustic principles and empirical analysis used to hide watermark

• Ambient Detection

• Spectral Processing • Payload Recovery

Technique Frequency Domain WM

Description Frequency Domain Spread Spectrum

Characteristics Watermark signal added to audio spectrum (frequency magnitude)

Payload Extensible 24 bit

Audio required for detection

1-3 Seconds


Porting to the Qualcomm® HexagonTM

Goal: People don’t have “ear-lids” – sound is the most pervasive and rich source of context in the human experience, our sub-conscious is continually (even when asleep) categorizing audio and applying policies on when and how to engage our consciousness. Mobile & wearable devices should do the same!

• Phase 1: Re-factoring (pipeline architecture), Fixed Point, Memory Reduction • Phase 2: Hexagon Port of Detector • Phase 3: Low-Power Activator Implementation

• Result: Developers able to create always-on contextually aware applications that are fast, reliable and accurate.

Qualcomm Hexagon is a product of Qualcomm Technologies, Inc.


Phase 1: Re-factoring Detector (Pipeline Architecture)

• Mobile implementation optimized for OS & application processor

• Each block re-coded, reducing memory consumption and functional granularity to aide in porting

• 8MB -> 256K Runtime

FFT Intermediate Accumulation Magnitudes

Spectral Filter

Final Accumulation Magnitudes

Normalize Magnitudes

For every 512 samples @ 16 Khz For each 1.024 second

CRC SNR Gating Decode Select Decode Candidates

Version Detection

For each 1.024 second For each 1.024 second + Scaling

Audio

Digimarc ID

Spectral Processing

Payload Recovery


Phase 2: Hexagon Port

Detector Application

Hexagon & Audio Codec

Audio

• Application reliant on High Level Operating System (HLOS) to retrieve audio samples and deliver to Detector

• Audio sampled at 16Khz but processed only once a second.

• Difficult for Application Processor cores to enter low-power state.

Codec

Application Processor

Digimarc Qualcomm Technologies Hexagon SDK


Phase 2: Hexagon Port

Detector Stub Application


Skel

Fast RPC

Stub

Detector

• Hexagon SDK to Port detector C++ source.

• Significantly reduced load

on Application Processor cores

• Utilized library of optimized functions for Hexagon (FFT, etc.)

• QAIC handled complexity of FastRPC marshaling

Audio

Codec




Phase 3: Low-Power Activator



Audio

Skel

Fast RPC

Stub

Codec

“Activator” S

ervice & P

olicy Engine

• Worked with Qualcomm Technologies to

build Activator and route audio from Codec • Eliminated need for HLOS to manage audio.


Detector






Audio

Skel

Fast RPC

Stub

Detector Codec

“Activator” S

ervice & P

olicy Engine



• Policy Engine able to wake Application

Processor and invoke appropriate application when Digimarc Audio ID detected

• Network bandwidth consumed after object identification.


Policy Engine






Audio

Skel

Fast RPC

Stub

Detector Codec

“Activator” S

ervice & P

olicy Engine



• Policy Engine able to wake Application

Processor and invoke appropriate application when Digimarc Audio ID detected

• Network bandwidth consumed after object identification.

• Result: Application Processor & RF allowed

to enter low-power state.

• Developers able to create always-on contextually aware applications that leverage the ubiquity of audio in our everyday lives


Policy Engine



Conclusion : Snapdragon Sense Audio + Digimarc Audio ID’s

• People don’t have “ear-lids” – sound is the most pervasive and rich source of context in the human experience, our sub-conscious is continually (even when asleep) categorizing audio and applying policies on when and how to engage our consciousness.

• Mobile and wearable devices are now able to do the same, remain always aware and wake the

Application Processor as needed. • Able to maintain an always-on posture with no network dependencies – ramps power and

bandwidth consumption after identification has occurred (unique to Digimarc ID’s). • Enabled by Snapdragon Sense Audio & Digimarc Audio ID’s that provide accurate, reliable context. • Result: Accurate and application specific context can be incorporated in all sources of audio.

• Applications enabled for developers: Retail, Second-Screen, Indoor Location, Radio, Concerts, etc.

©2014 Digimarc Corporation


‘Hear’ & Now: Snapdragon Sense Audio

User Experience


2 Digimarc

Watermark Detection

3 Opalum

Speaker Protection

4

© 2014 Opalum AB

High-End audio from constrained devices

© 2014 Opalum AB

Agenda

• Trends in Mobile Audio • Opalum Overview and Tech • Collaboration with Qualcomm Technologies • What’s next?

© 2014 Opalum AB

Trends in Mobile Audio

• Maximizing loudness and bass – Power doesn’t kill a speaker – Pushing the limits on thinner form-factors

• Multi-channel audio goes mobile – Virtualization techniques for movies and games. – Parametric audio

• Reducing power consumption – “Always on” features demand DSP core offloading – Increasing level of integration into main chipset – Other ways of reducing power consumption?

© 2014 Opalum AB

Overview and Tech

© 2014 Opalum AB

© 2014 Opalum AB

© 2014 Opalum AB

© 2014 Opalum AB

© 2014 Opalum AB

© 2014 Opalum AB

Collaborating with Qualcomm Technologies

• Seamless integration between LabSuite and Snapdragon • Audio tuning lead-time drastically reduced • Working with the Hexagon tools • Dynamic loading enables really cool android app possibilities

© 2014 Opalum AB

What’s Next?

© 2014 Opalum AB

For more information on Qualcomm, visit us at: www.qualcomm.com & www.qualcomm.com/blog

©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm, Snapdragon, Hexagon, Brew and Fluence are trademarks of Qualcomm Incorporated, registered in the United States and other countries. Uplinq and Snapdragon Sense iaretrademarks of Qualcomm Incorporated. All Qualcomm Incorporated trademarks are used with permission. HY-FI is a trademark of Qualcomm Atheros, Inc., registered in the United States and other countries, used with permission. Other products and brand names may be trademarks or registered trademarks of their respective owners.of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.

Thank you FOLLOW US ON:

Devices & Hardware

HH QUALCOMM ‘hear’ & now qualcomm® snapdragon™ audio