audio

Audio Spatialization Effects in the BrowserJanessa Det - WEBHACKER @TWITTERNYC

experiential

summary

nodeS2noTE& PANNING

START & resourcesTHE WEB AUDIO API

APPLICATIONSMOTIVATION

sound&space CONVOLUTION DSP COMPONENTS

SPATIALIZATION experiential

audio

primer SIMULATING

ENVIRONMENTS

whoami.experientialaudio

janessa det (@jandet)

@twitternycSoftware engineer - webDiscovery Front-End

thefuture.fmlead front-end devDJ Defined Radio

columbia universityM.S. Computer scienceGraphics & Vision > UX/HCI

summary

sound&space.sound in the digital age

experientialaudiojanessa det (@jandet)

time vs. frequency

digitalization

browsers are 2d

<audio></audio>

soundhas become

flatsummary

sound&space.sound defines a space

experientialaudiojanessa det (@jandet)

summary

Sound interacts with surrounding space

3d space

environment

Environments perceived through sound

human perception

One of the key ways weperceive our surroundings

Janet CardiffThe Forty Part Motet

[1]

primer.alas! the web audio api

experientialaudiojanessa det (@jandet)

compatibilityfeatures

• audio signals processing for your browser

• easy to create, process, and control audio within web applications

• complements webgl

• chrome 10.0-24.0

• safari 6.0

• safari ios 6.0

• firefox NOOOOPE (BUT PLANNED)• AUDIO DATA API

summary

audio context

primer.audio routing graph

experientialaudiojanessa det (@jandet)

summary

audioDESTINATION

gain(volume)

audiosource

convolverfilter(s)audio

source

if  ('AudioContext'  in  window)  {    context  =  new  AudioContext();}  else  if  ('webkitAudioContext'  in  window)  {    context  =  new  webkitAudioContext();}  else  {    throw  new  Error('AudioContext  not  supported!');}

nodes2note.types of audio nodes

experientialaudiojanessa det (@jandet)

• raw audio data into the system • final node to connect to

• outputs to lineout• speakers/headphones

AUDIO SOURCE AUDIO DESTINATION

var  request  =  new  XMLHttpRequest();        request.open("GET",  options.path,  true);        request.responseType  =  "arraybuffer";

request.onload  =  function()  {createSoundSource(request.response);

};request.send();

soundSource.connect(context.destination);

function  createSoundSource(audioData)  {soundSource  =  context.createBufferSource();soundSource.buffer  =  

context.createBuffer(audioData,  false);soundSource.loop  =  true;

};summary

nodes2note.types of audio nodes

experientialaudiojanessa det (@jandet)

• apply gain to change volume • 0: lowpass• 1: highpass• 2: bandpass• 3: lowshelf

• 4: highshelf• 5: peaking• 6: notch• 7: allpass

filterNode  =  context.createBiquadFilter();filterNode.type  =  0;filterNode.frequency.value  =  220;

volumeNode  =  context.createGainNode();volumeNode.gain.value  =  0.5;

summary

gain (volume) filters

types of audio nodes

experientialaudiojanessa det (@jandet)

summary

• delay audio for specified duration

nodes2note.

delay oscillator

delayNode  =  context.createDelayNode();delayNode.delayTime.value  =  5;

• 0: sine• 1: square• 2: sawtooth

• 3: triangle• 4: custom

oscillatorNode  =  context.createOscillatorNode();oscillatorNode.type  =  0;

types of audio nodes

experientialaudiojanessa det (@jandet)

summary

• scriptprocessornode

• synthesize and process audio directly with javascript

• custom DSP effects

• exposing internals of dsp to web applications

nodes2note.

fancypants jsJS

http://www.w3.org/TR/webaudio/

audio context

audio routing graph

experientialaudiojanessa det (@jandet)

summary

audioDESTINATION

gain(volume)filter

audiosource

nodes2note.

//  initialize  nodes...

sourceNode.connect(filterNode);

filterNode.connect(volumeNode);

volumeNode.connect(context.destination);

spatialization.INTRODUCTION TO panning

experientialaudiojanessa det (@jandet)

panning algorithm

• distance effects• sounds softer farther away

• orientation• what direction are you facing

• channels• adjust volume to l & R outputs

• mimick human perception of space[1]

summary

USING PANNING nodes

experientialaudiojanessa det (@jandet)

summary

• connect to an audio source

• makes sound source positional

• sound cones• inner angle• outer angle• outer gain

• note: velocity!

• one listener per context

• position (x, y, z)

• orientation• front vector• up vector

spatialization.

panning node listener: )

pannerNode  =  context.createPanner();pannerNode.setPosition(x,  y,  z);pannerNode.coneInnerAngle  =  30;pannerNode.coneOuterAngle  =  40;pannerNode.coneOuterGain  =  0.5;

context.listener.setPosition(x,  y,  z);var  front  =  getFrontVector(angle);var  up  =  [0,  0,  1];context.listener.setOrientation(

front.x,  front.y,  front.z,  up.x,  up.y,  up.z);

convolution.INTRODUCTION TO convolution

experientialaudiojanessa det (@jandet)

what is convolution?

h(t)x(t) y(t) = h(t) x(t)

• “cross-multiplying” 2 signals

• LTI system theory

• impulse response

summary

USING convolver nodes

experientialaudiojanessa det (@jandet)

summary

• set buffer as impulse response

• connect sound source to convolver node

• convolves input through impulse response

• emulate sound in any environment

convolution.

convolver nodevar  request  =  new  XMLHttpRequest();        request.open("GET",  "sounds/ir/church.m4a",  true);        request.responseType  =  "arraybuffer";      request.onload  =  function  ()  {

createConvolverNode(request.response);}request.send();

function  createConvolverNode(audioData)  {convolverNode  =  context.createConvolver();convolverNode.buffer  

=  context.createBuffer(audioData,  false);}

position orientationaudio context

audio routing graph

experientialaudiojanessa det (@jandet)

summary

audioDESTINATION

convolution.

//  initialize  nodes...

sourceNode.connect(convolverNode);

convolverNode.connect(pannerNode);

pannerNode.connect(context.destination);

pannerconvolveraudio

source

demotime.experientialaudio

janessa det (@jandet)

interaction controls

START sound&space nodeS2noTE SPATIALIZATION CONVOLUTION APPLICATIONSprimer MOTIVATION THE WEB AUDIO API DSP COMPONENTS & PANNING SIMULATING

ENVIRONMENTS& resourcesexperiential

audio

summary

• player• listener• panning

sound

• sound• panning

sound

• panning sound• panner node• sound node

classes

• wasd - movement• panning sound

• q/e - rotate ccw/cw

• x - make tweeting sound

• expansions to come

panner-demo.jit.su

applications.experientialaudio

janessa det (@jandet)

gamingenriched

uxmobile web

(ios6)augmented

reality

summary

experientialaudiojanessa det (@jandet)

summary

future workareas to explore

• VISUAL INTERFACE & TOOLS TO BUILD AUDIO ROUTING GRAPHS

• INTERFACING WITH WEBGLexplore full 3d orientation & positioning

• INVESTIGATION OF AUDIO DATA APIcloser to the actual dsp implementations

• WEB AUDIO API ON IOS6 SAFARIdifferent ui, applications, and event handling

resources.

• rtfmhttp://www.w3.org/TR/webaudio/

• safari developer libraryhttp://bit.ly/RcVHxY

• DAFX (ref [1])

• JOHN WILEY & SONS• MAR 8, 2011

• impulse responseshttp://fokkie.home.xs4all.nl/IR.htm

SUMMARYPRESENTATION OVERVIEW

GOBACKORCHOOSEAsection

applications& resources

spatialization& Panning

simulating environmentsconvolution

nodes2notedsp components

START experiential audio

sound&space motivation

the web audio apiprimer