Upload
360conferences
View
4.128
Download
0
Embed Size (px)
DESCRIPTION
code coverage measurement tools (Flexcover has gotten a lot of attention this year)visual editor frameworks (I'm just about to release a new open source project called Moccasin that distills a lot of Noteflight's editing infrastructure into a general framework)
Citation preview
Dynamic Audio Synthesis
on the Flash/Flex Platform
Joe Berkovitz
Overview
• Bit o’ background
• Flash 9 sound capabilities
• The Flash 10 Sound API
• StandingWave audio synthesis library
• New avenues for Flash audio and music
About me
Samples ‘n’ frames ‘n’ acoustics
Player 9 Sound API
• Sound objects can be loaded from URLs...
• ...or embedded as assets
• Load ‘em, play ‘em, change volume or stereo pan – that’s about it!
• Unpredictable latency, synchronization
Player 9 Audio// loading a sound over the networkvar s:Sound = new Sound();s.addEventListener(Event.COMPLETE, handleComplete);s.load(new URLRequest(“TheBomb.mp3”);
function handleComplete(e:Event) { Sound(event.target).play();}
Player 9 Audio// loading a sound from a library asset[Embed(source=”TheBomb.mp3”)]private static var MySoundAsset:Class;
function playSound():void{ var s:Sound = new MySoundAsset(); var sc:SoundChannel = s.play(); sc.soundTransform = new SoundTransform(0.5 /*volume*/); // set other transform properties... // examine playback position trace(sc.position);}
What we want to do...
• Dynamically compute sounds from scratch
• Process samples on the fly to shift pitch, add effects, equalization...
• Play more than 32 sounds at the same time
• Compute continuous sound output
FP10 to the rescue
• Synchronization and continuity: SAMPLES_CALLBACK event
• Access to codecs: Sound.extract()
• The Vector data type
• Adobe Pixel Bender runtime
• Alchemy
Dynamic Audio Outputvar s:Sound = new Sound();s.addEventListener(SampleDataEvent.SAMPLE_DATA, handleSampleData);var sc:SoundChannel = s.play();
function handleSampleData(e:SampleDataEvent) { for (i = 0; i < /*number of frames*/ ; i++) { e.data.writeFloat(/* left channel */); e.data.writeFloat(/* right channel */); }}
Codec Access[Embed(source=”TheBomb.mp3”)]private static var MySoundAsset:Class;
function getSoundSamples(numFrames:uint):void{ var s:Sound = new MySoundAsset(); var bytes:ByteArray = new ByteArray(); s.extract(bytes, numFrames, _position);
bytes.position = 0; for (var i:Number = 0; i < numFrames; i++) { ... = bytes.readFloat(); // Left channel ... = bytes.readFloat(); // Right channel }}
Some things to know
• Between 2048 and 8192 samples per event
• The fewer samples you provide, the lower the latency (time before a sample is heard)
• Initial latency > steady-state latency
• More samples/callback = more stability
• Don’t take too long to generate samples!
Basic Audio Demo
Going beyond theFP10 sound API
• Goal: to work at a higher level than samples
• High level concepts: Sources, Filters
• Play back timed, overlapping sequences of sonic or musical events: Performances
• Create a modular, extensible system
The StandingWave Project
• Open source AS3 library
• Outgrowth of Noteflight
• Requires Player 10
• Enables new class of audio applications
Audio Processing Concepts
Sample
Descriptor: 44.1 kHz Stereo
left[0]
left[1]
left[2]
left[3]
right[0]
right[1]
right[2]
right[3]
... ...
SampleSample
Audio Processing Concepts
AudioPlayerSource
raw samples,algorithms...
Samples
Audio Processing Concepts
AudioPlayerFilterSource
transformations
SampleSampleSamplesSampleSampleSamples
Audio Processing Concepts
Filter
AudioPlayer
Source
FilterSource
FilterSource
Perf
orm
ance
starts at 0.0 sec
starts at 0.2 sec
starts at 0.7 sec
(...continues...)
timed sequenceof sounds
Audio Processing Concepts
Filter
AudioPlayer
Source
FilterSource
FilterSource
Perf
orm
ance
starts at 0.0 sec
starts at 0.2 sec
starts at 0.7 sec
...
AudioPerformer
SampleSampleSamples
SampleSampleSamples
Audio Processing Concepts
Filter AudioPlayer
Source
FilterSource
FilterSource
Perf
orm
ance
starts at 0.0 sec
starts at 0.2 sec
starts at 0.7 sec
(...continues...)
FilterAudioPerformer
IAudioSourcepublic interface IAudioSource{ function get descriptor():AudioDescriptor; function get frameCount():Number;
function get position():Number; function resetPosition():void;
function getSample(numFrames:Number):Sample;
function clone():IAudioSource;} implementations: Sample, SineSource, SoundSource...
IAudioFilterpublic interface IAudioFilter extends IAudioSource{ function get source():IAudioSource; function set source(s:IAudioSource):void;} implementations: EnvelopeFilter, EchoFilter, BandpassFilter...
IPerformancepublic interface IPerformance{ function getElementsInRange (start:Number, end:Number) :Vector.<PerformanceElement>;} implementations: ListPerformance...
Noteflight: A Music Notation Editor
in Flash
What’s missing?
• Direct microphone input
• More codec libraries
• Fully functional and supported audio processing in PixelBender/Alchemy
Where to get the StandingWave library
http://code.google.com/p/standingwave
Places and Projects
http://joeberkovitz.com
http://www.hobnox.com
http://labs.adobe.com/wiki/index.php/Alchemy:Libraries
http://www.getmicrophone.com/
http://void.andre-michelle.com