Hard real-time constraints in the Digital Music...

Hard real-time constraints in the Digital MusicIndustry

Leonardo Gabrielli

Universita Politecnica delle Marche, Ancona, Italy

June, 26th, 2014 - RTS-LIKE - Pisa

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 1

3MediaLabs - what

3MediaLabs: Multimedia Information Processing

Made of two research sub-groups: A3LAB and SEMEDIA.LEADER: Prof. Francesco Piazza

A3LAB: DSP Algorithms and Adaptivesystems for Audio applications

Data Processing Approach

People: 2 Assistant Professors, 5PostDocs, 4 Phd Students

SEMEDIA: Semantic Web and Multime-dia

MetaData Processing Approach

People: 3 PostDocs

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 2

3MediaLabs - A3LAB

Research Topics

Computational Intelligence: Neural Networks, Extreme LearningMachine, Blind Signal Processing.

Speech and Music Processing : Speech Enhancement, Speech/SpeakerRecognition, Acoustic Echo and Feedback Cancellation,Speech-interfaced Systems for HMI, Music Signal Analysis-Synthesis.

Immersive Audio Rendering : Digital Equalization, Wave-field Synthesisand Analysis, Loudspeakers array, Digital effects, 3D Audio.

In-car Audio: Advanced Car Information System, Professional Audio.

Projects

Privately and publicly (EU, National, Regional) funded

More than 1.5 Million Euro in the last 5 years

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 3

Collaborations

Companies: Texas Instruments, Faital Spa, Leaff Engineering,VideoWorks, CRN Yacht, FBT, Radvision, and so on so forth.

Universities: Technical University of Munich, Aalto University, Universityof Illinois at Chicago, South China University of Technology, ImperialCollege (UK), Chinese Academy of Sciences and others.

Member of the Texas Instrument European University Program

Related Companies

Leaff Engineering : former Spin-off operating in the Digital Audio field.

Sensible Logic: former Spin-off operating in the Semantic Web andDigital Libraries field.

DowSee: brand-new Spin-off operating in the Energy Management andSmart Grid fields.

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 4

The New Lab

Big acoustic semi-anechoic chamber (11m x 9m x 6m, ISO3744, ISO3745, Ft100Hz)

Media-lab, Texas Instrument DSP lab, and more

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 5

The New Lab

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 6

Speaker’s Bio

LEONARDO GABRIELLI

PhD candidate at UNIVPM:Physics modeling of ElectroacousticMusical InstrumentsMusic Technology: wirelessconnection and embedded platforms

Co-founder and CTO of DowSee srl:

Spinoff company of UNIVPMWireless Sensor Networks for theWater GridComputational Intelligence forDomestic Gas, Water andElectricity

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 7

Physically Informed Sound Synthesis

Analysis and Synthesis of Stringed Instruments

Virtual Acoustic Feedback: digital emulation of feedbacks betweensound sources and capture devices

e.g. guitar - loudspeaker feedbackcan be used to augment any kind of pitched instrument

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 8

WeMUST

A tool for live music at home / studioslow-latency, uncompressed audiocan be integrated into existingtechnologies (e.g. PCs, tablets)built on existing technologieseasy configuration and connectionaudio + MIDI + OSC...

Emphasis is put on embedded devices

Applications

Live: rehearsal studios

Laptop orchestras

Art installations and exhibitions

Currently we are improving reliability and studying techniques foroutdoor/stage use

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 9

Some history

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 9

40 years of Digital Music Industry

Prophet 5 (1978)

Arch: MCU controlling a full Analog signal path

5-voices polyphony

MCU: (Zilog Z80)

VCO, VCF: SSM (Rev.1,2), Curtis CEM (Rev.3)

6000 units sold

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 10

40 years of Digital Music Industry

Fairlight CMI (1979)

Sampling system

dual Motorola 6800

Memory: 16 kB per voice, System: 64 kB, Video: 16 kB (512x256pixels)

street price: 28,000£!!

Compare: KORG M-1 (1988): cost 1500£, 250,000 units sold

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 11

40 years of Digital Music Industry

MIDI (Musical Instrument Digital Interface) (1981)

Current-loop UART at 31250 baud

3-byte control messages

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 12

40 years of Digital Music Industry

Yamaha DX7 (1983)

Digital FM synthesis, 6 operators

Patches, Storage disk

early MIDI adopter

200,000 units sold

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 13

40 years of Digital Music Industry

Computer Music SW

Max Mathews’ MUSIC (1957), computer language

Miller Puckette’s Max (mid-1980s): visual programming language

...

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 14

40 years of Digital Music Industry

Virtual Analog

DSP to emulate analog waveshapes: Virtual Analog

Motorola 56002 DSP (80MHz, 40MIPS, 24bit fixed point)

Motorola 68331 MCU

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 15

RECAP: HW

Transition era: late 70s to the 80s:MCUs control a full Analog/Discrete Digital signal path

1990s: DSP era (TI, AD, Motorola)Sample-wise, Assembly, Fixed-pointEarly integration with MCU cores (e.g. TMS320C5x + ARM7)

2000s: PCs get to grips with real musicIntel SSE (1999) enhances FP calculus in x86 CPUsMany contemporary music software houses take offNote: Atari/Apple music SW since the late 80s

2000s: Peak point in discrete DSPs usage:Floating+Fixed point, 100MHz-1GHz, C coding, frame-wiseApplication processors: DSP + MCU

Note: ARM attempts successfully at Sig.Proc in 2005 with NEON

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 16

Current HW /2

2008: begin of the smartphone eraARM cores everywhere!

Consequences:ARM dominating in handheldsx86 dominating in PCsLast years: end of the all-SW era: musicians want tangible interfaceshybridization between interface (control) and engine (synthesis):

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 17

40 years of Digital Music Industry

Tux is playin’ (and payin’)

Viscount UNICO Organs (Italy, 2004)

KORG Oasys (2005): Intel Pentium 4 and a ”customized Linux”

KORG Triton (2011): Intel Atom and a ”custom OS over Linux”

Viscount Physis piano (Italy, 2011)

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 18

40 years of Digital Music Industry

Physis Piano (2011)

6 TI OMAP L137 (C67x + ARM9) + ARM9 master MCU

Master core runs Debian

Kernel drivers contributions

And latest news: Japanese manufacturer to release a low-end workstationbased on all-ARM processing?

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 19

all-ARM musical instruments

converging to standardized HW platforms

no SW convergence

side note: Android audio API are lame!

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 20

Constraints

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 20

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 21

Technical Constraints and Issues

Response below perceptual latency (ON-LINE vs REAL-TIME!)

No dropouts

FLOPS!

Robust against EM noise and surge

Gig-proof Build quality

User Requirements

Aesthetics

Sound Quality

Flexibility

User-friendliness

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 22

Typical Team

2 HW Eng

N SW Eng (2-3x Algo, 1x Control, 1x DSP-Sys Integration, 1x Linuxspecialist)

1-3 ”Musicians”

1 Tech Officer

Manufacturing staff

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 23

Tech stuff

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 23

ARM-based systems and GNU/Linux

ARM cores are not designed for DSP, but:leveraging SDKs reduces time to marketSW libraries include optimized DSP routinesReference designs facilitate HW designEVM boards allow parallel HW/SW developmentDSP programming headaches avoided (parallelism, mem sharing, ...)

Viable solutions for ARM development:

eCos

freeRTOS

QNX

Windows CE

Linux

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 24

Current SW stack for audio processing

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 25

Audio servers

JACK is *the* solution

Pulseaudio resamples everything (no synch, adds overhead)

ALSA can only give exclusive control

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 26

Latency: HW and SW

With JACK, latency is HW-dependent only

synchronous processing

Time unit = 1 period p of 2i samples

Total input-output latency:

Lio = (Ni +No)× pFs

+AL

with Ni, No number of buffers (min:2, ping-pong)

AL is the algorithmic latency (0 if no components with memory)

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 27

Scheduling Jack

Jack currently runs as a SCHED FIFO thread plus some SCHED OTHER

threads.

Jack clients run as a DSP SCHED FIFO thread plus SCHED OTHER threads

every Jack client gets a RT priority lower than that of Jack

so Jack can preempt clients breaking the deadline

when a client does not finish within the next IRQ: XRUN!

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 28

Scheduling Jack

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 29

Scheduling Jack

SCHED DEADLINE is aware of periodicity and CPU bandwidth

Can SCHED DEADLINE yield better performance than SCHED FIFO?

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 30

Previous Works

Cucinotta et al. ”Low-latency Audio on Linux by Means of Real-TimeScheduling”, LAC2011

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 31

Scheduling ALSA

Remove the need for:

synchrony

interconnection

extra features of JACK

Take exclusive control of ALSA and have a single audio process running on thesystem.Periodicity can be gained with SCHED DEADLINE

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 32

References

Cucinotta, Faggioli, and Bagnoli. ”Low-latency audio on linux by meansof real-time scheduling.” In Proceedings of the Linux Audio Conference(LAC 2011)

Gabrielli, Leonardo, Stefano Squartini, Emanuele Principi, and FrancescoPiazza. ”Networked beagleboards for wireless music applications.” InEducation and Research Conference (EDERC), 2012 5th European DSP,pp. 291-295. IEEE, 2012.

Gabrielli, Leonardo, Stefano Squartini, and Francesco Piazza.”Advancements and Performance analysis on the Wireless Music Studio(WeMUST) framework.” In Audio Engineering Society Convention 134.Audio Engineering Society, 2013.

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 33