Building an Open-Source based audio streaming platform

Maxime Bugeia

2 February 2020

2

We stream audio!

3

French public broadcasting service

6 national channels47 local channels23 music webradios

2 Million listeners per day200k simultaneous listeners

4

Before

Black Box3rd party Icecast

5

After

Icecast

HLSOur own cloud and open source basedinfrastructure

6

Audio streaming

ICECAST

Streaming server

Master/Relay Architecture

Single bitrate

No cache

HLS

Segmented audio/video files

Sliding Playlist (m3u8)

Adaptive bitrate

Better mobile experience

Cacheable content

Just “static” filesLong persistent TCP connections

Getting audio to the cloud

7

Audio to listeners

Getting audio to the cloud

8

Audio to listeners

Producing stream formats

Icecast HLS

Getting audio to the cloud

Deliver content to listeners

9

Audio to listeners

Producing stream formats

CDNIcecast HLS

Getting audio to the cloud

Deliver content to listeners

10

Audio to listeners

Producing stream formats

CDN

Operate the platform

Icecast HLS

11

Getting audio to the cloud!

LOCAL CHANNELS

WEBRADIOS

NATIONAL CHANNELS

WAN

12

MULTICAST

Transport: multicast problem

DIRECT CONNECT

LOCAL CHANNELS

WEBRADIOS

NATIONAL CHANNELS

WAN

13

MULTICAST

Transport: multicast problem

? DIRECT CONNECT

LOCAL CHANNELS

WEBRADIOS

NATIONAL CHANNELS

WAN

14

MULTICAST

Transport: multicast problem

?No multicast in AWS!

DIRECT CONNECT

15

Transport: SRT to the rescue

LOCAL CHANNELS

WEBRADIOS

NATIONAL CHANNELS

WAN

16

MULTICAST

Transport: to the cloud!

DIRECT CONNECT

17

SRT CALLER 1

SRT CALLER 2

SRT LISTENER 1

SRT LISTENER 2

mainbackup

mainbackup

5 inputs per channel

Transport: resiliencybackup sat

backup sat

18

Audio to cloud

DIRECT CONNECT

SAT BACKUP

19

Producing stream formats!

20

Liquidsoap

Open Source Audio & Video Streaming Language

Collaboration with Radio France

One program to rule them all

Receive Transcode Control

SRT listener Output AACOutput MP3Produce HLSProduce Icecast

Switch between sourcesFallback logicExpose metrics

21

Producing the streams

DIRECT CONNECT

SAT BACKUP

IcecastTranscoderLiquidsoap

HLS

22

LiquidsoapIcecastMaster

Nginx

Icecast stream

HLS stream

Icecast stream

Transcoder Architecture

HLS files

Radio as Code

23

Define inputs

Radio as Code

24

Live: what we want to play

Radio as Code

25

Fallback logic

Radio as Code

26

Fallback logic

Never fail!

27

Output HLS

Radio as Code

Source that never fails

28

Output HLS

Radio as Code

29

Output HLS

Radio as Code

30

Output Icecast

Radio as Code

Source that never fails

31

LiquidsoapIcecastMaster

HLS files

live.m3u8live_hifi.m3u8segment1.tssegmentxxx.ts

Transcoders Architecture

HLS streamNginx

Icecast stream

Icecast stream

32

Scaling to 200k listeners!

33

https://icecast.radiofrance.fr/fip-hifi.aac

Icecast Master

HTTP :80HTTPS :443

Scale: Icecast

Transcoder 1

Transcoder 2

Icecast Relay 1

Icecast Relay 2

Icecast Relay X

AWS NLB

Icecast Relay

34

https://icecast.radiofrance.fr/fip-hifi.aac

Icecast Master

HTTP :80HTTPS :443

Scale: Icecast

Transcoder 1

Transcoder 2

Icecast Relay 1

Icecast Relay 2

Icecast Relay X

AWS NLB

Icecast Relay

No cacheNo autoscalingIcecast Relay automatic fallback

35

HLS base stream

Scale: HLS

CDN

Cache layer

Transcoder 1

Transcoder 2

Varnish 1

Varnish 2

Varnish X

AWS NLB

https://stream.radiofrance.fr/fip/fip.m3u8https://stream.radiofrance.fr/fip/fip_hifi.m3u8

fip_aac_hifi_2_301960_1579533023.ts

Segments

Playlists

36

HLS base stream

Scale: HLS

CDN

Cache layer

Transcoder 1

Transcoder 2

Varnish 1

Varnish 2

Varnish X

AWS NLB

https://stream.radiofrance.fr/fip/fip.m3u8https://stream.radiofrance.fr/fip/fip_hifi.m3u8

fip_aac_hifi_2_301960_1579533023.ts

Segments

Playlists

Autoscaling at cache layerCDN level fallback possible

37

HLS base stream

Scale: HLS

CDN

Cache layer

Transcoder 1

Transcoder 2

Varnish 1

Varnish 2

Varnish X

AWS NLB

https://stream.radiofrance.fr/fip/fip.m3u8https://stream.radiofrance.fr/fip/fip_hifi.m3u8

fip_aac_hifi_2_301960_1579533023.ts

Segments

Playlists

Just like web!

38

Operate

39

LiquidsoapIcecastMaster

Nginx

Icecast stream

HLS stream

Icecast stream

Transcoder Architecture

HLS files

40

LiquidsoapIcecastMaster

Nginx

Liquidsoap xxxx

Icecast stream

HLS stream

Icecast stream

Transcoder Architecture

One Liquidsoap per channel

HLS files

Liquidsoap xxxxLiquidsoap xxxx

Liquidsoap xxxxLiquidsoap xxxx

Liquidsoap xxxx

Automate

41

Declarative everything

Generating:

● SRT callers● Liquidsoap scripts● Icecast master & relay conf

42

Monitor

VisualizeCollectExport metrics

Node Exporter

Liquidsoap

Icecast exporter

43

Monitor

44

HLS access logs

HTTP POST Json

Custom logs Ingester

HLS

Icecast

Monitor: real time listeners

Icecast prometheus Exporter

45

Monitor

46

Learn more

https://github.com/mbugeia/srt2hls

Receive SRT streamProduce HLSPrometheus MetricsHTTP APICommand and fallback logic

https://github.com/Haivision/srt

https://www.liquidsoap.info/

47

Special thanks to the Liquidsoap team!

https://www.liquidsoap.info/

Learn more

For internal use only / prepared with love

Questions?

For internal use only / prepared with love

50

HLS / DASH Compatibility

51

mp3 / aac Compatibility

52

ogg / flac Compatibility

53

NLB

Transcoders

Segments HLSHTTP POST

Segment Forwarder

Storekeeper

PGMPostgreSQL

BusRabbitMQ

Cluster Kubernetes

Segment .ts

Playlists .m3u8

Player

Real HLS diffusion