Quality Assurance for WebRTC Services

Quality Assurance for WebRTCServicesMarch 2016, Kranky Geek India

Tsahi Levent-Levi

People don’t usually test their WebRTC implementationsSeriously

2

And when they do, they test things manually

3

Testing and automation are mandatory and widely usedMeanwhile, on other software fields…

4

How do you go about automating your WebRTCtestbed?7 suggestions for you

5

W. Heath Robinson

• WebRTC changes… monthly

• You have more than a voice callto test

Rely on the browser

6

4041

4243

4445

4647

4849

3536

3738

3940

4142

4344

45

Dec-14 Apr-15 Jul-15 Oct-15 Jan-16 May-16

Chrome

Firefox

Selenium is your friend

1

Rely on the browser… and peer into the future

7

1

Stable What your customers are using

Beta What you should expect in the next release

DevWhere you can complain about breakage – and expect fixes in time

Canary A nightly build of whatever is available at that moment in time

Testing only against the Stable version of browsers means you find out issues only after the service breaks for your customers.

> ffmpeg -i MYFILE.mp4 -pix_fmt yuv420p

> sed -i '0,/C420mpeg2/s//C420/' *.y4m

> google-chrome --use-fake-device-for-media-stream

--use-file-for-fake-video-capture=MYFILE.y4m

See http://testrtc.com/y4m-video-chrome/ for more details

Handle your media feeds

8

and think about the type of content you need2

For some arcane reason (called AGC or something), browsers will loop your fake video but won’t look your fake audio

Loop an input 6 times (=expand 5 minutes to 30)> ffmpeg -f concat -i <(for i in {1..6}; do echo "file '$PWD/INPUT.wav'"; done) –c

copy OUTPUT.wav

> google-chrome --use-fake-device-for-media-stream

--use-file-for-fake-audio-capture=OUTPUT.wav

9

2 Handle your media feedsand make sure your audio recording is suitably long

Take care of the network

10

Use netfilter.org iptables

Network Impairments

• Bitrate limitation• Packet loss• Jitter & latency

Firewall Configuration

• None• Force TURN relay• HTTP only• HTTPS only

Multiple Locations

• Virtualization sucks• Stability & Throughput

is what you need• Explore barebone

options

3

• Caller / Callee

• Caller / Agent

• Caller / Expert

• Meeting point

• Conference

• Webinar

Handle synchronization

11

4

Can you see the problem?

Visualize the results

12

5

Visualize the results

13

Chrome 48 with VP8 Chrome 49 with VP9

5

Are these real network issues with packet losses or bugs in these products and how they report packet loss?

Visualize the results (2)

14

5

• How much time does it take to connect the session?

• Are the media channels you need really there?

• Are you sure there aren’t any “additional” media channels?

• Is the bitrate following your restrictions?

Don’t forget to set your expectations

15

6

You never know when things will become handy…

Here are a few thoughts:

• chrome://webrtc-internals dump

• Browser console logs

• Media recordings

• Machine performance

Collect EVERYTHING

16

7

Don’t take WebRTCtesting lightly

17

Thank YouTsahi Levent-Levi

http://bloggeek.me

tsahil@bloggeek.me

18