Testing in the world of Internet of Things

Dawid Pacia

Tomasz Janiszewski

“The best way to predict the future

is to invent it.

Alan Kay, Xerox PARC, 1971

What is IoT?The Internet of Things (IoT) is the network of physical objects - devices, vehicles, buildings and other items embedded with electronics, software, sensors, and network connectivity - that enables these objects to collect and exchange data.

https://en.wikipedia.org/wiki/Internet_of_Things

What is going to be the next "big thing"? IoT?

Silvair products● Based on TI CC2540/41 &

Nordic nRF51/52● Bluetooth Low Energy v4.0● Radio range: up to 500 meters

Silvair products

Silvair products

● Android 5.0+ app

● iOS 9.0+ app

Testing Scope & Challenges

Hardware

○ Drivers○ Switches○ Sensors○ Luminaries○ Motors○ Light sources / bulbs○ Multi-protocol

gateways

Firmware○ Bluetooth Low

Energy v4.0 protocol support

○ Silvair Smart Mesh protocol support

○ Provisioning○ Discovery○ Groups and Scenes○ OTAU○ High Security

Software○ Network setup○ On / Off, dimming, color change○ Associations○ Factory Reset○ OTAU○ Cloud

“In the beginning there was only Chaos”

● Event-driven development (for CES, BT World, etc.)● Chaotic environment with constantly changing

priorities (on a daily basis)● Lack of time to test SF/HW/FW on all test levels● Ignoring product/project risk analysis ● No TDD/FDD/XP, lack of Unit tests● PoC….. PoC everywhere● No documentation● Not enough testers!

Main focus● Building and maintaining test environment

● Test automation

● Helping other teams with their testing activities

Let’s imagine a simple test scenario...

Story: The user turns on a light bulb from an app and turns it off from a wireless switch

Step 1: Launch mobile app and create a home network with 1x wireless switch and 2x light bulbsStep 2: Turn on the bulbs from an appStep 3: Turn off the bulbs from a switch

Flashing App installation

Devices pre-configurationSTART Init

Test scenario flow:

Changing bulb state

(Step 2)

Switch control(Step 3)

Test suite execution

Closure activities

Creating “Home” network(Step 1)

END

Simple… right?

Well... not quite...

○ How to flash a BLE module with the firmware which is inside a bulb or a switch?

○ How to control hardware buttons on a switch on demand?

○ How to verify whenever a bulb has been turned on or not?

○ How to verify if desired color, brightness, color temperature or saturation level has been reached?

Challenges:

○ How to combine the results from multiple sources (mobile app, firmware, hardware, cloud) and be sure that the test cases is passed and functionality under test is working as expected?

○ How to to know which component has failed and when?

○ How to detect potential problems like unpleasant LED flickering while dimming?

○ How to reset everything to default state at the end of the test scenario?

Challenges:

Hardware

Firmware

Embedded platforms

Cloud

Vision system

Continuous integration

Electric

circuitsMobile

applications

AUTOMATIZATIONAutomatic reporting

REST

Multiple

platform

s

Want big impact?

Use big image.

THEY SAID THIS JOB WOULD BE EASY

YOU GOTTA BE KITTEN ME

Let’s make it step by step !

● Following common methodology● Facilitating external processes

● Automation tools freedom● Considered architecture ● Brainstorming sessions

● Team autonomy

“...Possibilities kills creativity...”

Requirements of IoT E2E Test Environment

● Dirt-cheap! ● Simplicity - built with a KISS

("Keep it simple, stupid!") methodology, extremely low entry point for system newcomers

● Modularity - the system should handle flawlessly a multiple of different DUT’s (Devices Under Test)

Requirements of IoT E2E Test Environment

● Scalability - a multiple instances of DUT modules and Test Arbiters should be able to operate simultaneously without any hassle

● Reliability and stability - the testing environment should be able to operate with a MTBF of 7 days

“...Possibilities kills creativity...”

Let’s prototype!

Every engineering

starts with LEGO

CloudREST API

SSH

UART

Module Flashing

MASTER

SSH

REST API

BLE

DEBUGGER UART

FW DB

AUTH TOKEN / REST

BLE

DEBUGGER

Switch Control

class switchTest(unittest.TestCase): self.authorization = ["rpi-devDav.local","qateam","123"] …

self.channelOff2 = self.config["channelOff2"] def testOnOff(self): switch = switchLib(self.config["switchAddress"], self.config["login"], self.config["password"], self.config["platform"], self.channelOn1, self.channelOn2, self.channelOff1) switch.switchClick(self.channelOn1) …

switch.switchHold(self.channelOn1,3) switch.switchHold(self.channelOff1,3) switch.switchWakeUp()

switch.factortReset() def tearDown(self): pass

Switch Control

Vision System

The very first thing - prepare your workstation!

● Unchangeable conditions○ Lighting (!)○ Filters

● Stability (!)● Usability

○ Effort in reconfiguration

○ Reusage● Costs

The very first thing - prepare your workstation!

Trigger signal

Set proper parameters /

disable automatic

Send output

How it really looks like:

Take a snapshot, image

preprocessing (filtering)

Kill the process Is camera

buffer released?

Image processing (blob detection, check elements, etc.)

Thresholding Create

binary ROI matrix

Trigger signal

Gather value matrix

Send output

How we would like it:

Vision System

v4l2-ctl -c exposure_auto=1 | v4l2-ctl -c exposure_auto_priority=0 | v4l2-ctl -c white_balance_temperature_auto=0 | v4l2-ctl -c absolute_exposure=171

Set proper parameters (always manually!)

*Collection of libraries which adds a thin abstraction layer on top of video4linux2 devices. The purpose is to make it easy for application writers to support a wide variety of devices without having to write seperate code for different devices in the same class.

brightness (int) : min=0 max=255 step=1 default=128 value=128 contrast (int) : min=0 max=255 step=1 default=32 value=32 saturation (int) : min=0 max=255 step=1 default=32 value=32white_balance_temperature_auto(bool) : default=1 value=0 gain (int) : min=0 max=255 step=1 default=0 value=0 power_line_frequency (menu) : min=0 max=2 default=2 value=2 white_balance_temperature (int) : min=0 max=10000 default=4000 value=4000 sharpness (int) : min=0 max=255 step=1 default=24 value=24 backlight_compensation (int) : min=0 max=1 step=1 default=1 value=1 exposure_auto (menu) : min=0 max=3 default=3 value=1 exposure_absolute (int) : min=1 max=10000 default=166 value=171 exposure_auto_priority (bool) : default=0 value=0

...and this is why

[109, 5, 2, 6, 2, 3, 3, 3, 104]

Good Exposure (value 171)

[173, 58, 33, 54, 47, 51, 38, 48, 169]

Overexposure(value 800)

Image preprocessing (filtering)

filtering (blur), thresholding

noise reduction, morphological

(erosion)

(146, 193, 255)

(255, 64, 17)

Response analysis

Purpose of the extended blob detection

# Filter by Area.params.filterByArea = Trueparams.minArea = 1500

# Filter by Circularity# (4 * PI * Area) / (Perimeter^2)params.filterByCircularity = Trueparams.minCircularity = 0.6

# Filter by Inertia# It measures how elongated a shape isparams.filterByInertia = Trueparams.minInertiaRatio = 0.8

# Filter by Convexity# Area of the Blob / Area of its # convex hullparams.filterByConvexity = Trueparams.minConvexity = 0.87

Purpose of the extended blob detection - example

BAD VERY BAD OK

QA CI Environment

Test Arbiter

JIRA TEST

RESULTS

iOS iOS

APK APK Cloud

TEST RESULTS

E2E Test Environment

Appium Appium

Test Arbiterclass Silvair(unittest.TestCase): driver = None app = None @classmethod def setUpClass(cls):

……

cls.restAPIClientBulb1 = RestAPIClient(cls.config["restAPI_IP_bulb1"],cls.usernameRestAPI,cls.passwordRestAPI,cls.projectName) cls.restAPIClientBulb2 = RestAPIClient(cls.config["restAPI_IP_bulb2"],cls.usernameRestAPI,cls.passwordRestAPI,cls.projectName) cls.restAPIClientSwitch = RestAPIClient(cls.config["restAPI_IP_switch"],cls.usernameRestAPI,cls.passwordRestAPI,cls.projectName) cls.restAPIClientBulb1.fullFlashingFlow(cls.config["bulbOrderID"]) cls.restAPIClientBulb2.fullFlashingFlow(cls.config["bulbOrderID"]) cls.restAPIClientSwitch.fullFlashingFlow(cls.config["switchOrderID"])

Test Arbiter... print("[LOG] " + datetime.now().strftime("%Y-%m-%d %H:%M:%S") + " Appium server started.")

downloadFromS3(cls.config["s3ApkPath"], os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + "/apk", cls.config["APP_NAME"]) cls.driver = create_driver('Android', configFile, bool(int(cls.config["reinstall_app"]))) cls.driver.implicitly_wait(5) cls.bulbMac1 = cls.config["bulbMac1"] cls.bulbMac2 = cls.config["bulbMac2"] cls.switchMac = cls.config["switchMac"] cls.retryNumber = int(cls.config["retryNumber"]) cls.bulbSlot1 = int(cls.config["bulbSlot1"]) cls.bulbSlot2 = int(cls.config["bulbSlot2"]) cls.bulbProvisioningWorkaround = cls.config["bulbProvisioningWorkaround"] cls.factoryResetWorkaround = cls.config["factoryResetWorkaround"] cls.HWbulbWorkaround = cls.config["HWbulbWorkaround"]

Test Arbiter

... cls.androidLib = android(cls.driver) cls.bulbEnv = BulbEnv(cls.config["deviceNumber"], cls.config["resolutionX"], cls.config["resolutionY"], cls.config["rowsNum"], cls.config["columnsNum"]) cls.switchEnv = switchLib(cls.config["switchAddress"], cls.config["login"], cls.config["password"], cls.config["platform"], cls.config["channelOn1"], cls.config["channelOn2"], cls.config["channelOff1"]) cls.app = Pages(cls.driver, cls.switchEnv) cls.e2eEnv = e2eLib(cls.app, cls.bulbEnv, cls.switchEnv, cls.config["minThreshold_on"], cls.config["maxThreshold_on"])

Number of commits 212

Number of code lines added 69293

Number of code lines deleted 46028

Number of Github contributors

3

Number of Jenkins jobs 18

Number of builds 777

QA CI Environment

QA CI Environment

Test Arbiter

JIRATEST

RESULTS

TEST RESULTS

DUT DUT DUT

def test12_associations(self): self.app.clickDeviceOnAddedList(self.switchMac) self.switchEnv.switchWakeUp() self.e2eEnv.switch_turnOffBulbs(self.config["channelOff2"], [self.bulbSlot1, self.bulbSlot2], 0, self.retryNumber) self.app.doAssociation(self.switchMac, 0, "none") self.switchEnv.switchWakeUp() self.e2eEnv.switch_turnOffBulbs(self.config["channelOff2"], [self.bulbSlot1, self.bulbSlot2], 0, self.retryNumber) self.app.swipeUp() self.app.doAssociation(self.switchMac, 1, self.bulbMac1, "Lightness") self.driver.back()

Tests creation? Couldn’t be simplier !

def test18_FRswitchHW(self): self.androidLib.resetBT() self.switchEnv.switchWakeUp() self.app.addDevice(self.switchMac) self.app.findAddedDeviceOnList(self.switchMac) self.e2eEnv.switch_FR(self.switchMac, 5)

def test08_provisioningOfNodOn(self): self.switchEnv.switchFactoryReset() self.switchEnv.switchWakeUp() self.app.addDevice(self.switchMac) self.app.findAddedDeviceOnList(self.switchMac)

Tests creation? Couldn’t be simplier !

def test14_sharing(self): self.app.share("Gmail") self.app.share("Save to Drive")

USE YOUR BRAIN !

IoT BAT

● Long time SSH connection (fast switch state change)

● PWM high frequency (UART testing) ● Electrostatic Discharge ● Busy camera buffer● Appium server / flashing / setting devices test

steps timeouts● Different screen sizes / language versions /

platforms

but where is a problem...

Just a few problems met...

● Long time SSH connection ● PWM high frequency (UART

testing)● Electrostatic Discharge● Busy camera buffer● Appium server / flashing /

setting devices test steps timeouts

● Different screen sizes / language versions / platforms

...there is a solution !

● Cert Authorization

● I/O stream log deletion

● Humidifier

● Monitoring camera process and usage

● Moved as set-up activity

● Gather necessary informations and determine while class initiation

… unfortunately, not for all of them :(

Performance Mobile Phone

Number of ADV packets received from a Bulb within 15 minutes The percentage of

packets received [%]Clean RF environment

Noisy RF environment (1200 packets per

second)

VERY GOOD Galaxy Tab 7.0 (T230) 3921 2312 59

GOOD

LG G2 3395 1594 47

Nexus 6 5294 1948 37

Samsung Galaxy S6 5818 2130 37

AVERAGE

Samsung Galaxy Note 4 5070 1607 32

Nexus 9 5377 1541 29

Sony Xperia Z2 5862 1604 27

Samsung Galaxy S5 5784 1494 26

Nexus 5 5964 1518 25

BAD

Samsung Galaxy S5 mini 6397 1197 19

Samsung Galaxy Note 10.1 5268 751 14

Samsung Galaxy S3 6018 641 11

VERY BAD

Xiaomi Mi3 3237 278 9

LG G2 mini 4081 328 8

Samsung Galaxy A3 4050 319 8

Samsung Galaxy S4 mini 4037 329 8

HTC One M8 3868 279 7

Sony Xperia T3 4040 283 7

Takeaway...● Does Scrum really work? Yes, it does!● Think big, but act small● Iterate and refine● Unleash team creativity● Focus on interoperability● Modularity is the key● Incorporate RBT - focus on risky functionalities,

e.g. OTAU● Physics is ruthless● IoT is also hardware● Security!

Want to know more about IoT protocols? Just grab our free ebook at: https://silvair.com/media

Thanks!Any questions?

You can find us at: Dawid Pacia:

◎ e-mail: paciadawid@gmail.com◎ / paciadawid

Tomasz Janiszewski: ◎ e-mail: janiszewski.tomasz@gmail.com◎ / tjaniszewski

https://silvair.com/careers