SONiC Ansible Test Automation… · SONiC Ansible Test Automation Taskin Ucpinar Senior Director of SW Development Edgecore Networks EW: ... Release SAI Version No of Features No

SONiC Ansible Test Automation

Taskin UcpinarSenior Director of SW DevelopmentEdgecore Networks

EW: Networking / SONiC

SONiC Development & Testing: Goals

• End-to-end testing in development workflow• Coverages and Automation• HW platform validations• Pre-SI RAS, Performance testing • Sonic ecosystem integrations

NETWORKING

TestedConfigurations

Edgecore’s Dedication to SONiC

AS7716-32XB

AS7326-56X

AS5712-54X

AS7116-54X

AS7212-54X

AS7312-54X

AS7312-54XS

AS7816-64XAS6712-32X

AS7712-32X

AS7716-32X

AS7712-32X

AS9716-32D

AS7312-54XSAS4222-28PE

MiniPackAS7512-32X

NETWORKING


https://www.google.com/imgres?imgurl=x-raw-image:///6664c389e0b59fe9f251643bef4763cb0907ce2e024ddf5d73eb74df61cc7f1e&imgrefurl=https://www.edge-core.com/_upload/images/AS7512-32X_ONIE_DS_Cavium_R03-20170928.pdf&docid=Em5ig9B1vxkegM&tbnid=6Z-14InDZGaw2M:&vet=10ahUKEwjM24Cdu-_gAhXD854KHf71DkQQMwhBKAYwBg..i&w=425&h=173&bih=1277&biw=2025&q=AS7512-32X&ved=0ahUKEwjM24Cdu-_gAhXD854KHf71DkQQMwhBKAYwBg&iact=mrc&uact=8

https://www.google.com/imgres?imgurl=https://www.edge-core.com/timthumb.php?src%3D_upload/images/1812271018362.png%26h%3D357%26w%3D490%26zc%3D3&imgrefurl=https://www.edge-core.com/productsInfo.php?cls%3D1%26cls2%3D154%26cls3%3D155%26id%3D544&docid=C4Z0DUqrOoUOpM&tbnid=QpDjKoY99_zReM:&vet=10ahUKEwjep_3Uuu_gAhUEsp4KHZgBAiAQMwg_KAEwAQ..i&w=489&h=357&bih=1277&biw=2025&q=AS7326-56X&ved=0ahUKEwjep_3Uuu_gAhUEsp4KHZgBAiAQMwg_KAEwAQ&iact=mrc&uact=8

https://www.google.com/imgres?imgurl=https://www.epsglobal.com/getattachment/1a4ad313-7266-41f6-9db3-2a6726b8168a/Network-World-Chooses-Edge-core-s-AS5712-54X-White&imgrefurl=https://www.epsglobal.com/about-eps-global/latest-news-3/january-2016/network-world-chooses-edge-core-s-as5712-54x-white&docid=bZUZwugiFsOLtM&tbnid=hFrWuNplDUaEwM:&vet=10ahUKEwjLj-vdue_gAhXG_J4KHZnXDCwQMwhCKAQwBA..i&w=1200&h=627&itg=1&bih=1277&biw=2025&q=AS5712-54X&ved=0ahUKEwjLj-vdue_gAhXG_J4KHZnXDCwQMwhCKAQwBA&iact=mrc&uact=8





https://www.google.com/imgres?imgurl=https://www.edge-core.com/timthumb.php?src%3D_upload/images/1701201534492.png%26h%3D278%26w%3D278%26zc%3D3&imgrefurl=https://www.edge-core.com/&docid=sxFAn-ZTnsmKFM&tbnid=0wcgQ7jUd4I9EM:&vet=10ahUKEwiG2u-Vo-7gAhU1JzQIHbX7AN0QMwg_KAEwAQ..i&w=278&h=202&bih=1276&biw=1944&q=accton%20network%20switch&ved=0ahUKEwiG2u-Vo-7gAhU1JzQIHbX7AN0QMwg_KAEwAQ&iact=mrc&uact=8








Edgecore’s Dedication to SONiC

• SONiC Feature Set Growth• Rapid Development Environment• Ensure Stability/Reliability• Testing• Interoperability• Regression• CI/CD

NETWORKING


Regression Testing

• SONiC and Branching• Getting the full picture• Completing the coverage• Hardening • Community Services

NETWORKING


SONiC Branches

And How Edgecore Handles Them…

NETWORKING


1803 1807 1811 1904

1 2 3

1 2 3 4 5

SONiC and Releases

NETWORKING


1811 1904

1

EC1811

2 3 4

1

EC1904

2 3 4 5

Edgecore’s Release Test Branches

NETWORKING


Testing Releases

(and some more)

NETWORKING


Getting The Full Picture

SONiC201705 BGP

ECMP

LAG

LLDP

QoS - ECN

One Image

...

SONiC201709 VLAN

ACL Per/Den

IPv6

Tunnel Decap

Mirroring

BGP MP

...QoS - RDMA

NETWORKING


Getting The Full PictureRelease SAI Version No of Features No of Ansible Tests

SONiC201705 0.9.4 18 9

SONiC201709 0.9.4 8 4

SONiC201712 1.0 7 2

SONiC201803 1.2 5 2

SONiC201807 1.3 3 0

SONiC201811 1.3 6 2

SONiC.201904 1.4 16 5

Total 63 24

NETWORKING


Testing From All Angles

• Test units/components/functions/functionality• Not only for testing, but for training purposes• Independent on features/platforms• Unit Tests: Typically implemented by developers• Functional Tests: Implemented by developers and testers• Test Frameworks

• gtest• pyunit

NETWORKING



100% Coverage

• All Features with Tests

• Coverage % Unknown

• Stability Unknown• Stability Improvements

• Measure Coverage %

• New tests

Hardening

Edgecore Community Labs

• Multiple Community Labs Across Globe

• SONiC Devices and Packet Generators

• Available to Edgecore Partners, Customers, Community

AAutomation

• Auto detect and execute

• Regression Testing

62%

• Features Missing Tests

• Coverage % Unknown

• Stability Unknown

Completing The Picture

NETWORKING

Development Environment

Edgecore

Cloud

Build

Tools gcc/cc/gdb

make/autoconfig/pkg-config

cmake

Source Codesonic-swss

sonic-swsscommon

…

Test(s)

Dependencieslibnl

python

…

PyUnitDevEnv

CI/CD PyUnit

Code Review

Regression UT Regression FT Test(s)

Static Analysis Performance …

NETWORKING


https://www.google.com/imgres?imgurl=https://cdn-images-1.medium.com/max/1200/1*RmcSmwPhUn8ljLiiwYxK0A.png&imgrefurl=https://medium.freecodecamp.org/how-to-make-an-ios-on-demand-build-system-with-jenkins-and-fastlane-8eb1e02c73d1&docid=euHLfyQfzypbqM&tbnid=w9cbKkYK3iFB1M:&vet=10ahUKEwif_fr7s-7gAhXRBjQIHcD1BScQMwhpKAAwAA..i&w=690&h=710&bih=1276&biw=1944&q=jenkins&ved=0ahUKEwif_fr7s-7gAhXRBjQIHcD1BScQMwhpKAAwAA&iact=mrc&uact=8

https://www.google.com/imgres?imgurl=https://www.cybercon.com/media/catalog/product/cache/1/image/9df78eab33525d08d6e5fb8d27136e95/b/l/blade_unix_250x250_3.gif&imgrefurl=https://www.cybercon.com/dedicated-unix-server-hosting.html&docid=VoSrye4SGd8LgM&tbnid=XfmPdidNA-b3JM:&vet=10ahUKEwitj5W6rO7gAhWPjp4KHaupBWIQMwhcKBEwEQ..i&w=250&h=250&bih=1276&biw=1944&q=linux%20server%20icon&ved=0ahUKEwitj5W6rO7gAhWPjp4KHaupBWIQMwhcKBEwEQ&iact=mrc&uact=8

https://www.google.com/imgres?imgurl=https://git-scm.com/images/logos/downloads/Git-Icon-1788C.png&imgrefurl=https://git-scm.com/downloads/logos&docid=zURF9QyvmP0d4M&tbnid=jpq4xLgv36ZQ1M:&vet=10ahUKEwjqtpTrtu7gAhWDvJ4KHdoyCsMQMwhWKAAwAA..i&w=383&h=383&bih=1276&biw=1944&q=git%20%20icon&ved=0ahUKEwjqtpTrtu7gAhWDvJ4KHdoyCsMQMwhWKAAwAA&iact=mrc&uact=8

https://www.google.com/imgres?imgurl=https://cdn-images-1.medium.com/max/1200/1*RmcSmwPhUn8ljLiiwYxK0A.png&imgrefurl=https://medium.freecodecamp.org/how-to-make-an-ios-on-demand-build-system-with-jenkins-and-fastlane-8eb1e02c73d1&docid=euHLfyQfzypbqM&tbnid=w9cbKkYK3iFB1M:&vet=10ahUKEwif_fr7s-7gAhXRBjQIHcD1BScQMwhpKAAwAA..i&w=690&h=710&bih=1276&biw=1944&q=jenkins&ved=0ahUKEwif_fr7s-7gAhXRBjQIHcD1BScQMwhpKAAwAA&iact=mrc&uact=8

Automated Testing Environment

NETWORKING


https://www.google.com/imgres?imgurl=https://about.gitlab.com/images/press/logo/jpg/gitlab-logo-gray-stacked-rgb.jpg&imgrefurl=https://about.gitlab.com/press/press-kit/&docid=jENCnVRcPgClgM&tbnid=qWSGRl99RJI1zM:&vet=10ahUKEwjJiYju8_bjAhUBOH0KHXuxAU8QMwhoKAAwAA..i&w=1730&h=1568&bih=792&biw=1300&q=gitlab%20logo&ved=0ahUKEwjJiYju8_bjAhUBOH0KHXuxAU8QMwhoKAAwAA&iact=mrc&uact=8

https://www.google.com/imgres?imgurl=https://a0.awsstatic.com/libra-css/images/logos/aws_logo_smile_1200x630.png&imgrefurl=https://aws.amazon.com/&docid=DQFYE7-PMXDa7M&tbnid=C0-Qy9ehSkUHYM:&vet=10ahUKEwjwjcKYo4XkAhXGpZ4KHXsNDYQQMwhnKAAwAA..i&w=1200&h=630&bih=1021&biw=1464&q=Amazon%20AWS%20png&ved=0ahUKEwjwjcKYo4XkAhXGpZ4KHXsNDYQQMwhnKAAwAA&iact=mrc&uact=8

https://www.google.com/imgres?imgurl=https://logodix.com/logo/989411.png&imgrefurl=https://logodix.com/jenkins&docid=Y4wmN9dP-htp7M&tbnid=d8q00HPoxDh1yM:&vet=10ahUKEwj5uOSXqIXkAhVgHTQIHfv0AVEQMwhpKAEwAQ..i&w=676&h=411&itg=1&bih=1021&biw=1464&q=Jenkins%20logo%20png&ved=0ahUKEwj5uOSXqIXkAhVgHTQIHfv0AVEQMwhpKAEwAQ&iact=mrc&uact=8


NETWORKING


https://www.google.com/imgres?imgurl=https://about.gitlab.com/images/press/logo/jpg/gitlab-logo-gray-stacked-rgb.jpg&imgrefurl=https://about.gitlab.com/press/press-kit/&docid=jENCnVRcPgClgM&tbnid=qWSGRl99RJI1zM:&vet=10ahUKEwjJiYju8_bjAhUBOH0KHXuxAU8QMwhoKAAwAA..i&w=1730&h=1568&bih=792&biw=1300&q=gitlab%20logo&ved=0ahUKEwjJiYju8_bjAhUBOH0KHXuxAU8QMwhoKAAwAA&iact=mrc&uact=8






NETWORKING







0

10

20

30

40

50

60

70

80

1 2 3 4

No of Testbeds vs Execution Time (hrs) *

• Target AST7816-64X

• Release 1904

• 234 Test Cases

• Clean build before tests

• Cleanup Between Runs

• Reset Image & Config

• Reboot

NETWORKING


Edgecore’s Contribution on 1904

In Progress Plan to

Submit

Submitted Merged Total

unit test 5 12 11 11 38

SONiC 6 4 10

sonic-mgmt

(test case)

10 7 17

Total 5 12 27 22 65

NETWORKING



• Target AST7816-64X

• Release 1904

• 234 Test Cases

• Clean build before tests

• Cleanup Between Runs

• Reset Image & Config

• Reboot

0

50

100

150

200

250

Failed Passed Total

NETWORKING


Documents

SONiC Ansible Test Automation… · SONiC Ansible Test Automation Taskin Ucpinar Senior Director of SW Development Edgecore Networks EW: ... Release SAI Version No of Features No