Embed Size (px)
Citation preview
Continuous testing:What exactly is it and why use it?
Oleksandr Metelytsia
Software Testing IT Weekend Lviv 2016
1
About me
Test Automation Engineer in SoftServe6+ years experience in software testing on different projects
2
Agenda
● What is Continuous Testing?
● Build, Test, Deploy!
● Divide and Conquer allthetests
● Docker and AWS for the win
● Tips and Tricks
● Q&A
3
What’s the problem?
4
Continuous Testing
5
Process of executing automated tests as part
of the software delivery pipeline to obtain
immediate feedback on the business risks associated with a software release candidate.
Continuous Testing
6
Be the change
7
When Continuous Testing Done
8
Sample project from my experience
On-premises server version of a product shipped as virtual image (OVA or AMI)● Server with CLI● Admin web panel● API● Clients: web, mac, win, linux, ios, android
9
Build Test Deploy
10
Simplified Pipeline
11
Our example pipeline
● Merge PR into component repo● Trigger component build● Run component unit tests and build .deb file● Trigger system build● Build product and run “test_build” tests● Trigger all CI plans (acceptance, CLI, perf, etc)
● Press “Deploy” button
12
Version control
● All tests code under version control (Git, svn, etc)○ No excuse not to use
● Distributed version control (GitHub, Bitbucket)● Same PRs flow
○ Don’t hesitate to add devs as reviewers ● Store tests with production code (Unit, Integration)
○ Easy to get older version of the software with related tests● Separate repo for Acceptance test, Perf, etc
13
Testing plans
● Basic regression (selenium, nose, python)● CLI (bash, selenium)● AD/LDAP (another flow of users setup)● Integrations (special pre conditions and selenium)● Web client (nodejs, karma)● Perf (locust, new relic)● Export -> Import● Upgrade plans (RC, dev -1, with reboot, etc)● Component integration tests● ...
14
Basic rules
● Test plans owned by QA team ● Fight tests flakiness
● Plans should be green● Test team checks all “reds”● Jira tickets linked to every valid test fail● Dev team checks labeled CI tickets● Skip test if needed (@skip(“JIRA-ID”))● Execution time for plan < 1 hour (*)
15
Practical tip: parallel jobs with nose
16
Tip: parallel jobs
17
Practical tip: headless mode
18
Tip: headless mode
19
Do not use PhantomJS (*)● not a real browser● additional setup● limitations (e.g. incorrect alerts handing)● “own” bugs (e.g. drag&drop)● not stable on “big” runs (*IMHO)● Xvfb + FF or Chrome are faster* all above valid for version 1.x
Lesson learned: Run Real Browsers Headless.
20
Tip: Automatic screenshot on a fail is awesome
21
Tip: two browsers on one screen1. Pass parameter num_of_browser_instances2. Calculate driver screen resolution:
screen_width() / num_of_browser_instances1. Launch browser at desired position
driver.set_window_position(width_coordinate, 0)
22
●Long running test environmentvs
●Launch new fresh env on each CI run
Test environment
23
AWS
EC2 AMIs For The Win:● production like environment● fast deployment● “super” speed between S3 buckets● secure isolated environment (VPC)● pay as you go
24
● AWS CLI for your CI● Route53 for DNS● AWS Lambda - serverless computing● Additional servers for infrastructure stuff like
AD, mail or proxy server● Different instance types● CI agents in EC2● and many more...
and more AWS
25
Typical test development env problems● difficult to set up and maintain● hard to track dependencies
- python versions- browser versions- selenium and other packages versions- other tools and software
● managing several environments● OS differences between CI and local env● inconsistency of executions
26
Docker
Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment.
27
Docker diagram
28
Dockerfile
29
DockerBuild container:$ docker build -t firefox:42 .Push to registry:$ docker push your_registry/firefox:42Pull from registry:$ docker pull your_registry/firefox:42Run container:$ docker run -t -i -v “/app:/app” firefox:42
30
● Docker is not a virtual machine solution● thousands images in docker hub● own private docker registry● run the same container on linux, mac, win ● lightweight to run● lightweight to store● lightweight to change
Docker
31
Docker in testing CI
Simplified pattern:
32
Practical tip: CI your images
1. Put Dockerfiles under version control2. Create CI plans to build and push images3. Trigger plans on repo commit4. Profit
33
Testing is not a destination but a journey
34
Lessons learned
● Page object pattern is good● Decorators are good @with_user, @retry and so on● Turn off rate limits if needed, but don’t forget to cover them with
tests first● Api testing is fast and reliable. You should have more Api tests than
end-to-end● Run Integration tests on server (port forwarding is not the same)● Adopt tests from other teams in your CI
36
Lessons learned● Dockerize your tests running in CI● CI your docker images● Think how AWS can boost your testing● Use CI artifacts to store screenshots and page source of fails● Save logs
○ Plan execution logs
○ Test execution logs
○ Server components logs● Branch builds with unit test● Do perf testing continuously● Fight tests flakiness
37
Further reading
● Docker Basics: A practical starters guide● How Docker Makes Testing More Efficient● Getting Started with AWS● Continuous Testing Book
38
