Embed Size (px)
Citation preview
COLORFUL WORLD of
VISUAL TEST AUTOMATIONOnur Başkırt
Who am I?
▪ I am Onur Başkırt
▪ Head of Software Testing at
▪ Co-owner of
▪ Formerly: Teknosa, Ericsson, Huawei, STMicroelectronics, BAU & USCS
Online Employment and Recruitment Market Leader in TURKEY since 1999
5.6 millionVisitors
180 millionPage views
25 millionResumes
75.000+Employers
Technical Software Testing Blog
Automation Performance Security DevOps
Outline• What is Visual Test Automation?
• Why Visual Testing?
• Why Automation?
• Challenges
• Visual Test Automation Tools
• Brief Overviews
• Detailed Reviews (Applitools Eyes, Percy.io, Galen Framework)
• Selenium & ImageMagick & AShot Example
• Q&A
What is Visual Test Automation?• Verifies the GUI is displayed as expected to the end users.
• Focuses on visual contents: • Colors• Shapes• Sizes• Locations of elements
UI Test vs Visual Test
Why do We do Visual Testing?
WordPress Template
CSS animations Widgets Sliders etc.
Mobile App Generation
Engine
.ipa
.apk
GUI MUST APPEAR CORRECTLY
USER INTEARCTIONS
COMPANY BRAND
CRITICAL PRODUCTS and SERVICES
Why do We Automate Visual Testing?
Variety of Test Environments• Browsers – Mobile Devices – Resolutions etc.
Time to Market Pressure• Release cycles getting shorter.
Manual Visual Testing is Cumbersome• Human eyes error-prone and getting blind.
In Mobile World, you will live with visual bugs longer• Can not update mobile app daily.
What are the Challenges? • Anti-aliasing
What are the Challenges? • Dynamic Content (CSS Animations, GIFs)
What are the Challenges?
Dynamic content
Dynamic Content
Auto-Kerning Offsets
BRIEF OVERVIEW of Well-Known
VISUAL AUTOMATION TOOLS
PhantomCSS (Open-Source by Huddle, 4197+ Stars on GitHub)
• PhantomJS (Webkit) and SlimerJS (Gecko)Browsers
• CasperJSNavigation
• ResembleJSComparison
• Ignores Colors, Antialiasing, Moving ElementsFeatures
WebdriverCSS (Open-Source, 447+ Stars on GitHub)
• Chrome, Firefox, PhantomJSBrowsers
• WebdriverIO v2.0 Selenium Bindings for NodeJSNavigation
• GraphicsMagickComparison
• Ignores Color, AntialiasingFeatures
Wraith (Open-Source by BBC, 3883+ Stars on GitHub)
• PhantomJS, SlimerJSBrowsers
• Configuration Based .yaml file• CasperJSNavigation
• ImageMagickComparison
• Fuzz Option• Antialiasing and small changesFeatures
• and many tools exists in the market both open-source and commercial.
• FBSnapshotTestCase• Needle• Rspec Page Regression• Pix-Diff • Selenium Visual Diff• Vizregress • VisualCeption• Specter• Fighting Layout Bugs• dpdxt• Huxley• SiteEffect• Shoov• Screener.io
• BaskstopJS• CSSCritic• Grunt Photobox• VIFF• GreenOnion• Kobold• CSS Visual TestAsd• Snap And Compare• Grunt-Vigo• Hardy• Browsershots• Screenbeacon• VisualReview• Gemini
APPLITOOLS EYES
What is Applitools Eyes?• Automated Visual Testing Tool for Web sites and Mobile apps
• Finds Visual Differences with Several Match Levels
• Eyes SDK for Selenium & Appium
• Cloud-Based Clean and Informative Dashboard
How Does it Work?
Visually notifies the differences
At 2nd run and more it compares actual screenshots with the baselines.
At 1st run it saves all screenshots as baseline
Takes screenshots with Eyes SDK and sends to the Applitools cloud
Match Levels and Their Meanings?
• Pixel by Pixel ComparisonEXACT
• Mimics the human eyes. • Ignores trivial mismatches (antialiasing, small pixel movements, etc.)STRICT
• Similar with STRICT plus it ignores color changes.CONTENT
• Compares the layouts.• Checks the elements locations.LAYOUT
4 Main Steps of Eyes SDK
Close the Testeyes.close();
Add UI validation Check Pointseyes.checkWindow("Validation Window Name");
Start a Testeyes.open(driver, APP_NAME, TEST_NAME);
Create and Configure an Eyes ObjectEyes eyes = new Eyes();
Mobile Visual Testing ExampleOpen BitbarSampleApp.apk
* Visually check that start screen as expected?
Click second option “Use Testdroid Cloud”
*Visually check that Second Option is clicked?
Write “SW Test Academy” to text bar
Click Answer button
Wait until the second page appear
*Visually check that the answer is correct?
Close the driver.
Mobile Visual Testing ExampleStep-1: Install Eyes SDK with Maven Dependency
<dependency> <groupId>com.applitools</groupId> <artifactId>eyes-selenium-java</artifactId> <version>RELEASE</version></dependency>
Step-2: Do Appium, Android SDK and GenyMotion Settings
Mobile Visual Testing ExampleStep-3: Configure Eyes SDK in your Setup Method
Mobile Visual Testing ExampleStep-4: Open Eyes Do Visual Check Close Eyes
Open Eyes!
Perform a Visual Check!
Close Eyes!
Abort if Eyes not closed!
Mobile Visual Testing ExampleStep-5: Run and Check Results – First RUN (BASELINES)
Mobile Visual Testing ExampleStep-5: Run and Check Results – PASSED RESULTS
Mobile Visual Testing ExampleStep-5: Run and Check Results – FAILED RESULTS
GALEN FRAMEWORK
What is Galen Framework?Automated LAYOUT and RESPONSIVE testing framework.
• Runs in Selenium Grid (SauceLabs & BrowserStack)
• Supports Parallel Testing
Programming Languages
• Basic Syntax
• JavaScript
• JAVA & Selenium Webdriver & TestNG/Junit
Galen Reporting• Error Reporting
Generates HTML reports where you can see all your test objects
• ScreenshotsHighlights the misaligned elements
• Image ComparisonCompares images and shows differences
Galen Example with JAVA & Webdriver & TestNG
Logo: 31 pixel below from the
header
Header: 0px to the right, left, top
Navigation Links: Horizontally aligned.
Logo should displayed correctly.
Galen Example with JAVA & Webdriver & TestNGStep-1: Installation with Maven Dependency
<dependency> <groupId>com.galenframework</groupId> <artifactId>galen-java-support</artifactId> <version>2.3.2</version> </dependency>
Galen Example with JAVA & Webdriver & TestNGStep-2: Write a Galen Specifications (.gspec)
web elementsfor desktop
Declaring locations
tagging
Image Comparison
loops
5
Galen Example with JAVA & Webdriver & TestNGStep-3: Write your Test Code
Get Layout Report
Get Galen Test Information from
Layout Report
Generate HTML Report based on Galen Test Information
Galen Example with JAVA & Webdriver & TestNGStep-4: Folder Structure
Galen Example with JAVA & Webdriver & TestNGStep-5: Run the Test and Check The Results
Galen Test Report
Galen Example with JAVA & Webdriver & TestNGHomepage Layout Report
Galen Example with JAVA & Webdriver & TestNGLAYOUT ERRORS
Galen Example with JAVA & Webdriver & TestNGIMAGE COMPARISON ERRORS
What is Percy?Visual regression testing and review platform.
• COMPARES images, FINDS differences.
• We can REVIEW and APPROVE changes.
• Supports RESPONSIVE Design testing.
• Prevents FALSE-POSITIVES.
• PARALLEL TESTING support.
• Integrated with GitHub and many CI tools.
Which Technologies Does it Support?CI Tool Integrations Languages/ClientsTravis CI EmberCircleCI Ruby – CapybaraSemaphore Python – Selenium (BETA)Codeship Static websitesBuildkite Drone Jenkins
How Does it Work?
When commit any code changes.
Trigger Jenkins by git push or Pull Request then it starts to build the project and run the tests
Percy takes DOM screenshots and send them to the Percy Server
For 1st Run: Percy creates BASELINE IMAGES (No Comparison)
For 2nd + Runs: Percy compares Actual & Baseline images and notifies us.
At last step, we review the results. (Approve/Not Approve)
Installation Ember PercyRequirements for Ember Percy with Jenkins and GitHub
git node.js ember.js ember-percy
Phantom.js bower Jenkins GitHub
Percy Installation Steps (Percy Side)• Go to Percy.io and click Sign In WITH GITHUB.
• Go to Percy Dashboard and Create an Organization
• Install GitHub Integration
• Create a project and get your CI integration variables:
PERCY_TOKEN & PERCY_PROJECT
Percy Installation Steps (Jenkins Side)• Set your PERCY_TOKEN and PERCY_PROJECT environment variables
• Do GitHub Integration settings and Create a Jenkins Job.
• Do your Ember Project’s Repository settings.
• Add build and test Windows Batch commands:
• For BUILD:
• For TEST:
npm install && bower install
npm test
How to Write Test with Ember-Percy?
percySnapshot(name, options);
scope For element to snapshots. Default is full page.
percySnapshot('homepage', {scope: '#header'});
Takes ScreenShots
Sample Test Code Official TODO Example of Percy
Navigate the homepage
Take ScreenShot
Write sth to the bar and Press Enter Assertions
Take Final ScreenShot
Responsive Design Test Code
Define Resolutions by Screen Widths
First Run No Comparison (BASELINES)
Second Run PASSED!
Third Run FAILED!
Visual Test Automation by UsingImageMagick & Selenium & AShot
What is ImageMagick?• Open-source software suite that manipulates
images in a variety of formats (over 200) including PNG, JPEG, GIF, etc.
• Edits, Converts, Resizes, Compares, Rotates, Adjusts Colors etc.
• For JAVA it has Im4java which is a pure-java interface.
What is AShot?• Yandex's screenshot utility which takes, crops, prettifies and
compares screenshots.
• Entire Page
• Web Element
Example Test Scenario (Basic CSS Animation Check)
• Open www.kariyer.net
• Unhide text area of Uzman Photo
• Hover on the “UZMAN” weblement on main page.
• Wait 2 seconds to finish CSS animation
• Take screenshot of an element and save as
Baseline for the 1st Run.
• For 2nd and more runs COMPARE baseline image
with actual image.
• Put all differences in "Differences folder"
HOVER
BASELINE
Example Test Scenario (Basic CSS Animation Check)• Step-1: Install ImageMagick with all options and add to system path.
Example Test Scenario (Basic CSS Animation Check)• Step-2: Create a Selenium & TestNG project and add dependencies
Example Test Scenario (Basic CSS Animation Check)Test Method’s Code
AShot
ImageMagick
JavascriptExecutor
Example Test Scenario (Basic CSS Animation Check)How to Take Screenshot with AShot
Example Test Scenario (Basic CSS Animation Check)If Baseline Exists, Do Comparison!
ImageMagick Comparison
ImageMagickConfigurations
Add images to ImageMagick Operation object for comparison
Do Comparison!
First Run
Test ScreenShot Folder is created by using Test Name
Third Run!Arithmetic Error
Number
Diffe
renc
es F
olde
r
Test
Fol
der
Test
Fai
led!
