Upload
leebarnes7
View
353
Download
2
Embed Size (px)
DESCRIPTION
The explosion of multi-platform mobile application development has created an extensive testing burden for quality organizations – and a huge opportunity for test automation ROI. Factoring in the idiosyncrasies of individual device models and carriers magnifies both the challenge and opportunity. Attend this session to learn how to extend test automation to the mobile platform and realize significant ROI. You will come away with best practices for implementing test automation effectively and efficiently across multiple mobile platforms and devices utilizing various types of mobile test automation tools. Specifically, tool independent techniques for creating a single test suite capable of executing across multiple platforms and devices will be discussed.
Citation preview
© Utopia Solutions
Extending Test Automation To Mobile Applications
Lee Barnes, CTO Utopia Solutions
© Utopia Solutions
Agenda
• What is Successful MTA? • How Do We Achieve It? • Getting Started • Visual Based Tools • Object Based Tools • Evaluating Your Apps • Tool Evaluation • Summary & Questions
Mobile Test Automation (MTA)
© Utopia Solutions
SECTION 1 What Is Successful Mobile Test Automation?
© Utopia Solutions 4
WIFI / Carrier Networks
Mobile Test Automation – Goal
Automation Framework
Test Cases
Backend System
Single set of test cases executing across a diverse set of real devices
© Utopia Solutions 5
Test Automation Success Criteria
• Issue detection and recovery • Accurate verification • Unattended execution
Reliable
• Minimum sensitivity to application and test case changes
• Test case definition separate from automation code
Maintainable
• Test coverage expanded efficiently • Automated test cases created by non-
automation resources Scalable
© Utopia Solutions
SECTION 2 How Do We Achieve These Goals?
© Utopia Solutions 7
Test Automation Approach
Mobile Test Automation Framework
Automated Test Cases Test Results
Test Definition
Test Execution
Non-technical / Test Analysts & Subject Matter Experts
Technical / Automation Engineers
© Utopia Solutions
Mobile Test Automation Framework
Business / Application Testing Components Login
GoTo Stocks GoTo News
Add Stock Edit Stock
Remove Stock
Logout Get Position
Verify Position
…
… Object Interaction Methods
Object Exists Set Value
Get Value
Select Item Touch
Touch and Hold
Get Image Get Text
Send Key
Test Case
Login GoTo Stocks Add Stock Verify Position Logout
Mob
ile A
utom
atio
n Fr
amew
ork
© Utopia Solutions 9
Test Automation Challenges
• Defining appropriate scope • Integrating with SDLC • Addressing technical issues • Reliability, maintainability, scalability
General
• Platform diversity • Device diversity • Tool diversity • Tool immaturity • Lack of skilled resources • Rate of technology change • Rapid development cycles
Mobile Specific
© Utopia Solutions
SECTION 3 Let’s Get Started
© Utopia Solutions 11
Mobile Test Automation Tool Categories
Native Platform Frameworks
Multi-Platform Tools Visual Based Object Based
• Provided by mobile platform vendor as part of SDK
• Typically interact with the application at the UI object level
• Interacts with devices using visual methods (text and image recognition)
• Interacts directly with application UI objects using native methods and properties
• Object based interaction with app UI controls
• Greatest degree of support for native UI objects
• Control of device settings
• Supports multiple platforms
• Ability to execute tests across multiple devices
• Interact with entire device
• Supports multiple platforms • Ability to execute tests
across multiple platforms • Interacts with apps at the
object level for greater reliability
• Limited to a single platform • Requires code level access
to the app • May be limited to simulator
(e.g. Blackberry)
• Generally less reliable • No access to object
properties • May require jail-breaking /
rooting
• Interaction is limited to app under test
• Requires app to be instrumented
• Android UI Automator • iOS UI Automation
• SeeTest (Experitest) • eggPlant (TestPlant) • Perfecto Mobile • Zap-Fix (Zap Tech.)
• Trust (Mobile Labs) • TouchTest (SOASTA) • M-eux Test (Jamo Solutions)
Des
crip
tion
Pros
C
ons
Exam
ple
© Utopia Solutions 12
Start With a Plan
Scope
OS/Device Selection
Test Case Selection
Non-Testing Tasks
Technical Challenges
Platform/Device Diversity
Object Identification
Process Integration
Test Suite Maintenance
Continuous Integration
© Utopia Solutions 13
Automation Scope Selection
• Frequently tested • Predictable results • Easy to automate • Manually tedious
“Traditional” rules still apply
• Tests executed across all target devices • Avoid apps and tests that face many mobile
automation challenges (to be discussed)
Mobile specific
© Utopia Solutions
SECTION 4 Visual-based Tool Approach
© Utopia Solutions 15
Visual Based Identification
Label
Label + Boundary
Image Recognition
Label Offset
Multi-Label Offset
© Utopia Solutions 16
Cross Platform Issues – Inconsistent UI/UX
Button Image/Position
Screen Titles
Button Labels
© Utopia Solutions 17
Visual Based Techniques – Complex Interaction
iOS On/Off Switch
Function iosSwitchGetState (Label)
Define OCR parameter ranges
Search for Label on device screen
If Label not found
status = switch not found error
Else
Loop through OCR parameter range
Search for switch state(ON/OFF)
If state found
status = switch state
Else
status = state not found error
End If
End Loop
End If
Return status
End Function
Function iosSwitchSetState (Label, State)
CurrentState = iosSwitchGetState (Label)
If CurrentState = error
status = error
Else
If CurrentState = desired state
status = success
Else
If desired state = ON
Touch right side of switch
Else
Touch left side of switch
End if
End if
status = success
End if
Return status
End Function
Hidden Complexities
© Utopia Solutions 18
Visual Interaction – Reusability?
Maybe…
• Label / object offset • On / Off reversed
not
• Label offset right • No On / Off label
© Utopia Solutions 19
Object Identification - Defining Master Object Properties
Environment Object Type Attribute Value
iPhone Screen Method Unique Text iPhone Screen Label Screen name
iPhone Screen Location 20,25:40,200 iPhone Edit Label Object name
iPhone Edit Offset 0,-20
…
• Defines general rules for identifying objects by platform • Avoids adding all application objects to Object Repository • One master file per application (typically)
Master Property File
© Utopia Solutions 20
Object Identification - Define Object Map
Object Screen Platform
iOS
Login
User Name
Password
Add Stock Symbol
• Contains any screens and objects that don’t meet criteria defined in Master Property File
• Objects organized hierarchically by platform and screen
UI Object Map
© Utopia Solutions
SECTION 5 Object-based Tool Approach
© Utopia Solutions 2
© Utopia Solutions
Object Based Interaction
Tool Libraries Included in Project Build
Object Properties Visible to Tool
Class: UIButton Label: Save
Class: UITextField Label: Name
© Utopia Solutions 23
Object Based Interaction – Cross Platform Differences
Simple Interaction – “Tap” the Save/Done Button
Class: avwButton Method: Push Implementation: avwWindow(“MyStock”).avwButton(“Save”).Push
Class: iosButton Method: Tap Implementation: iosView(“EditStock”).iosButton(“Done”).Tap
© Utopia Solutions 24
Object Identification – Handling Cross Platform
Properties Class: Button Name: Save
Create Mapping of Universal Objects to Platform Specific Objects
Properties Class: avwButton Label: Save Method Name Touch: Push Object Path avwWindow(“MyStock”)
Properties Class: iosButton Label: Done Method Name Touch: Tap Object Path iosView(“EditStock”)
Step 1
© Utopia Solutions 25
Function TouchButton (Name)
Get execution platform
Retrieve platform specific object info
from Object Map
Construct statement to perform operation
Execute operation
Return operation status
End Function
Object Identification – Handling Cross Platform
Create Universal Method for All Platforms Step 2
Properties Class: Button Name: Save
Properties Class: iosButton Label: Done Method Name Touch: Tap Object Path iosView(“EditStock”)
iosView(“EditStock”).iosButton(“Done”).Tap
© Utopia Solutions
SECTION 6 Evaluating Your Apps for Potential Challenges
© Utopia Solutions 27
Analyze Business Processes for Differences
Differences • Menu always visible
on iOS • Different edit buttons • Different column
headers
© Utopia Solutions 28
Analyze Business Processes for Differences
Differences • Different screen titles • Different “add”
buttons • Different “save”
buttons • Different background
color on list
© Utopia Solutions 29
Analyze Business Processes for Differences
Differences • Cancel button
embedded in search field on iOS
• Different “expand” icons
© Utopia Solutions
SECTION 7 Handling Workflow Differences
© Utopia Solutions 31
Handling Workflow Differences – Go To Stocks
Android 1. Tap “MENU” 2. Tap “Stocks” 3. Sync on “My
Stocks” Screen
iOS 1. Tap “My Stocks” 2. Sync on “My
Stocks” Screen
Implement at the Business Process Component Level
Public Function GoToStocks Platform = GetPlatform() Select Case Platform Case “iOS” Tap (“MENU”) Tap (“Stocks”) status = WaitScreen (“My Stocks”) Case “Android” Tap (“My Stocks”) status = WaitScreen (“My Stocks”) Case Else status = INVALID_PLATFORM End Select GoToStocks = status End Function
© Utopia Solutions 32
Mobile Test Automation Tool Evaluation
Top Ten Questions to Ask Tool Vendors • Is your tool visual or object based? • Does your tool require jailbreaking or rooting? • How quickly do you support new mobile OS releases? • How strong is the company (e.g. sales, customers, years, etc.)? • How do I execute a single test across multiple platforms with
your tool? • Can (or must) your tool integrate with other scripting engines? • Do your support our mobile development platform? • Can your tool interact with devices remotely? • Does your tool integrate with our test management solution? • Will you assist me with a proof of concept?
© Utopia Solutions 33
Key Summary Points
• Set appropriate automation goals • Start with a plan
• Scope • Technical Challenges • Process Integration
• Understand how visual/object based tools work • Analyze your app for platform differences • Address differences at appropriate level in your
framework • Evaluate tools carefully • Start small and expand over time
© Utopia Solutions 34
Questions…
…. and Answers!
Direct future questions to: Lee Barnes Founder and CTO Utopia Solutions, Inc. Email: [email protected] Twitter: twitter.com/USI_LeeBarnes LinkedIn: linkedin.com/in/leebarnes Blog: www.RaiseYourMQ.com