-
COLLABORATIVE AND STRATEGIC
INTEGRATION OF HP ALM, HP UFT,
AND HP MOBILE CENTER
Dean CarvinDirector of Professional ServicesCheckpoint
TechnologiesEmail: [email protected]
Vivit Phoenix EventSeptember 16, 2015
-
Agenda
Introduction
Challenges with Mobile Testing
Mobile Testing Approaches
Mobile Test Planning Strategies
Strategic Solutions
HP ALM (Tips & techniques including Sprinter & BPT)
HP UFT
Mobile Labs
Mobile Center
Integrations with UFT, LoadRunner, Network Virtualization &
AppPulse
Mobile application test automation
-
About Checkpoint Technologies.
Incorporated in January, 2003
QA and QC in expertise focused on functional, performance
and
application security validation
HP Software Gold Partner, HP Authorized Software Support
Partner, &
HP Certified Training Partner
Mobile Labs Partner
Turnkey Partner
Zephyr Partner
QAI Training Partner
Expert Services
SoftwareSolutions
HardwareSolutions
IT Performance
-
Discussion
Stage 4: Mobile Everything
Stage 3: Mobile First
Stage 2: Mobile Something
Stage 1: What is Mobile?
Where is your organization?
-
The Challenges of
Mobile Application Testing
-
Discussion
When it comes to mobile application testing what are
your organizations biggest challenges?
-
Why is mobile app testing a challenge?
1World Quality Report 2012-13, September
2012
Enterprises surveyed1 in nine
industries reported top
problems:
1. Do not have the right tools.
2. Do not have the right
devices.
Simply adding an automation
tool to an existing framework
cannot solve both problems.
For many
customers,
solving #1
and #2
may
address
remaining
problems.
-
Emulation vs. Simulation
Emulation Complex application that emulates real
mobile device software, hardware and OS, allowing to test and
debug applications
System that behaves exactly likesomething else
Complete replication of system. Operates in different
environment
Abides by all the rules of system being emulated. Rules are
fixed - cannot be changed or the system fails
Simulation Less complex application that simulates
internal behavior of device, but does not emulate hardware and
does not work over the real OS
System that behaves similar tosomething else
Implemented in entirely different way. Provides basic behavior
of a system
May not necessarily abide by all rules of system being
simulated.
Ex: Flight simulator. Disconnected from reality of flying plane
and you can bend or break those rules. E.g.; Fly upside down
without breaking airplane
-
A closer look at the challenges
One Shared Set of Challenges
Having the right devicesFinding devices
Needing too many devicesControlling device content
Security exposuresMany different devices, OSs
Purchasing restrictions Overnight delivery
Tester efficiency App quality
ManualTesting
AutomatedTesting
-
Mobile Test Devices
Security
Physical security
Device integrity
Virtual access
Visibility
Device type
OS version
Availability
Process
BYOD
Share assets
Device support
-
Mobile Testers ( . . . Developers)
Security
Who can login
What can they see
What can they do
Visibility
Who did login
What did they use
How did they use it
Process
Administration
Onshore/offshore
Segment users
-
Mobile Apps
Security
Application
Application data
Who can access the app
Visibility
App version (post release vs. pre-release)
OS version (Android vs. iOS)
Who is using the app
Process
Where its built
How its built
Version support
-
Mobile App Testing
Security
Who is testing
What are they testing
Where are they testing
Visibility
How was it tested
When was it tested
For how long
Process
Test strategy (CI, TDD, BDD, etc.)
Testing tools
Manual or Automated
-
Easy to be Overwhelmed
As of early 2014 there were
2,582 device types running
106 OS versions services by
691 carriers worldwide
If goal is to test every permutation possible then
every test case will be executed 189,121,172times!
* Data compiled by mobile metrics vendor Crittercism
-
Discussion
Given following mobile phone brands in which order (if at all)
would you prioritize test plan?
Apple
RIM (Blackberry)
HTC
Sony Ericsson
Nokia
Samsung
-
Popular Phone Manufacturers Worldwide
Samsung Apple Nokia Other
Wordwide 26 21 21 32
0
5
10
15
20
25
30
35
Pe
rce
nta
ge
Android dominant OS in world Samsung most popular in 92 of 230
countries Apple and Nokia each most popular in 60+ of 230
countries
-
Most Popular Mobile Phones (by Country)
U.S.
Apple 52%
Samsung 19%
RIM 6%
HTC 5%
Nokia 2%
Canada
Apple 64%
Samsung 18%
RIM 6%
HTC 3%
China
Apple 16%
Samsung 14%
Nokia 5%
HTC 5%
Russia
Nokia 20%
Apple 17%
Samsung 17%
Sony Ericsson 5%
India
Nokia 44%
Samsung 27%
Apple 1%
-
Developing a Strategic
Test Plan
-
The BIG Picture
Testing effort is a project! Regardless of challengesor
perhaps better to say that especially due to
challengestest planning is critical!
It requires a strategic approach.
-
The Mobile EcosystemServices
Applications
App Frameworks
Platform/OS
Devices
Manufacturers
Networks
Operators
Explores trends, orgs, companies, and stats at center of mobile
landscape.
-
Types of Mobile Applications
Web Apps
Not real applications - really websites
Run by a browser
Typically written in HTML5, CSS3, JavaScript, etc.
Native Apps
Installed on device through app store
Accessed through icons on the device home screen
Developed specifically for one platform or multiple
platforms
Can take full advantage of all the device features
Hybrid Apps
Part native apps, part web apps
Like native apps, obtain from an app store and can take
advantage of the many device features available
Like web apps, rely on HTML being rendered in a browser, with
the caveat that the browser is embedded within the app
-
Test Plan Structure
Will you develop a plan based on...
Agile approach?
Requirements?
Risk Analysis?
Mobile Application Demographics?
Mobile Ecosystem?
Other?
-
Test Prioritization
How would you prioritize what to test?
Depends on your business and application
What does it do?
What is it used for?
Whats demographic of users?
Geographical location
Age range
Income range
What are your testing objectives?
Risk analysis
Impact
Likelihood
-
Risk Analysis - Impact
Cost to business if realized
Saturation & frequency of use (Think total cost)
Consider adding weight factor
If functionality with high impact score is seldom used then
assigned lower
weight factor
Requires in-depth knowledge of business
-
Risk Analysis - Likelihood
New technology
New functionality
Complexity of design and development
Complexity of use
Legacy data
Has it been a problem area in the past?
Has it been more of a problem for specific
Platform/OS
Device
Manufacturer
Networks
Providers
App framework
-
Types of Testing
Install/uninstall
Functional/Performance/Security
Interrupt
User acceptance
Compatibility
Service
Many others
-
Strategic Solutions
-
Tools On Hand Strategic Business Value
HP Application Lifecycle Management (ALM)
Requirement & Test management
Defect tracking
Test execution (manual & automated)
Reporting
HP Sprinter
Interactive testing
Business Process Testing (BPT)
Component-based manual and automated testing
-
Tools On Hand - Strategic Business Value (cont)
HP Unified Functional Test (UFT)
Functional and API test automation
HP LoadRunner
Performance and load testing
HP Mobile Center (More on this later)
Mobile device management and real device testing
Mobile Labs
Private cloud for mobile testing
-
HP Application Lifecycle Management
Simplify processes, informed decisions, collaboration,
optimization
Cross-project sharing of data/metrics
Reporting and Share Analysis Items
Create graphs/reports based on your
business needs
Increased efficiencies around test
planning/execution
(HP Sprinter)
Asset sharing & reuse
Cross- project standardization
(Templates)
Linking and Traceability
Requirements, Tests, Defects
PMO
Biz AnalystQA Manager
DevelopmentDirector
-
ALM QUICK TIPS & TECHNIQUES
Useful user-defined fields to track metrics for test
execution:
Device Manufacturer
Device Model
OS
OS Version
Test executed on:
Emulator
Simulator
Real device
Defects
Use List Customization to point to a secondary different Device
Model list
based on the Device Manufacturer selected then the OS
HP ALM can be should be customized and tailored to suit your
strategy
-
The next generation of interractive testing
HP SPRINTER (VERSION 12)
Accelerate manual mobile testing and improved
testing accuracy
Better communication of defects to development
(accelerated defect remediation)
Unobtrusive, easy to use user interface
Supports exploratory testing
Reduce time & resources by testing multiple
environments
Key Benefits
Key Capabilities
Toolkit Screen capture, annotations, measure objects
Record user actions to work with exploratory testing and create
new test cases
Data injection and Mirror testing
Scanners and Storyboard Export
Log defects with associated information directly into HP Quality
Center/ALM
-
BPT provides the framework to design and develop reusable test
components and flows based on business process models
Non-technical subject matter experts can
quickly build, data-drive, and document
tests
Execute BPT tests with HP Sprinter or
UFT
Centralize test maintenance so application
changes are automatically propagated
through test assets
HP BUSINESS PROCESS TESTING (BPT)
Key Benefits
Centrally define a set of test environments and data
configurations to be
used in both manual and automated business process tests
Create input parameters and iteratively insert corresponding
data into actual
test steps during execution
-
HP UNIFIED FUNCTIONAL TEST (UFT)
A unified test development environment
One application creates both GUI and API tests
Modern IDE
True IntelliSense support
Open and edit multiple tests at once
Manage test flows visually
Mobile Support
Script once One script can be created and
automatically executed on all different mobile
platforms, devices and browsers.
Employs a hybrid object approach where both
native controls and visual analysis-based controls
are supported.
A Modern Test Development Experience
-
Secure, cloud-based testing for mobile applications
MOBILE LABS
Key Benefits
deviceConnect A private mobile device cloud
that enables the secure management and
accessibility of mobile test devices, mobile
applications and test data
Mobile Labs Trust deviceConnects
automation engine, extends HPs UFT to real
mobile devices, providing mobile application
test automation utilizing the same skills testers
use for Web/desktop testing
-
Provides a completely secure approach to device management,
access and testing using a private cloud
Allows testers located anywhere within an enterprises
worldwide,
private infrastructure to easily share pristine mobile devices
for
manual or automated mobile app, website and device testing
Improves tester productivity by providing 24/7 remote access to
the
entire fleet of testing devices
Minimizes the complexity of managing and tracking mobile
devices,
multiple mobile platforms and operating systems
Reduces time wasted on updating devices or search for lost
devices
Requires minimum space, power, expertise or IT oversight
Supports enterprise Bring Your Own Device (BYOD) policies
MOBILE LAB DEVICECONNECT
Key Benefits
-
Provides a completely secure approach to device management,
access and testing
Certified add-in to HPs UFT
Utilized native object-based scripting so testers can interact
with individual native mobile
object, their properties and methods
Test directly on mobile devices and does not require the use of
emulators or simulators
For current QTP/UFT users, no additional training required
Execute the same script on multiple mobile platforms with little
to no adjustment to the
script
MOBILE LAB TRUST
Key Benefits
-
Mobile Test AutomationHPs Unified Functional Testing and HPs
Mobile Center
-
HP Mobile Center Log In
HP MOBILE CENTER
-
MOBILE CENTER APPLICATION MANAGEMENT
Application management:
Filter by OS support
Upload new apps
Remove apps
View App details
-
MOBILE CENTER - DEVICE MANAGEMENT
Device Management:
OS and status filtering
Device remove access
Device properties
-
MOBILE CENTER USER MANAGEMENT
User Management:
add and remove users
assign passwords
-
Mobile Center Options Side Bar
HP MOBILE CENTER
Return
Home
Screen Rotation
Multiple Screens
Phone Call Text Message
MavigationReset
-
UFT INTEGRATION WITH HP MOBILE CENTER
Include the Mobile add-in
The automation engineer supplies the server address for the
Mobile Center server, either by IP address or host name.
The engineers Mobile Center username and password are also
required
This information is then stored for all future tests or until it
is changed
-
Prompts you to select the application and the device the script
will be recorded
against.
It also provides control options when execution the script.
HP UNIFIED FUNCTIONAL TEST & MOBILE CENTER
The Record and Run Settings
-
The Record and Run Settings also allows you to select the
metrics you want to track
during execution.
HP UNIFIED FUNCTIONAL TEST & MOBILE CENTER
-
Clicking the Record button opens a browser window and displays
the device.
HP UNIFIED FUNCTIONAL TEST & MOBILE CENTER
Recording a test
-
A sample recording...
HP UNIFIED FUNCTIONAL TEST & MOBILE CENTER
Device("Device").InsightObject("InsightObject").ClickDevice("Device").App("Expensify").MobileObject("Expense").Tap
Device("Device").App("Expensify").MobileEdit("amount").Set
DataTable("Amount",
dtGlobalSheet)Device("Device").App("Expensify").MobileEdit("merchant").Set
DataTable("Merchant",
dtGlobalSheet)Device("Device").App("Expensify").MobileButton("2015-05-29").TapDevice("Device").App("Expensify").MobileButton("Done").Tap
If Device("Device").App("Expensify").MobileLabel("More
Details").Exist(1)
ThenDevice("Device").App("Expensify").MobileLabel("More
Details").Tap
End If
Device("Device").App("Expensify").MobileLabel("Category_2").MakeVisibleDevice("Device").App("Expensify").MobileLabel("Category_2").Tap
Device("Device").App("Expensify").MobileObject("Category
Select").Tap
If
Device("Device").App("Expensify").MobileLabel("Category").Exist(1)
ThenDevice("Device").App("Expensify").MobileLabel("Category").MakeVisibleDevice("Device").App("Expensify").MobileLabel("Category").Tap
End If
-
Device object options
HP UNIFIED FUNCTIONAL TEST & MOBILE CENTER
-
Mobile Object Classes
HP UNIFIED FUNCTIONAL TEST & MOBILE CENTER
object.LongPress [x], [y], [Duration], [NumOfTouchPoints],
[Gesture]
-
SPRINTER INTEGRATION WITH HP MOBILE CENTER
Sprinter is now included with Mobile Center as a stand alone
installation.
Versions 12.20 or 12.21
Supports exploratory and interactive testing
-
ADDITIONAL INTEGRATIONS WITH HP MOBILE CENTER
HP LoadRunner
Create and execute performance tests using the standard
capabilities of LoadRunner and Performance Center
The mobile device serves as additional single users used to
measure the transactions from a real device perspective
HP Network Virtualization (NV)
HP Mobile Center installation provides 2 Network Virtualization
flows.
HP NV emulates production environments using real-world network
conditions profiling mobile technologies such as 3G, 4G or WiFi;
carriers such as AT&T and Verizon; combined with either
business hours or off hours.
HP AppPulse
Continually measure user experience from the users
perspective
-
USING UFT WITH NV
-
USING UFT WITH NV
-
USING UFT WITH APPPULSE
Measure and monitor everything the user does and everywhere
the
user goes within the app
Identify issues having the most impact on the user such as
crashes:
-
USING UFT WITH APPPULSE
Monitor Battery usage:
-
Summary
Challenges of mobile application testing in managing:
Devices, user, apps and (overall) testing
Easy to be overwhelmed
Emulation vs. Simulation
Chaotic test plan
Mobile test planning strategies. Design based upon:
Risk analysis
Mobile Ecosystem
Mobile application test automation
Mobile device management
Internal
Public cloud
Private cloud
-
Summary
Strategic solutions:
Application Lifecycle Management (with BTP)
HP Mobile Center integrated with:
Unified Functional Test (UFT) for GUI and API testing
Sprinter for enhanced manual interactive testing
LoadRunner for performance testing
Network Virtualization (NV) to emulated mobile network
environments
AppPulse for mobile application real-time monitoring and
reporting
-
Thank you!
Dean [email protected] (814) 758-5959 O (813)
818-8324 ext. 129
mailto:[email protected]
