Upload
vikram-panchal
View
36
Download
0
Tags:
Embed Size (px)
Citation preview
Mobile Handset and Mobile App Testing
Session for Test Management CoPSession for Test Management CoP
Date – 22 Sep 2011
The contents of this deck and the session conducted for TMCoP leverages
• iGATE Patni’s past experiences in this space
• Research references from Industry Research firms
• Individual opinion and research conducted on this subject over a period of time
This is not a snap-shot of iGATE Patni’s capability in Mobile Testing
For any comments and queries regarding this presentation please contact
Disclaimer
Proprietary and ConfidentialSeptember 27, 2011 - 2 -
For any comments and queries regarding this presentation please contact
Mobile Telephony Trends� More than 60 % of world’s population today has access to a Mobile Phone!
� Ratio of Feature phone to SmartPhone users was 80:20 ( Q4 2009 )
� Projected to become 70:30 by Q4 2011 world-wide- The Nielsen Company
Proprietary and ConfidentialSeptember 27, 2011 - 3 -
Nokia, 453
Samsung, 280.2
LG, 116.7
RIM, 48.8
Apple, 47.5
Others, 413.8
Worldwide 2011 Q2 Smartphone sales by Operating System - Gartner
Market Share Handset Makers ( mn units 2010) : Strategy Analytics 2011
Sybase –Afaria/
Unwired
IBM DB2
EveryplaceRhomobile
Broadbeam/
MobileawareMotorola
SFA/FFA Distribution/RetailManagement
Dashboard
Patient
Monitoring
LegacyEnterprise
Systems
Applications
Middleware/
Platforms
LAN, PSTN, DSL, GSM, GPRS, CDMA, CDPD, UMTS, WLL, WiFi, Satellite
Blackberry
Enterprise Server
HIS Supply Chain Financial
Sierra/
Jasper/
Telenor
Credit Card/Claims
Mobility Landscape
Proprietary and ConfidentialSeptember 27, 2011 - 4 -
PDA Smart-
phonesMobile
phones
Rugged
Terminal
POS/
Tablets
Laptop
RFID
readerBarcode
Scanner Camera
GPS
ReceiverPrinter Sensors
RS232, USB , Bluetooth, Zigbee, UWB, HomeRF, IrDA
OS/ Platforms
Mobile Devices
Peripherals
LAN, PSTN, DSL, GSM, GPRS, CDMA, CDPD, UMTS, WLL, WiFi, Satellite
Reference – iGP Mobility Deck
Mobility Challenges
• Real-time, Off-line, Near-real-time
• Security, Authentication, Authorization
• Technology Convergence
• Provisioning and device management
• Scalability
• Services – Messaging, location
based, notification
Application Challenges
• Commercial grade apps
Complexity of
• Applications
Enterprise
Systems
Applications
Middleware
Mobility Landscape and Challenges
Proprietary and ConfidentialSeptember 27, 2011 - 5 -
• Commercial grade apps
• Content Aggregation
• Customizations and personalization
• Multi-device support
• Multi-lingual support
• Application categories
Device Challenges
• Device capabilities
• Programming models
• No dominant standard
• Wide range of target
environments
• Range of devices
• Connectivity options
• Technology options
• Vendors and
• Platforms
OS
Mobile Devices
Peripherals
Reference – iGP Mobility Deck
Handset / Mobile Device
Test Handset from OEM
Native Applications
Web Based Applications
Third Party Applications
Mobile Handset Testing /
Device QualificationMobile Handset App / Mobile WAP Testing
Mobile Testing Landscape
Proprietary and ConfidentialSeptember 27, 2011 - 6 -
Test Handset from OEM
Test Branded Pre-Released Handsets
supplied by Telecom Operators
Examples
New / Upgrade of Handset from OEM
directly
Co-branded Handset with Telecom Operator
Test Apps that reside on the device and may
or may not require network connectivity
Examples
Messaging,
Multimedia, Games,
PIM etc
Test Applications that are accessible through
mobile browsers
Examples
Mobile Banking
Payment solutions
Test Third party Apps that are specifically
installed on the mobile handset
Examples
Facebook, Twitter
• User Interface
• Branding
• Mobile Phone Controls
• Mobile Handset Applications
• Partner Applications
• Multimedia and Graphics
• User Interface
• Branding
• Mobile Phone Controls
• Mobile Handset Applications
• Partner Applications
• Multimedia and Graphics
Types of Testing
• Functional• Regression • Integration • Automation
Mobile Handset Testing
Proprietary and ConfidentialSeptember 27, 2011 - 7 -
• Multimedia and Graphics
• Integration with other systems
• Multimedia and Graphics
• Integration with other systems
Focus Areas
• Automation •Multi-lingual • Usability• Field / Coverage• Quality of Service• Performance Testing• Security Testing
The customer in this case could be a Telecom Operator or an OEM for e.g. Nokia
Please note – Types of testing highlighted in Italics are not offered as a service in this space by iGATEPatni today
1. User Interface and Branding : Verifying menus, navigations and the soft keys etc. are as per the
specifications given in the UI specification document.
2. Mobile phone controls : Verify that all the controls or buttons on the handset are mapped correctly
for instance pressing a dedicated camera key should launch the camera application only and not
any other application.
3. Mobile Handset application : Verifying the functional correctness of all the applications on the
handset for e.g. verifying that the alarm on the handset can be set and that it goes off at the given
Mobile Handset Testing – Focus Areas
Proprietary and ConfidentialSeptember 27, 2011 - 8 -
handset for e.g. verifying that the alarm on the handset can be set and that it goes off at the given
time.
4. Partner applications : This test is to verify whether all the 3rd party applications present on the
handset work properly an example of this could be verifying that the 3rd party application like MSN
messenger is launching properly and is working as needed.
5. Multimedia and graphics : This test is to verify whether the graphics on the handset i.e. wallpapers
, screen savers, themes (both preloaded and downloaded) appear properly.
6. Integration with other systems : if required
Network and Non-Network Testing
Mobile Handset Testing
Requires Network
Doesn’t Require Network
DATA
• WAP | SMS/MMS | DOWNLOAD CONTENTS
• DOWNLOAD GAMES
• CONTACTS
• TOOLS (calculator,voice memo,stopwatch,etc.)
• SETTTINGS
Proprietary and ConfidentialSeptember 27, 2011 - 9 -
• DOWNLOAD GAMES
• WAP deck and 3rd party apps
• INSTANT MESSAGING (like yahoo,msn.google)
• EMAIL(3rd part email apps)
CALLS
• MAKE/RECEIVE CALLS | VOICE MAIL | SPECIAL NUMBERS
• RECENT CALL LISTS | INTER CARRIER SWITCH
• ALARMS AND SCHEDULER APPS.
• RECEIVING VARIOUS ALERTS (low balance, promo alerts etc.)
• SETTTINGS
• CAMERA
• PRE LOADED CONTENTS
• GAMES
• HELP & INFORMATION CONTENT
• MEMORY
• BLUETOOTH
• MEDIA PLAYER (audio and video)
• LANGUAGE UI (in cases of another language being
present like Spanish)
Sample Handset Specification
Proprietary and ConfidentialSeptember 27, 2011 - 10 -
Reference – Collateral from past engagement
Types of Testing
• Functional
• Compatibility
• Automation
Mobile App Testing
Proprietary and ConfidentialSeptember 27, 2011 - 11 -
Mobile
App Testing
• Automation
• Regression
• Performance
• Security
• Usability
The customer in this case could be an Enterprise OR a content Provider or an App/WAP provider
Please note – Types of testing highlighted in Italics are not offered as a service in this space by iGATEPatni today
Popular Testing Tools used in this space
Commercial
• Device Anywhere
• Perfecto Mobile
• Test Quest
Open Source
• Robotium (Android Apps)
• FoneMonkey
Proprietary and ConfidentialSeptember 27, 2011 - 12 -
• Test Quest
• Squish
• Jamo Solutions
• Eggplant
• FoneMonkey (iphone Apps)
• TestMaker
• T-Plan Robot
Please note – This is only a sample list of tools. This by no means is a list of most recommended tools by iGATEPatni in this space
Parameters Device Anywhere TestQuest Perfecto Mobile Eggplant Robotium
Device and Platform
Support
All Leading Handset OEM s
Android, Blackberry,
Windows, Symbian, iOS,
Win 7, WebOS
Android,
Blackberry, BREW,
UTS, Windows
Android, Blackberry,
Nokia, iOS,
Android, iphone,
Symbian, Windows
Blackberry
Android
Scripting Drag drop functionality
Ability to script in Java
No Scripting Yes Yes Yes
Functional Testing Yes Yes Yes Yes Yes
Automation Testing Yes Yes Yes Yes Yes
Feature Set comparison of Popular Tools
Proprietary and ConfidentialSeptember 27, 2011 - 13 -
Automation Testing Yes Yes Yes Yes Yes
Performance Testing No Yes No No No
Test Management Built-in Test Case Manager Built-in Test Mgr No No No
Remote Connectivity Yes Yes Yes Yes No
Integration with
industry std. tools
HP QTP and QC
IBM Rational Quality
Not available HP QTP and QC No No
Licensing Model Subscription based SaaS*
[ 12 mth commitment ]
Cost Subscription based
SaaS*
Cost Opensource
* Each package is a collection of devices with some common characteristics such as network (e.g. Verizon Package), manufacturer (Samsung Package), operating system (Android package) etc.
Reference – Collateral from Individual Websites of respective toolset vendors | These are some of the key features ; however one needs to do a thorough POC to arrive at a recommendation
What is Remote Device Testing?
Proprietary and ConfidentialSeptember 27, 2011 - 14 -
100 s of “real” devices / phones connected to live
network are “installed” in their Data centre available
for use based on a subscription package (SaaS)
Through the DeviceAnywhere Test Centre a tester can
remotely connect to these devices / phones via an https
web based connection in real time and can remotely
perform any action..
� No need not buy all the devices/phones ( including SIM card )
� Test multiple combinations of devices / operating systems
� Leverage feature of the Testing tool to automate and test manage the whole testing process
� Conduct this testing from a remote location / offshore – thus making it further attractive
Some key benefits of device testing tools
Test Planning Test Design Test Execution Test Closure
Analyze Requirements
Design Test Cases
Setup Test
Functional,
Automated
Non-
Document Closure Report
Approach for Testing
Proprietary and ConfidentialSeptember 27, 2011 - 15 -
Document Test Strategy
Tool Selection and POC *
Setup Test Environment
Create Test Data
Non-Functional
Defect Management
Present Findings
Go Live!
using automated tools OR Emulators
using actual devices
Please note – this is a technical opinion of the author.
Key considerations for Drafting a Strategy
1. Clearly establish the scope of Testing – Functional, Automation etc
2. Agree the number and types of devices that need to be tested for example Blackberry Bold, Blackberry
Storm, iPhone etc . Do not leave any loose ends – for e.g. “ Test this on Blackberry” – which models??
3. Identify the testing toolset of choice – based on the following key criteria
• Budget for Testing
• Number of devices / operating systems / browsers ( including versions of each ) to be tested
• Duration of the programme
• Results of Technical Proof of Concept
4. If certifying a common functionality / WAP / APP across devices account for Regression testing of defects
Proprietary and ConfidentialSeptember 27, 2011 - 16 -
4. If certifying a common functionality / WAP / APP across devices account for Regression testing of defects
across the devices/operating systems/browsers , else you will run out of time during re-tests!
5. Testing report should highlight the features that are working / not working across devices
6. Conduct automation post a thorough ROI calculation only
7. Some tools do not integrate with traditional Test Management and OR defect management tools –
therefore account for the overhead ( effort and cost ) of managing test cases / defects in a different toolset
8. Defect Management – Prioritize defects upfront with the Dev/Vendors and Business
9. In third party testing scenario ( OEM ) account for additional rounds of discussions for defect triage
Please note – this is a technical opinion of the author.
Activations
Port In/Out
Help/Self Care
Security
Change MIN/ESN
Top Ups
Register CC
Adhoc Top Up
VMU Web Portal
Internet
Sales
Handset
Plans
Services
Phones
Contact Center/ IVR
“Virgin Xtras”
Ringtones
Wallpapers
Handset
Virgin Mobile USA Customers
Virgin Mobile USA – Handset Testing
• First MVNO in US and UK
• Customer base ~ 5 M
• “Pay as you go” mobile
services and handsets
• Uses Sprint’s network
• Target segment 15-30 yr old
Proprietary and ConfidentialSeptember 27, 2011 - 18 -
CRM Billing CDR
Distribution
Center
Credit Card
TransactionsTelcordia
Gateway
PromosPromos Paypal Top Up
Auto Top Up
VirginXtras Mobile Web
Messaging
Games
3rd Party Content
• Target segment 15-30 yr old
• Partnered with iGATEPatni
for QA and UAT of the IT stack
• Handset, IVR, Web-
Portal, CRM, DW and Billing
platforms
• 5 year relationship
• 52 person strong team in
onshore / off
Areas of Testing
Handset Testing
Handset
� Functionality
� User Interface
� Branding
� Mobile Phone controls
� Partner Applications
WAP
� WAP Enabled Application
� Third Party Applications
� Third Party Content
Integration
� Integration with
Mediation Managers
Billing system
Customer Care system
Proprietary and ConfidentialSeptember 27, 2011 - 19 -
� Partner Applications
� Multimedia and Graphics
Functional Testing | Regression Testing | End to End Testing |
Compatibility Testing
VMU
KYOCERANOKIA UT STARCOMM
HANDSETUNDERSTANDING
DOCSTEST CASE PREP
REQUIREMENTS
UI SPEC &
HANDSET
Testing Process and Flow
Proprietary and ConfidentialSeptember 27, 2011 - 20 -
END CUTOMER
VMU
FLASHING
TESTINGBUG REPORTING
NEW BUILD
PATNI
UNSTABLE
STABLE
HANDSET
HANDSET,UI SPEC ,HRD
Offshore
HANDSET PORTFOLIO
WDIG
GATEWAY
P100 P205
Handset Testing
Proprietary and ConfidentialSeptember 27, 2011 - 21 -
Onsite
DATA Platform
Telcordia SCH110LG200 STIC LAB
�Voice
�Contacts
�SMS
�MMS
�Browsing
�Settings
�Calendar
�Alarm
�Camera functionalities
�Hands free Calling
�Content Download
�Family Monitor
�Family Alerts
�Family Locator
�Photo Upload and
Download