1.By Waqar AhmadSwaam Tech (Pvt) LtdContact us: info@swaam.com for Ur App TestingNote: The slide was presented in the Software Testing Course at NUCES-FAST Lhr

2. Classification of Mobile Applications Web Applications Could be Simple Markup (WML , HTML, XHTML,XHTML MP) or AJAX Applications A Web Application may be Designed for A Single Type of Device or Multiple Different Devices with Customization for eachDevice (or each Family of Devices) Multiple Different DevicesA Transcoder/Proxy ResidingBetween Web Application Server and Device TransformsContent based on Device User-Agent Capabilities 3. Classification of Mobile Applications On-Device Applications May be Developed for Better Portability Java ME FlashLite Native Platform Applications IPhone: Objective C Symbian: C/C++ Windows Mobile: .Net CF Android : Java for Application and C/C++ for Low Level 4. Mobile App TestingChallenges Device Fragmentation is a Big Challenge Devices Vary in Screen Size, Memory, ProcessingPower, Hardware Features etc. Apple iPhone is Least Fragmented among All MobilePlatforms Testing on All Target Handset/Devices Almost Impractical if Number of Target Handsets is Large Testing on All Target Operator Networks Almost Impractical if Number of Target Operators is Large Network Operator may Impose Certain Constraints 5. Mobile App TestingChallenges Device Vendor (Not Necessarily Manufacturer) mayCustomize a Device Changes may be Done in Both User Interface and Functionality Many a Times Changes are Not Well-Documented Device is Not the Same as Released by Manufacturer Testing on All Versions of Operating System In Many Situations, it May Not be Possible to Upgrade the Device to New OS Version Same Device Models may have Multiple OS Configurationsbased on Target Market Requirements Localization Hundreds of Devices with Many Variants of Each Device 6. Mobile App TestingChallenges Testing in All Web Browsers Browsers Significantly Vary Across Devices May Support HTML, xHTML, WML, AJAX Web Content Transcoders Web Content may be Transformed to Meet User-AgentConstraints Transcoder may Not Respect User Experience Factors insome Situations 7. Mobile App TestingChallenges Inconsistencies in Specification Implementations Each Vendor is Responsible for Java ME Specifications Implementations Implementations may Not be Consistent across Devices and Vendors Performance Measurement Extremely Difficult to Benchmark Performance due to Highly Fragmented Market Detection of Display Issues Human Involvement is Must Generally it is Not Possible to Purchase All Target Handsetsdue to Financial Constraints 8. Mobile App TestingChallenges Application Usability Testing End Users may Not even be Skilled at Using a Mobile Device End User may Use Application in Multiple Contexts Driving in Rush Hour Walking in Crowded Market Multiple Input / Output Modalities Diverse User Interface Styles In Reality, A Testing Lab Cannot Simulate all PossibleEnd User Contexts 9. Mobile App TestingChallenges Development and Deployment Platforms are Different Development Platforms Windows, Linux Deployment Platforms iPhone, Symbian, Android, Blackberry, Windows Mobile Emulators Do Not Capture All the Characteristics of aDevice Multiple Design Specifications may be Required forSingle Application due to Device FragmentationRequires Test Cases Accordingly 10. Testing Techniques: General Gather Useful Data Decide Target Handsets as Early as Possible inDevelopment Lifecycle (Better in Requirements AnalysisPhase) Document Incompatibilities in Devices e.g. Differences in Installation Processes Audio/Video Capabilities Memory, Processing Power, Screen Size etc. Document Device Software (Firmware) Limitations Bugs in Firmware are Not Unusual Document Device Web Browser Constraints Supported Version of HTML AJAX Supported or Not 11. Testing Techniques: General Gather Useful Data Better Acquire Meta Data of All Target Handsets from anExternal Source Extensive Databases Containing Device Meta Data WURFLAn Open Source XML Database of Device Characteristics DeviceAtlasA Proprietary Database of Device Characteristics Some Websites can Capture and Return HTTP Headers For instance, Access the Website using a Phone, Website will ReturnHeaders in HTTP Requests Useful when Data is Not Available in Databases and also to Verify Already Available Data in Meta Data Source A Development House may also Maintain an InternalDatabase Containing Data Not Available Otherwise 12. Testing Techniques: GeneralClassify Devices Classify Devices Category1: All Devices that Support XHTML Category2: All Devices that Support AJAX Category3: All Devices that Support Java ME Pick Representative Device(s) from each Category forTesting Purposes Prepare Test Cases based on Data Gathered aboutTarget Handsets 13. Testing Techniques: GeneralPerform Testing on Real Devices Start Testing on Real Devices as Early as Possible There is No Good Alternative to a Real Device forTesting 14. Testing Techniques: GeneralEmulator is Useful but is Not a Real Device Testing on Emulators An Emulator is a Software Application that Imitates a Smartphlone A Device Emulator may be Used to Test Application on Desktop Some Emulators Emulate General Characteristics of a Platform (like Java ME Emulator) Some Emulator Emulate Relatively Faithful Representation of Specific Phones (BlackBerry Dev Kits) Emulator may or may Not Require Compilation for Target Platform Single Compiled Code for both Emulator and Target Devices Java ME Windows Mobile Different Compilers for Compilation for Symbian Emulator and Target ARMDevice Useful for Quick Testing in Development Environment Emulators are Not Real Devices Testing on Emulator does Not Guarantee Application Success on Real Device 15. Testing Techniques: General Application UI Should Preferably be Consistent with Device UI Style Test Application User Interface Consistency with Device Interface Style Is the Application User Interface Consistent with UserInterface of Underlying Device Applications? Does the Application Offer Interaction MechanismSimilar to Interaction Mechanism GenerallyImplemented in Native Device Applications? Softkey Mapping Look&feel 16. Testing Techniques: GeneralDifferent Test Cases are Required to Cater DiverseInput / Output Modalities Perform Testing for Various Input Modalities Touch Screen Testing with Single and Multiple Touch Input Testing with Virtual Keyboard Trackball/Track Wheel Numeric Keypad QWERTY Keypad 4- or 5-Way Navigator Perform Testing for Various Output Modalities Different Screen Sizes 17. Testing Techniques: GeneralTest Applications in Real User-Contexts Test the Application in Relevant User Contexts User Context Involves User, Application and Surroundings Design Various User Interaction Scenarios in Applicable Contexts (user driving in rush hour, walking, in classroom etc.) Test Applications in Real Contexts 18. Testing Techniques: General Mobile Ecosystem Requirements must be Addressed Application are Generally Launched through VariousApplication Stores (Mobile Ecosystem) Apple App Store, Nokia Ovi, Blackberry AppWorld, AndroidMarket, GetJar, Handango etc. Test Applications to Meet Requirements of variousEcosystem Stakeholders Manufacturers/Vendors UI Guidelines from Apple for Iphone Applications Symbian Signed Operators Operators may also have some Requirements Third Parties Java Verified Process 19. Testing Techniques: Web Applications Rules-Based Testing Rules-Based Testing with User-Agent Impersonation Create Testing Rules based on Device Capabilities Known Bugs in the Device Device Data in Data Sources (like WURFL, Internal Database) Application Requirements Test Cases are Written that Checks HTTP Response for TargetedContent Impersonate a Device and Communicate with Web Server Change HTTP Headers like Accept: image/gif, image/jpeg User-Agent: NokiaN95.+Profile/MIDP-2.1 + .. Check the Server Response Against Rules and IdentifyDiscrepancies, if any 20. Testing Techniques: Web Applications Rules-Based Testing Rules-Based Tests Testing is Automated for Multiple Types of User-Agents Testing can be Done Using An HTTP Library for Sending Requests and Receiving Response Web Browser (FireFox, Safari, Internet Explorer) Useful to View the Rendered Screen with some Limitations Rules Based Testing is Useful for Automated Testing of Web Applications that Deliver Content on Phone Regression Testing Web Content Accessibility 21. Testing Techniques: Web ApplicationsTranscoder Behavior must be Thoroughly Tested Transcoder Behavior Web Content may be Transformed by a Transcoder to Meet User-Agent Constraints A Transcoder Resides Between User-Agent and Web Server Cookies should Not be Shared Use Automated Tests for Communicating with Server inParallel User Experience Should Not be Affected Manual TestingSee and Record the Behavior 22. Testing Techniques: Web ApplicationsW3C mobileOK Standard W3C has Defined mobileOK Tests Tests for Checking Mobile Readiness (W3C Best Practices) of a Website http://www.w3.org/TR/mobileOK-basic10-tests/ W3C mobileOK Checker Performs mobileOK Tests on a Web Page to Determine its level of Mobile-Friendliness http://validator.w3.org/mobile/ 23. Testing Techniques: On-Device Applications Unit Testing and On-Device Debugging Unit Testing Frameworks Exist for almost AllPlatforms GUI Layer, Event Handling and System Testing is Not Possible On-Device Debugging is Available on Most of thePlatforms 24. Testing Techniques: On-Device Applications Custom Test Applications may Reveal Major Issues Custom Test Applications Can be Used to Test Critical Aspects Whether an Application has Access to Device File System Performance of Device Database System Write Applications to Diagnose Key Issues e.g., Develop an Application that Helps in Testing When Lot of Free Memory is Available on Device Very Limited Memory is Available on Device 25. Testing Techniques: On-Device ApplicationsNetwork Availability and Bandwidth Perform Application Testing in Different Network Signal Strengths No Network Intermittent Connectivity Strong Network Signals Presence Application Testing May be Required in Various Network Types 2G, 3G 26. Testing Techniques: On-Device ApplicationsExternal Events may Affect Application Behavior Test Application against Events that may Occur during Application Installation and Execution Phone Call Received SMS Received Battery Removed Device Switched Off Bluetooth Device Requested to Contact 27. Testing Techniques: On-Device ApplicationsCombine Automated Input with Manual Verification Generate Automatic Events Key Press Pointer Press Touch Capture Screenshots during Event Generation Process and Compare them 28. Testing Techniques: On-Device ApplicationsTrack Behavior when App is being Used by Consumers Track Application Behavior during Various Lifecycle Stages when in Production Installation Execution Removal Application may Send Important Data to Server that can be Used to Identify and Fix some Bugs Java ME Includes some Parameters for this Purpose GetJar Offers Such a Service Application can Call Server and Report some Information http://www.getjar.com/ 29. Testing Techniques: CommercialGet Online Services for Devices Not Available In-house Testing on Real Devices that Developers Dont Own A Service that Enables Application Testing on Real Devices Devices are Located on Remote Servers Devices are Connected to Various Operator Networks Worldwide Desktop Client Software/ Web Application Enables All Testing Features Remotely Device Acquisition, Cleanup Application Installation, Execution, Removal Device Shutdown, Battery Removal and Installation Service Providers DeviceAnywhere (http://www.deviceanywhere.com/) Over 2000 Devices Connected to over 30 Worldwide CarrierNetworks across USA, Canada, UK, France, Germany, and Spain Perfecto Mobile (http://www.perfectomobile.com/) 30. Testing Techniques: Commercial Crowdsourcing is also an Option CrowdSourcing (Crowd + Outsourcing) Testing by Human Testers (Mobsters) Located Worldwide If Done Effectively, Real Usability Testing Results can also be Obtained Testing in Real User Contexts is Possible How it Works? Developers and Testers are Registered with a Third Party ServiceProvider Third Party Service Provider is Responsible for Coordination of AllParties Involved in the Process (Application submission, monitoringtools, results reporting etc.) Service Providers Mob4Hire (http://www.mob4hire.com/) GetJar (http://www.getjar.com/) uTest (http://www.utest.com/) 31. Testing Techniques: CommercialUser Feedback is Important App Stores may Provide Useful Feedback from End Users User Reported Bugs User Reviews on Usability User Ratings Data is Helpful for Quickly Fixing some Bugs Future Application Releases 32. Testing Tools: Web Applications HttpUnit Unit Testing for Web Applications HtmlUnit Unit Testing with JavaScript Support Urllib Provides APIs for Web Access Check a Web Application for Mobile Readiness mobiReady: http://ready.mobi/launch.jsp?locale=en_EN W3C mobileOk Checker: http://validator.w3.org/mobile/ 33. Testing Tools: Web Applications User-Agent Capture Tools Captures Characteristics of User-Agent Content Type, HTML version Supported on Device etc. Pycopia: http://www.pycopia.net/webtools/headers FireFox and Others BrowsersUse User-AgentSwitcher Add-On. However, there are Some Issues Differences in how Desktop and Mobile Browsers LayoutContent may Render Testing Results Unreliable Performance Differences in Desktop and MobileBrowsers Feature Differences in Desktop and Mobile Browsers 34. Testing Tools: Web Applications Desktop Browser Automation Tools WebDriver (http://code.google.com/p/webdriver/) Java APIs that can be Integrated with Eclipse Selenium (http://seleniumhq.org/) Firefox add-on that records clicks, typing, and other actions tomake a test, which can be played back in the browser. Selenium RC can be Used in a Desktop Browser with HTTPHeader Emulation IPhoneDriver A WebDriver based Tool for Testing on UIWebView 35. Testing Tools: On-Device Applications iPhone Embedded Browser Component (UIWebView) iPhoneUnitTesting Memory Sweep (For memory snapshot of iPhone) Java ME J2MEUnit (For Unit Testing) JInjector (For System Testing) ASM (To Instrument the Code) 36. Testing Tools: On-Device Applications Android Dalvik Debug Monitor Server (DDMS) Thread and Heap Information on Device Screen Capture on Device Incoming Call, SMS, Location Data Spoofing Monkey Tool (for Both Simulator and Device) Generate Random Streams of User Events Click and Touch Windows Mobile Hopper Simulates Input Stress by Rapidly Sending Keystrokes and Screen Taps in a Random Fashion 37. Testing Tools: On-Device Applications Symbian Bullseye Bullseye Coverage is a code coverage analyzer for C++ and C that tells how muchof the application source code was tested. Code coverage analysis is useful duringunit testing, integration testing, and final release. EUnitPro EUnit Pro enables creation and automation of unit and module tests for SymbianC++ Software. Symbian AnalyseHeap Utility This tool analyses the contents of Symbian OS heaps to produce meaningfulrepresentations of their structure and content. TRK (Target Resident Kernel) Enables On-Device Debugging Yucca Tools File browsing , Crash Monitor, Supported Fonts Preview, Memory Status etc.