Upload
techwellpresentations
View
26
Download
2
Tags:
Embed Size (px)
Citation preview
4/23/15
1
Mobile App Testing: The Good, The Bad, and The Ugly
Jon D. Hagar, Consultant, Grand Software Testing [email protected]
Author: Software Test Attacks to Break Mobile and Embedded Devices
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices” 1
* Gaming Testing Story
* It only takes a few minutes using an App before users like or hate it
* Worse than that. . . * Many users will post a social media review of the app
* You don’t want to be a BAD
Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “So9ware Test ACacks to Break Mobile and Embedded Devices”
2
The Mobile Opportunity
4/23/15
2
* Depth
* Passion
* Speed
What Does it Take to be a Great Mobile App Tester?
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – Software Test Attacks to Break Mobile and Embedded Devices 3
* As the names imply, these are devices—small, held in the hand, connected to communication networks, including * Cell and smart phones – apps * Tablets * Medical devices
* Typically have: * Many of the problems of classic embedded systems * The power of PCs/IT * More user interface (UI) than classic embedded systems * Fast and frequent updates
* However, mobile devices are “evolving” with more power, resources, apps, etc.
* Mobile is the “hot” area of computers/software * Testing rules and concepts are still evolving * Now starting to include IoT
You know what they are right? Mobile and Handheld?
Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “So9ware Test ACacks to Break Mobile and Embedded Devices”
4/23/15
3
* Requirements verification checking * Necessary but not sufficient
* Risk–based testing * Tried and true in many contexts including mobile, but we need more
Here comes the Good, Bad and Ugly
We Need Better App Testing
Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “So9ware Test ACacks to Break Mobile and Embedded Devices”
The Bad
You are between a Management Rock and a Hard App
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices” 6
4/23/15
4
* Management directed “No testing” * Dev-‐ops without enough “thinking” of context and risk to find the big BUGS * Stupid requirements verification checking without GOOD test activities * Testing without thinking of * cost * schedule * users
Con: Current Badness
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices” 7
* Are you part of the problem?
* Do you help management “SEE” the info they need?
* Are you Agile?
* Are you using your testing skills daily?
* Bug are out there (and always will be)…………..
Pro: In the Bad
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – Software Test Attacks to Break Mobile and Embedded Devices 8
4/23/15
5
* From Wikipedia: Taxonomy is the practice and science of classification. The word finds its
roots in the Greek τάξις, taxis (meaning 'order', 'arrangement') and νόμος, nomos ('law' or 'science'). Taxonomy uses taxonomic units, known as taxa (singular taxon). In addition, the word is also used as a count noun: a taxonomy, or taxonomic scheme, is a particular classification ("the taxonomy of ..."), arranged in a hierarchical structure.
* Helping to “understand and know”
Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “So9ware Test ACacks to Break Mobile and Embedded Devices”
9
A Bad Situation
-‐ Lets look for bugs, but where?
Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “So9ware Test ACacks to Break Mobile and Embedded Devices”
10
Pro: Taxonomy (researched) Super Category
Aero-‐Space Med sys Mobile General Time 3 2 3 Interrupted -‐ Satura>on (over >me)
5.5 Time Boundary – failure resul>ng from incompa>ble system >me formats or values
0.5 1 Time -‐ Race Condi>ons
3 1 Time -‐ Long run usages 4 1 20 Interrupt -‐ >ming or priority inversions
0.7 3 Date(s) wrong/cause problem
0.5 1 Clocks 4 2 Computa>on -‐ Flow 6 23 19 Computa>on -‐ on data 4 1 3 1
4/23/15
6
Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “So9ware Test ACacks to Break Mobile and Embedded Devices”
11
Taxonomy part 2 Super Category
Aero-‐Space Med sys Mobile General Data (wrong data loaded or used) 4 5.00 2 Ini>aliza>on 6 2.00 3 5 Pointers 8 2.00 18 10 Logic and/or control law ordering
8 43 3 30 Loop control –Recursion
1 Decision point (if test structure) 0.5 1 1 Logically Impossible & dead code
0.7 Opera>ng system – (Lack of Fault tolerance , interface to OS, other) 1.5 2 6 Software - Hardware interfaces
16 13 So9ware -‐ Software Interface
5 2.00 3 So9ware -‐ Bad command- problem on server 3 5 UI -‐ User/ operator interface
4 5.00 20 10 UI -‐ Bad Alarm 0.5 3 UI -‐ Training – system fault resul>ng from improper training
3 Other 10.6 9.00 5 5
Note: one report on C/C++ indicated 70% of errors found involved pointers
* How many of you have a Mobile App taxonomy that you use?
Question
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices” 12
4/23/15
7
The Ugly We need Wisdom, Tooling, and Security
13 Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices”
* Some of you lack mobile tester skills
* Many of you suffer from group think and lack wisdom * We listen to the loudest voices
* Testers do not use available ideas to aide their skill base * Attacks, techniques, tools, concepts, standards, etc.
Con: Mobile can have an Ugly Face
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices” 14
4/23/15
8
* Danger of group think in Agile Mobile Teams
* Amplification
* Snowballing effect
* Polarization
* Ignoring critical minority opinions
Pro: You Need Test Wisdom
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices” 15
* Stop talking and LISTEN to all sides, particularly the ones you may not agree with * Question beliefs * Be passionate and follow your bliss about testing * Try to remain open minded * Do not submit to the negatives of group think * Consider the context of the testing and believe that context matters * Seek the council of people you believe to be wise * Reward your test team for being open and providing other views without fear * Try to take a role of “devil’s advocate” in your test team * Fight the “me too” syndrome and everyone falling in line to the loudest voice * Work to be a knowledgeable and skilled tester (they are different) * Be the voice of loyal opposition in the team and think outside of the group “box” * Don’t paint a viewpoint as totally invalid, when a few ideas of the viewpoint conflict with
local ideals
Seeking Test Wisdom (Pro: try these tricks)
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices” 16
4/23/15
9
Categories of Automation Tooling (Open Source and Commercial)
* Capture Playback -‐ Actual devices (cabinet vs a pile) vs Emulator -‐ API vs GUI/UI
* Planning and lifecycle support * Modeling
-‐ Risks -‐ Mind-‐mapping -‐ Formal models (UTP) -‐ Test Techniques
Pro/Con? -‐ Mobile/Handheld Test Tools
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices” 17
* To Automate or Not? * When testing configurations of hw/sw (good idea) * When testing combinations (combinatorial test tools) * When dealing with testing qualities * Security (very good idea) * Reliability (necessary) * Configuration management (can not be done without) * Usability (important but a hard one and questionable tools)
* When supporting Development * Structural testing (measures coverage) * Static code analysis (finds hard to test bugs) * Dev-‐Ops, Continuous Integration and Agile (really good)
More on Test Tools – Now in Mobile Support has Improved
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – Software Test Attacks to Break Mobile and Embedded Devices 18
4/23/15
10
* Your app gets on the nightly news
* Your team sees security as someone else’s problem
Real Ugly: Security and Privacy
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices” 19
* Mobile– IoT systems are highly integrated hardware–software–system solutions which: * Must be highly trustworthy since they handle sensitive data * Often perform critical tasks
* Security holes and problems abound * Coverity Scan 2010 Open Source Integrity Report -‐ Android * Static analysis test attack found 0.47 defects per 1,000 SLOC * 359 defects in total, 88 of which were considered “high risk” in
the security domain
* OS hole Android with Angry Birds * Researchers Jon Oberheide and Zach Lanier
* Robots and Drones rumored to be attacked * Cars and medical devices being hacked
The Current Security Situation
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices”
4/23/15
11
* Fraud – Identity * Worms, virus, etc. * Fault injection
* Processing on the run * Hacks impact * Power * Memory * CPU usage
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices”
Con: Mobile Security Bugs (taxonomy)
• Eavesdropping – “yes everyone can hear you” • Hijacking • Click-‐jacking • Voice/Screen
• Physical Hacks • File snooping • Lost phone
* A pattern (of testing) based on a common mode of failure seen over and over * Part of Exploratory Testing * May be seen as a negative, when it really is a positive * Goes after the “bugs” that may be in the software * May include or use classic test techniques and test concepts * Lee Copeland’s book on test design * Many other good books
* A Pattern (more than a process) which must be modified for the context at hand to do the testing
* Testers learn mental attack patterns working over the years in a specific domain
Pro: Apply Attack-‐based Testing What is an attack?
Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “So9ware Test ACacks to Break Mobile and Embedded Devices”
4/23/15
12
* Apply when the device is mobile and has * Account numbers * User-‐ids and passwords * Location tags * Restricted data
* Current authentication approaches in use on mobile devices * Server-‐based * Registry (user/password)
* Location or device-‐based * Profile-‐based
Security Attacks
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices”
* Attack 28 Penetration Attack Test * Attack 28.1 Penetration Sub–Attacks: Authentication — Password * Attack 28.2 Sub–Attack Fuzz Test * Attack 29: Information Theft—Stealing Device Data * Attack 29.1 Sub Attack –Identity Social Engineering * Attack 30: Spoofing Attacks * Attack 30.1 Location and/or User Profile Spoof Sub–Attack * Attack 30.2 GPS Spoof Sub–Attack
Security Attacks (Con: only a starting point, a checklist of things to start with)
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices”
4/23/15
13
* What kind of App software do you work on?
* Security concerns?
* Privacy concerns? What is missing?
Exercise
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices” 25
§ Security attacks must be done with the knowledge and approval of owners of the system and software
§ Severe legal implications exist in this area § Many of these attacks must be done in a lab (sandbox) § In these attacks, I tell you conceptually how to “drive a car very fast
(150 miles an hour) but there are places to do this with a car legally (a race track) and places where you will get a ticket (most public streets)”
§ Be forewarned -‐ Do not attack you favorite app on your phone or any connected server without the right permissions due to legal implications
Warnings When Conducting Security Attacks
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices”
4/23/15
14
Finally, The Good – Functional and Non-‐functional
Experiments and Attacks (Exploratory testing)
Skills App testers should have
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC –”So9ware Test ACacks to Break Mobile and Embedded Devices” 27
Attacks (from Software Test Attacks to Break Mobile and Embedded Devices)
* Attack 1: Static Code Analysis * Attack 2: Finding White–Box Data Computation Bugs * Attack 3: White–Box Structural Logic Flow Coverage * Attack 4: Finding Hardware–System Unhandled Uses in Software * Attack 5: Hw-‐Sw and Sw-‐Hw signal Interface Bugs * Attack 6: Long Duration Control Attack Runs * Attack 7: Breaking Software Logic and/or Control Laws * Attack 8: Forcing the Unusual Bug Cases * Attack 9 Breaking Software with Hardware and System
Operations * 9.1 Sub–Attack: Breaking Battery Power * Attack 10: Finding Bugs in Hardware–Software Communications
* Attack 11: Breaking Software Error Recovery * Attack 12: Interface and Integration Testing * 12.1 Sub–Attack: Configuration Integration Evaluation * Attack 13: Finding Problems in Software–System Fault Tolerance * Attack 14: Breaking Digital Software Communications * Attack 15: Finding Bugs in the Data * Attack 16: Bugs in System–Software Computation * Attack 17: Using Simulation and Stimulation to Drive Software
Attacks * Attack 18: Bugs in Timing Interrupts and Priority Inversion * Attack 19: Finding Time Related Bugs
* Attack 20: Time Related Scenarios, Stories and Tours
* Attack 21: Performance Testing Introduction * Attack 22: Finding Supporting (User) Documentation
Problems * Sub–Attack 22.1: Confirming Install–ability * Attack 23: Finding Missing or Wrong Alarms * Attack 24: Finding Bugs in Help Files * Attack 25: Finding Bugs in Apps * Attack 26: Testing Mobile and Embedded Games * Attack 27: Attacking App–Cloud Dependencies * Attack 28 Penetration Attack Test * Attack 28.1 Penetration Sub–Attacks: Authentication —
Password Attack * Attack 28.2 Sub–Attack Fuzz Test * Attack 29: Information Theft—Stealing Device Data
* Attack 29.1 Sub Attack –Identity Social Engineering
* Attack 30: Spoofing Attacks * Attack 30.1 Location and/or User Profile Spoof Sub–Attack * Attack 30.2 GPS Spoof Sub–Attack * Attack 31: Attacking Viruses on the Run in Factories or PLCs * Attack 32: Using Combinatorial Tests * Attack 33: Attacking Functional Bugs
Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “So9ware Test ACacks to Break Mobile and Embedded Devices”
4/23/15
15
Attack 1: Static Code Analysis (testing)
* When to apply this attack? * After/during coding
* What faults make this attack successful? * Many * Example: Issues with pointers
* Who conducts this attack? * Developer, tester, independent party
* Where is this attack conducted? * Tool/test lab
* How to determine if the attack exposes failures? * Review warning messages and find
true bugs
* How to conduct this attack * Obtain and run tool * Find and eliminate false positive * Identify and address real bugs * Repeat as code evolves * Single unit/object * Class/Group * Component * Full system
29 Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC –”So9ware Test ACacks to Break Mobile and Embedded Devices”
Attack 2: Finding White–Box Data Computation Bugs
* When to apply this attack? * After/during coding
* What faults make this attack successful? * Mistakes associated with data * Example: Wrong value of Pi
* Who conducts this attack? * Developer, tester, independent party
* Where is this attack conducted? * Development Tool/test lab
* How to determine if the attack exposes failures? * Structural-‐data test success criteria
not met
* How to conduct this attack * Obtain tool * Determine criteria and coverage * Create test automation with
specific values (really a programing problem) * NOT NICE NUMBERS
* Run automated test cases * Resolve failures * Peer check test cases * Repeat as code evolves
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC –”So9ware Test ACacks to Break Mobile and Embedded Devices”
4/23/15
16
* When to apply this attack? …when your app/device has a user * What faults make this attack successful? …devices are increasingly
complex * Who conducts this attack? …see chart on Roles * Where is this attack conducted? …throughout lifecycle and in user’s
environments
* How to determine if the attack exposes failures? * Unhappy “users” * Bugs found * See sample checklist
Attack : Testing Usability Mobile IoT Usability Tends to be “Poor”
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices”
* Refine checklist to context scope * Define a role * Watch what is happening with this role * Define a usage (many different user roles) * Guided explorations or ad hoc * Stress, unusual cases, explore options * Capture understanding, risk, observations, etc. * Checklist (watch for confusion of the tester) * Run Exploratory Attack (s) * Learn * Re-‐plan-‐design * Watch for Bias * Switch testers * Repeat
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices”
Usability Attack Pattern
4/23/15
17
The Good, Bad, and Ugly of Mobile App Testing
Lots of room for Growth
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
33
How to be Better after This Section Pick One or Two to work On
Cons: Bad and Ugly
* Taxonomy help only if you use them
* Skill improvement * Knowledge and Skill
* Security Testing * Attack, Attack, Attack
Pro: The Good
* Better and Faster * Functional testing * Test strategy and planning
* Test Attacks
* Tools and technique maturing
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices “
34 After Mobile comes IoT
4/23/15
18
* There will always be Good, Bad, and Ugly * Work with the Good * Work to over come the Bad * Change the Ugly into good
* Understanding your local context and error patterns is important (one size does NOT fit all)
* Attacks are patterns…you must still THINK and tailor
Wrap Up of this Session
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices”
* James Whittaker (attacks) * Elisabeth Hendrickson (simulations) * Lee Copeland (techniques) * Brian Merrick (testing) * James Bach (exploratory and tours) * Cem Kaner (test thinking) * Jean Ann Harrison (her thinking and help)
* Many teachers * Generations past and future * Books, references, and so on
Notes: Thank You (ideas used from)
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices”
4/23/15
19
* “Software Test Attacks to Break Mobile and Embedded Devices” – Jon Hagar
* “How to Break Software” James Whittaker, 2003 * And his other “How To Break…” books
* “A Practitioner’s Guide to Software Test Design” Copeland, 2004 * “A Practitioner’s Handbook for Real-‐Time Analysis” Klein et. al., 1993 * “Computer Related Risks”, Neumann, 1995 * “Safeware: System Safety and Computers”, Leveson, 1995 * Honorable mentions: * “Systems Testing with an Attitude” Petschenik 2005 * “Software System Testing and Quality Assurance” Beizer, 1987 * “Testing Computer Software” Kaner et. al., 1988 * “Systematic Software Testing” Craig & Jaskiel, 2001 * “Managing the Testing Process” Black, 2002
Book/Notes List (my favorites)
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices”
• www.stickyminds.com – Collection of test info • www.embedded.com – info on attacks * www.sqaforums.com -‐ Mobile Devices, Mobile Apps -‐ Embedded Systems Testing forum
• Association of Software Testing
– BBST Classes http://www.testingeducation.org/BBST/
• Your favorite search engine
More Resources
Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – So9ware Test ACacks to Break Mobile and Embedded Devices