1. TG AM Tutorial 4/30/13 8:30AM Test Management for Cloud-based Applications Presented by: Ruud Teunissen Polteq Test Services BV Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 904-278-0524 sqeinfo@sqe.com www.sqe.com

2. Ruud Teunissen An international test consultant at Polteq Test Services BV, Ruud Teunissen has performed several test functions in a number of IT projects: tester, test specialist, test consultant, and test manager. Ruud participated in the development of the structured testing methodology TMapTest Management Approach. Together with Martin Pol and Erik van Veenendaal, Ruud is coauthor of several books on structured testing, including Software Testing: A Guide to the TMap Approach.

3. Test Management for Cloud Based Applications Ruud Teunissen Polteq Test Services BV The Netherlands 1

4. searching, recording, accounting, paying, writing, reviewing, tracking, calculating, developing, listening, analyzing, transmitting, learning, controlling, purchasing, testing, alarming, changing, updating, deleting, accessing, rejecting, correcting, studying, booking, receiving, tracing, protecting, deciding, managing, teaching, facilitating, identifying, copying, removing, demonstrating, checking, showing, selecting, subscribing, unsubscribing, sharing, mailing, communicating, reading, playing, working, meeting, gambling, shopping, storing, cross checking, retrieving, configuring, sketching, saving, accelerating, enhancing, creating, growing, checking in, checking out, finding out, reaching, denying, talking, designing, making, verifying, measuring Develop and Test Operate and Manage Email Surf Transfer Store 4 2

5. Develop and Test Email Surf Transfer redundancy, proliferation limitations 80% unused storage claim environmentally unfriendly Operate and Manage Store 5 Develop and Test Email Surf Transfer SOA internet technology virtualization standard software Operate and Manage bandwidth Store 6 3

6. searching, recording, accounting, paying, writing, reviewing, tracking, calculating, developing, listening, Develop and transmitting, learning, controlling, Email analyzing, Test Surf purchasing, testing, alarming, changing, updating, Transfer deleting, accessing, rejecting, correcting, studying, booking, receiving, tracing, protecting, deciding, managing, teaching, facilitating, identifying, copying, removing, demonstrating, checking, showing, selecting, subscribing, unsubscribing, sharing, mailing, communicating, reading, playing, working, meeting, gambling, shopping, storing, cross checking, retrieving, configuring, sketching, saving, Operate and Manage accelerating, enhancing, creating, growing,Store checking in, checking out, finding out, reaching, denying, talking, designing, making, verifying, measuring 7 4

7. Question Please list your top 5 reasons why you are moving to the cloud 9 Top 5 Reasons 1 2 3 4 5 10 5

8. Essential characteristics On-demand service Self service provisioning, pay-per-use No human interaction US: National Institute of Standards and Technology http://www.nist.gov Essential characteristics On-demand service Broad network access Standard mechanisms over networks Any client US: National Institute of Standards and Technology http://www.nist.gov 6

9. Essential characteristics On-demand service Broad network access Resource pooling Multi-tenant Storage, processing, memory, virtual machines, Location independent US: National Institute of Standards and Technology http://www.nist.gov Essential characteristics On-demand service Broad network access Resource pooling Rapid elasticity Rapid scale in and out Any quantity at any time US: National Institute of Standards and Technology http://www.nist.gov 7

10. Essential characteristics On-demand service Broad network access Resource pooling Rapid elasticity Measured service Controlled resource use Transparency, pay-per-use US: National Institute of Standards and Technology http://www.nist.gov Essential characteristics On-demand service Broad network access Resource pooling Rapid elasticity Measured service Deployment models private cloud community cloud public cloud hybrid cloud Service Models Software as a Service Platform as a Service Infrastructure as a Service US: National Institute of Standards and Technology http://www.nist.gov 8

11. Question What is being moved into the cloud? 17 Pack List 1 2 3 4 5 18 9

12. What is done in the cloud? 500 Consumer Public Private Hybride Community Public *aaS IaaS, PaaS, DaaS, SaaS SaaS Taas *aaS Mail Storage Infrastructure CRM Finance Business processes 19 Data Centre Data Management Business processes Surf and mail Apps Social media Dropbox Google services Spotify Picasa Games Question Group Discussion Please list your top 5 cloud related risks when you are moving to the cloud 20 10

13. Top 5 Risks 1 2 3 4 5 21 Standards Cyber crime Continuity Privacy Legislation Multi platform Impact organisation 143 11

14. Performance Security Availability & Continuity Functionality Risks Manageability 23 Legislation & Regulations Suppliers & Outsourcing Performance Security YOUR Operational Profile Availability & Continuity Functionality Other customers Risks Manageability 24 Legislation & Regulations Suppliers & Outsourcing YOUR Operational Profile PLUS 12

15. Performance Security The idea: its safe Availability & Continuity Functionality Everything over the web Risks Manageability 25 Legislation & Regulations Home ground for hackers Suppliers & Outsourcing Performance Security No free choice of device. Availability & Continuity Functionality Bring Your Own Device Risks Manageability 26 Legislation & Regulations Endless possibilities. Suppliers & Outsourcing 13

16. Performance Taken care of. Security Availability & Continuity Functionality Backup and recovery Risks Manageability 27 Legislation & Regulations Who will support me? Suppliers & Outsourcing Performance Security Planned and controlled Availability & Continuity Functionality Updates, patches, fixes, Risks Manageability 28 Legislation & Regulations Do I have a choice? Suppliers & Outsourcing 14

17. Performance In house. Security Availability & Continuity Functionality Where is my data? And is that OK? Risks Manageability 29 Legislation & Regulations Somewhere Suppliers & Outsourcing Performance Security Availability & Continuity Functionality Risks Manageability 30 Legislation & Regulations Suppliers & Outsourcing 15

18. Question Group Session Lets look at the clustered risks What test measures would you consider applying? 31 Test Measures 1 2 3 4 5 32 16

19. Testing? Interview Check Trial Proof of concept Intake 33 Performance Testing Security Testing Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Interview Proof ofProef concept Testen Intake Functional Testing Migration Testing Testing caused by Legislation & Regulations 34 Testing in Production 17

20. Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability & Continuity Testing Functional Testing Manageability Risks Migration Testing 35 Legislation & Regulations Suppliers & Outsourcing Testing in Production Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Testing caused by Legislation & Regulations Availability & Continuity Testing Functional Testing Manageability Risks Migration Testing 36 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production 18

21. Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability & Continuity Testing Functional Testing Manageability Risks Migration Testing 37 Legislation & Regulations Suppliers & Outsourcing Testing in Production Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Architecture Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Testing caused by Legislation & Regulations Availability & Continuity Testing Risks From individual risks Functional Testing to Manageability individual test measures Migration Testing 38 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production 19

22. Performance Testing Selection Security Testing Manageability Testing Implementation Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Production Migration Testing Testing caused by Legislation & Regulations Testing in Production 39 Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability & Continuity Testing Functional Testing Manageability Risks Migration Testing 40 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production 20

23. Performance Testing Security Testing Manageability Testing Selection Criteria Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Testing caused by Legislation & Regulations 41 Testing in Production Question Group Session What criteria will you put on your list? 42 21

24. Knock Out List 1 2 3 4 5 43 Inspiration List CRITERION PRIO Functional Do the service and the specific business processes align? Does the service fit well in the E2E business process? Is the service sufficiently adaptable to specific requirements? Are many adjustments needed? Is customization possible Is (a lot of) customization needed? Are the required platforms supported? Are het nieuwe werken and BYOD supported sufficiently? Is it possible to connect / integrate the service with the other systems? Are sufficient manuals and/or courses available? Implementation Is the impact on current activities acceptable? Is a feasible route for migration towards the service available? 44 22

25. Inspiration List CRITERION PRIO Support Are changes in the service announced beforehand? Are sufficient test facilities available around the service (test environment, test tooling, testware, access to infrastructure, )? Are there sufficient support facilities? Is it clear how incidents can be reported? Are incidents resolved fast enough? Performance Are response times low enough? Is the number of possible simultaneous users high enough? Is bandwidth sufficient? Is sufficient potential for growth available? Is the actual use charged correctly? 45 Inspiration List CRITERION PRIO Security Are adequate authorization and authentication possibilities in place? Is the physical security of the service locations sufficient? Is the support access security of the service sufficient? Is mutual access security between customers sufficient? Are data changes traceable? Is data storage for the service reliable? Is deleting data in the service reliable? Is security of the connection to the service sufficient? Are security options for the customer sufficient? Does the supplier have security certificates? (for example SAS 70 type II)? Availability Is the level of availability for the service sufficient? Are back-up / fail-over / disaster-recovery provisions sufficient? 46 23

26. Inspiration List CRITERION PRIO Law and regulations Does the data location comply to all legal requirements? Does the data processing comply to all legal requirements? Do the terms contain parts that are conflicting to the duties of the customer? Supplier Is clear what happens when the contract ends, or in case of bankruptcy or conflict? Is a good helpdesk available? Does the supplier have experience in: - Offering this particular service? - Offering services in general? - Developing services? - The customers field? - Developing, testing and supporting services (know how)? Do methods used by supplier align with those of the customer (if relevant)? 47 Inspiration List CRITERION Supplier Is quality assurance arranged? Is the supplier ahead in its field? Is the size of the supplier in accordance with the expectations of the customer? Does the supplier have a good reputation (are there references)? Is providing services the core business of the supplier? Does the supplier have opportunities for future expansion? Does the supplier speak the same language? Is transparency and flexibility of the supplier sufficient? PRIO 48 24

27. Performance Testing Security Testing Manageability Testing Proof of Concept Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Testing caused by Legislation & Regulations Testing in Production 49 Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability & Continuity Testing Functional Testing Manageability Risks Migration Testing 50 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production 25

28. Performance Testing Security Testing Known measures tuned and tweaked Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing New measures developed Migration Testing Testing caused by Legislation & Regulations Testing in Production 51 YOUR Operational Profile Performance Testing Security Testing Manageability Testing Load Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing YOUR Operational Profile PLUS ACTUAL MOMENT 52 Testing caused by Legislation & Regulations Testing in Production 26

29. Performance Testing Security Testing Manageability Testing Online Offline Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Use case testing. Global testing. 53 Testing caused by Legislation & Regulations Testing in Production Multiplatform testing. Performance Testing Security Testing Manageability Testing Any device any platform Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Multiplatform testing. 54 Testing caused by Legislation & Regulations Testing in Production 27

30. Internet Explorer 6 Windows XP Internet Explorer 7 Windows Vista Internet Explorer 8 Windows 7 Firefox 3.5 Windows 2003 server Firefox 3.6 Browsers Windows 8 Firefox 4 Safari 4 Windows CE Safari 5 Linux Operating Systems Chrome11 Unix Opera11 Multiplatform Mac OS Lion PC Mac OS Snowleopard SUN Computer iOS Macintosh Android iPhone .. Windows Mobile Samsung Devices NOKIA Mobile Xxx ASUS.. Blackberry Tablet MOTOROLA Xxx 55 Multiplatform testing. Performance Testing Security Testing Manageability Testing Any device any platform Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Multiplatform testing. 56 Testing caused by Legislation & Regulations Testing in Production 28

31. Incidental testing. Performance Testing Security Testing Legislation + Regulations = Test basis Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functional Testing Migration Testing Testing caused by Legislation & Regulations Compliancy testing. Testing in Production 57 Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability & Continuity Testing Functional Testing Manageability Risks Migration Testing 58 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production 29

32. Performance Testing Continuous End-to-End Test Security Testing Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing during Selection Availability & Continuity Testing Functionals and non-functionals Functional Testing Migration Testing Testing caused by Legislation & Regulations 59 Testing in Production Question Group Session What would you like to (continue) test(ing) in production? And why? 60 30

33. Testing in production 1 2 3 4 5 61 Testing in production? Continuity in case of changes changes growth changes changes changes in the service at the supplier in the business process in connected resources in the internet Measuring guarantees Avalaibility Scalability - Performance - Security Evaluate original selection criteria 62 31

34. Production - Continuous End-to-End test Mechanism for Detection Important in an ever changing world Functionals and non-functionals 63 Standards Cyber crime Check Continuity Interview Privacy Legislation Proof of concept Trial Multi platform Intake Impact organisation 32

35. Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability Test starts earlier & Continuity Testing Test scope is widened Test will never Functional Testing stop Manageability Functionality Risks Migration Testing 65 Legislation & Regulations Suppliers & Outsourcing Testing in Production Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Testing caused by Legislation & Regulations Availability & Continuity Testing Questions? Functional Testing Manageability Risks Migration Testing 66 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production 33

36. Testing during Selection Performance Testing Security Security Testing Availability & Continuity Manageability Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s Performance Availability & Continuity Testing Thank you! Functional Testing Manageability Risks Migration Testing 67 Legislation & Regulations Suppliers & Outsourcing Testing caused by Legislation & Regulations Testing in Production About the speaker Ruud Teunissen Polteq Test Services, The Netherlands ruud.teunissen@polteq.com - http://www.polteq.com In the testing world since 1989, Ruud Teunissen has held numerous test functions in different organizations and projects: tester, test specialist, test consultant, test manager, etcetera. Ruud is co-author of several books on software testing and is a frequent speaker at (inter)national conferences and workshops. He was a member of the program committee for Quality Week Europe and EuroSTAR. Ruud is currently Senior Test Consultant at Polteq Test Services BV and responsible for the quality of Polteq services and assignments. 68 34