The cloud can deliver services over the Internet in three ways—software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS). Each of these approaches requires testers to focus on more than classical functional testing. Ruud Teunissen explores the new techniques and skills testers need to master for testing cloud services. Examples include testing for elasticity; testing fall back scenarios to guarantee continuity of business processes; testing for adherence to laws and regulations; and testing apps, web services, and the numerous platforms that need to be supported. Join Ruud and learn how to test these additional cloud requirements to get a grip on technical test issues, explore cloud services operations, and jump-start the broader scope of testing in the cloud. Take back practical approaches for tuning and tweaking your present test techniques to fly high in the cloud.
Citation preview
1. TM PM Tutorial 4/30/13 1:00PM High-flying Cloud Testing
Techniques Presented by: Ruud Teunissen Polteq Test Services BV
Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL
32073 888-268-8770 904-278-0524 [email protected] www.sqe.com
2. Ruud Teunissen An international test consultant at Polteq
Test Services BV, Ruud Teunissen has performed several test
functions in a number of IT projects: tester, test specialist, test
consultant, and test manager. Ruud participated in the development
of the structured testing methodology TMapTest Management Approach.
Together with Martin Pol and Erik van Veenendaal, Ruud is coauthor
of several books on structured testing, including Software Testing:
A Guide to the TMap Approach.
3. High Flying Cloud Testing Techniques Ruud Teunissen Polteq
Test Services BV The Netherlands 1
5. Develop and Test Email Surf Transfer redundancy,
proliferation limitations 80% unused storage claim environmentally
unfriendly Operate and Manage Store 5 Develop and Test Email Surf
Transfer SOA internet technology virtualization standard software
Operate and Manage bandwidth Store 6 3
7. Essential characteristics On-demand service Self service
provisioning, pay-per-use No human interaction US: National
Institute of Standards and Technology http://www.nist.gov Essential
characteristics On-demand service Broad network access Standard
mechanisms over networks Any client US: National Institute of
Standards and Technology http://www.nist.gov 5
8. Essential characteristics On-demand service Broad network
access Resource pooling Multi-tenant Storage, processing, memory,
virtual machines, Location independent US: National Institute of
Standards and Technology http://www.nist.gov Essential
characteristics On-demand service Broad network access Resource
pooling Rapid elasticity Rapid scale in and out Any quantity at any
time US: National Institute of Standards and Technology
http://www.nist.gov 6
9. Essential characteristics On-demand service Broad network
access Resource pooling Rapid elasticity Measured service
Controlled resource use Transparency, pay-per-use US: National
Institute of Standards and Technology http://www.nist.gov Essential
characteristics On-demand service Broad network access Resource
pooling Rapid elasticity Measured service Deployment models private
cloud community cloud public cloud hybrid cloud Service Models
Software as a Service Platform as a Service Infrastructure as a
Service US: National Institute of Standards and Technology
http://www.nist.gov 7
10. Question What is being moved into the cloud? 15 Pack List 1
2 3 4 5 16 8
11. What is done in the cloud? 500 Consumer Public Private
Hybride Community Public *aaS IaaS, PaaS, DaaS, SaaS SaaS Taas *aaS
Mail Storage Infrastructure CRM Finance Business processes Data
Centre Data Management Business processes 17 Surf and mail Apps
Social media Dropbox Google services Spotify Picasa Games Standards
Cyber crime Continuity Privacy Legislation Multi platform Impact
organisation 143 9
13. Performance Security The idea: its safe Availability &
Continuity Functionality Everything over the web Risks
Manageability 21 Legislation & Regulations Home gound for
hackers Suppliers & Outsourcing Performance Security No free
choice of device. Availability & Continuity Functionality Bring
Your Own Device Risks Manageability 22 Legislation &
Regulations Endless possibilities. Suppliers & Outsourcing
11
14. Performance Taken care of. Security Availability &
Continuity Functionality Backup and recovery Risks Manageability 23
Legislation & Regulations Who will support me? Suppliers &
Outsourcing Performance Security Planned and controlled
Availability & Continuity Functionality Updates, patches,
fixes, Risks Manageability 24 Legislation & Regulations Do I
have a choice? Suppliers & Outsourcing 12
15. Performance In house. Security Availability &
Continuity Functionality Where is my data? And is that OK? Risks
Manageability 25 Legislation & Regulations Somewhere Suppliers
& Outsourcing Performance Security Availability &
Continuity Functionality Risks Manageability 26 Legislation &
Regulations Suppliers & Outsourcing 13
16. Question Group Session Lets look at the clustered risks
What test measures would you consider applying? 27 Test Measures 1
2 3 4 5 28 14
17. Testing? Interview Check Trial Proof of concept Intake 29
Performance Testing Security Testing Manageability Testing Te s t M
e a s u r e s Te s t M e a s u r e s Testing during Selection
Availability & Continuity Testing Interview Proof ofProef
concept Testen Intake Functional Testing Migration Testing Testing
caused by Legislation & Regulations 30 Testing in Production
15
18. Testing during Selection Performance Testing Security
Security Testing Availability & Continuity Manageability
Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s
Performance Availability & Continuity Testing Functional
Testing Manageability Risks Migration Testing 31 Legislation &
Regulations Suppliers & Outsourcing Testing in Production
Testing during Selection Performance Testing Security Security
Testing Availability & Continuity Manageability Testing
Functionality Te s t M e a s u r e s Te s t M e a s u r e s
Performance Testing caused by Legislation & Regulations
Availability & Continuity Testing Functional Testing
Manageability Risks Migration Testing 32 Legislation &
Regulations Suppliers & Outsourcing Testing caused by
Legislation & Regulations Testing in Production 16
19. Testing during Selection Performance Testing Security
Security Testing Availability & Continuity Manageability
Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s
Performance Availability & Continuity Testing Functional
Testing Manageability Risks Migration Testing 33 Legislation &
Regulations Suppliers & Outsourcing Testing in Production
Testing during Selection Performance Testing Security Security
Testing Availability & Continuity Manageability Testing
Architecture Functionality Te s t M e a s u r e s Te s t M e a s u
r e s Performance Testing caused by Legislation & Regulations
Availability & Continuity Testing Risks From individual risks
Functional Testing to Manageability individual test measures
Migration Testing 34 Legislation & Regulations Suppliers &
Outsourcing Testing caused by Legislation & Regulations Testing
in Production 17
20. Performance Testing Selection Security Testing
Manageability Testing Implementation Te s t M e a s u r e s Te s t
M e a s u r e s Testing during Selection Availability &
Continuity Testing Functional Testing Production Migration Testing
Testing caused by Legislation & Regulations Testing in
Production 35 Testing during Selection Performance Testing Security
Security Testing Availability & Continuity Manageability
Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s
Performance Availability & Continuity Testing Functional
Testing Manageability Risks Migration Testing 36 Legislation &
Regulations Suppliers & Outsourcing Testing caused by
Legislation & Regulations Testing in Production 18
21. Performance Testing Security Testing Manageability Testing
Selection Criteria Te s t M e a s u r e s Te s t M e a s u r e s
Testing during Selection Availability & Continuity Testing
Functional Testing Migration Testing Testing caused by Legislation
& Regulations Testing in Production 37 Performance Testing
Security Testing Manageability Testing Proof of Concept Te s t M e
a s u r e s Te s t M e a s u r e s Testing during Selection
Availability & Continuity Testing Functional Testing Migration
Testing Testing caused by Legislation & Regulations 38 Testing
in Production 19
22. Testing during Selection Performance Testing Security
Security Testing Availability & Continuity Manageability
Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s
Performance Availability & Continuity Testing Functional
Testing Manageability Risks Migration Testing 39 Legislation &
Regulations Suppliers & Outsourcing Testing caused by
Legislation & Regulations Testing in Production Performance
Testing Security Testing Known measures tuned and tweaked
Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s
Testing during Selection Availability & Continuity Testing
Functional Testing New measures developed Migration Testing Testing
caused by Legislation & Regulations 40 Testing in Production
20
23. Question Group Session Suppose your CRM appliction is moved
into the Cloud. What would you test? 41 Test Measures When CRM
Moves into the Cloud 1 2 3 4 5 42 21
24. YOUR Operational Profile Performance Testing Security
Testing Manageability Testing Load Testing Te s t M e a s u r e s
Te s t M e a s u r e s Testing during Selection Availability &
Continuity Testing Functional Testing Migration Testing YOUR
Operational Profile PLUS ACTUAL MOMENT 43 Testing caused by
Legislation & Regulations Testing in Production Yes, you can!
Performance Testing Security Testing Manageability Testing Stress
Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing
during Selection Availability & Continuity Testing Functional
Testing Migration Testing Definitly NOT! 44 Testing caused by
Legislation & Regulations Testing in Production 22
25. Load and stress. Performance Testing Security Testing
Manageability Testing Elasticity Te s t M e a s u r e s Te s t M e
a s u r e s Testing during Selection Availability & Continuity
Testing Functional Testing Migration Testing Testing caused by
Legislation & Regulations Load and elasticity. 45 Testing in
Production Question How would you test Elasticity? 46 23
26. Test Elasticity 47 load test down load test up 101 100
boundary values 99 load path test extend? up tc 1: use=99, pay 100
tc 2: use=100, pay 100 tc 3: use=101, pay 200 no yes 200 charged
boundary values 100 charged down tc1: use=101, pay 200 tc2:
use=100, pay 100 tc3: use=99, pay 100 48 24
27. load test down load test up 101 100 boundary values
(Automatic) scaling up or down 99 does not perform as required load
At scaling moments functional problems test boundary values path
emerge extend? yes no Insight in use not sufficient 200 charged 100
charged up tc 1: use=99, pay based costspay100 is 100 tc 2:
use=100, tc 3: use=101, pay 200 down tc1: use=101, pay 200 tc2:
use=100, pay 100 tc3: use=99, pay 100 49 Security at: Network
Supplier User Performance Testing Security Testing Manageability
Testing Encryption Authentication and autorisation Test logs and
audit trails Security Audits 50 Te s t M e a s u r e s Te s t M e a
s u r e s Testing during Selection Availability & Continuity
Testing Functional Testing Migration Testing Testing caused by
Legislation & Regulations Testing in Production 25
28. Completeness and correctness of specifications Supplier
User Availibilty of test environments Management of: Defects
Changes Maintainability of the software 51 Performance Testing
Security Testing Manageability Testing Te s t M e a s u r e s Te s
t M e a s u r e s Testing during Selection Availability &
Continuity Testing Functional Testing Migration Testing Testing
caused by Legislation & Regulations Testing in Production
Manageablity of test environments Everything in the cloud 52
26
29. Manageablity of test environments Link all current
environments to the service 53 Manageablity of test environments
Link Production to the real service Link other environmnets to a
MOCK SERVICE 54 27
30. Question Who will solve the defects? And how? 55 Defect
resolution 56 28
31. Defect Management 57 Performance Testing Role of system
architecture Security Testing Monitoring and Logging Manageability
Testing Guarantees and SLAs Availability & Continuity Testing
Test fail-over mechanism Te s t M e a s u r e s Te s t M e a s u r
e s Testing during Selection Functional Testing Test online/offline
Migration Testing Testing caused by Legislation & Regulations
58 Testing in Production 29
32. Performance Testing Security Testing Manageability Testing
Online Offline Te s t M e a s u r e s Te s t M e a s u r e s
Testing during Selection Availability & Continuity Testing
Functional Testing Migration Testing Use case testing. Global
testing. 59 Testing caused by Legislation & Regulations Testing
in Production Fail-over testing 60 30
33. Performance Testing Security Testing Manageability Testing
Te s t M e a s u r e s Te s t M e a s u r e s Testing during
Selection Availability & Continuity Testing Functional Testing
Migration Testing Testing caused by Legislation & Regulations
61 Testing in Production Question So whats new here? 62 31
34. Functional Testing 63 Multiplatform testing. Performance
Testing Security Testing Manageability Testing Any device any
platform Te s t M e a s u r e s Te s t M e a s u r e s Testing
during Selection Availability & Continuity Testing Functional
Testing Migration Testing Multiplatform testing. 64 Testing caused
by Legislation & Regulations Testing in Production 32
35. Internet Explorer 6 Windows XP Internet Explorer 7 Windows
Vista Internet Explorer 8 Windows 7 Firefox 3.5 Windows 2003 server
Firefox 3.6 Browsers Windows 8 Firefox 4 Safari 4 Windows CE Safari
5 Linux Operating Systems Chrome11 Unix Opera11 Multiplatform Mac
OS Lion PC Mac OS Snowleopard SUN Computer iOS Macintosh Android
iPhone .. Windows Mobile Samsung Devices NOKIA Mobile Xxx ASUS..
Blackberry Tablet MOTOROLA Xxx 65 Multiplatform testing.
Performance Testing Security Testing Manageability Testing Any
device any platform Te s t M e a s u r e s Te s t M e a s u r e s
Testing during Selection Availability & Continuity Testing
Functional Testing Migration Testing Multiplatform testing. 66
Testing caused by Legislation & Regulations Testing in
Production 33
36. Performance Testing Security Testing Manageability Testing
Te s t M e a s u r e s Te s t M e a s u r e s Testing during
Selection Interruption of business processes is minimal All
migrated data can be tracked (audit trail) All data is converted
correctly All pending transactions are successfully finished after
the migration Defects in data before migration do not lead to
problems during migration Defects in data are solved and not
migrated defective No more data than necessary is migrated to the
service Availability & Continuity Testing Functional Testing
Migration Testing Testing caused by Legislation & Regulations
Testing in Production 67 Incidental testing. Performance Testing
Security Testing Legislation + Regulations = Test basis
Manageability Testing Te s t M e a s u r e s Te s t M e a s u r e s
Testing during Selection Availability & Continuity Testing
Functional Testing Migration Testing Testing caused by Legislation
& Regulations Compliancy testing. 68 Testing in Production
34
37. Testing during Selection Performance Testing Security
Security Testing Availability & Continuity Manageability
Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s
Performance Availability & Continuity Testing Functional
Testing Manageability Risks Migration Testing 69 Legislation &
Regulations Suppliers & Outsourcing Testing caused by
Legislation & Regulations Testing in Production Performance
Testing Continuous End-to-End Test Security Testing Manageability
Testing Te s t M e a s u r e s Te s t M e a s u r e s Testing
during Selection Availability & Continuity Testing Functionals
and non-functionals Functional Testing Migration Testing Testing
caused by Legislation & Regulations 70 Testing in Production
35
38. Standards Cyber crime Check Continuity Interview Privacy
Legislation Proof of concept Trial Multi platform Intake Impact
organisation Testing during Selection Performance Testing Security
Security Testing Availability & Continuity Manageability
Testing Te s t M e a s u r e s Te s t M e a s u r e s Performance
Availability Test starts earlier & Continuity Testing Test
scope is widened Test will never Functional Testing stop
Manageability Functionality Risks Migration Testing 72 Legislation
& Regulations Suppliers & Outsourcing Testing caused by
Legislation & Regulations Testing in Production 36
39. Testing during Selection Performance Testing Security
Security Testing Availability & Continuity Manageability
Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s
Performance Availability & Continuity Testing Questions?
Functional Testing Manageability Risks Migration Testing 73
Legislation & Regulations Suppliers & Outsourcing Testing
in Production Testing during Selection Performance Testing Security
Security Testing Availability & Continuity Manageability
Testing Functionality Te s t M e a s u r e s Te s t M e a s u r e s
Performance Testing caused by Legislation & Regulations
Availability & Continuity Testing Thank you! Functional Testing
Manageability Risks Migration Testing 74 Legislation &
Regulations Suppliers & Outsourcing Testing caused by
Legislation & Regulations Testing in Production 37
40. About the speaker Ruud Teunissen Polteq Test Services, The
Netherlands [email protected] - http://www.polteq.com In
the testing world since 1989, Ruud Teunissen has held numerous test
functions in different organizations and projects: tester, test
specialist, test consultant, test manager, etcetera. Ruud is
co-author of several books on software testing and is a frequent
speaker at (inter)national conferences and workshops. He was a
member of the program committee for Quality Week Europe and
EuroSTAR. Ruud is currently Senior Test Consultant at Polteq Test
Services BV and responsible for the quality of Polteq services and
assignments. 75 38