Embed Size (px)
Citation preview
logo van Flair
17-12-2010
Polteq logo_RGB.png
Chasing Quality In Cloud Computing
Testing Different Levels Of Quality
Requirements
Kees Blokland
Polteq Testing Services BV, The Netherlands
Download recent version from www.polteq.com
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
2
Going to the cloud…
ERP
(test) environments
storage
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
3
Going to the cloud…
ERP
(test) environments
storage
ENABLERS
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
4
Deployment models
– private cloud
– community cloud
– public cloud
– hybrid cloud
Service Models
Cloud Computing according to NIST
Essential characteristics
On-demand service
Broad network access
Resource pooling
Rapid elasticity
Measured service
Software as a Service
Platform as a Service
Infrastructure as a Service
US: National Institute of Standards and Technology
http://www.nist.gov
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
5
Cloud Computing: risks and requirements
Essential characteristics
On-demand service
Broad network access
Resource pooling
Rapid elasticity
Measured service
Deployment models
– private cloud
– community cloud
– public cloud
– hybrid cloud
Service Models
SaaS – Software as a Service
PaaS – Platform as a Service
IaaS – Infrastructure as a Service
SaaS
PaaS
IaaS
Security?
Performance? Legislation?
Privacy?
Vendor lock-in?
Elasticity?
Testability?
Multi platform?
User experience?
Migration? Continuity?
Integration?
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
6
From risk to test
Risk groups Test groups
Performance Security Continuity Functionality Maintainability Legislation and regulations Suppliers …
Performance Security Continuity Migration Functionality Maintainability Legislation End-to-end Selection Implementation Operation …
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
7
Risk Groups – so far
Performance Security Continuity Functionality Maintainability Legislation and regulations Suppliers …
Performance Security Continuity Migration Functionality Maintainability Legislation End-to-end Selection Implementation Operation …
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
8
Risk group: performance
• Response times too long
– insufficient concurrent users
– at (un)expected peaks
• Scalability, elasticity not working
• Latency too high
• Bandwidth, throughput too low
• Up/download speed insufficient
! Other customers
! Over-book, subscription model
! Slow internet connection
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
9
Risk group: security
• Unauthorized access
– administrators cloud service supplier
– authorization/authentication inadequate
– cyber crime, hackers, authorities
– into cloud equipment building
– „somewhere‟ on the connection
• Data integrity
– erased, not erased
– unusable (loss of decryption key)
! Insecure internet connection
! Insufficient data separation in equipment
! Bring Your Own, insecure behavior users
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
10
Risk group: continuity
• Cloud service unavailable
– % availability is not achieved
– supplier bankrupt or a conflict
– internet connection lost
• Fall back plan does not work
! Internet connection malfunction
! Other suppliers disturb the service
! Supplier redundancy failure
! Business instability supplier
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
11
• No fit on the business process
• Low score on user friendliness
• Not accessible everywhere
• Not all mobile devices are supported
• The equipment/configuration is not well performed
• Customization is not well built
• Integration with other systems fails
! Limitations in the Cloud Service
! Bring Your Own Device, New Ways of Working
! The evil Internet
Risk group: functionality
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
12
Risk group: maintainability
• Cloud service not testable
• Manuals are inadequate because of changes
• An end-to-end test is not possible
• Unclear who is to solve problems
• Cloud service not adaptable to new requirements
! Cloud service changes unannounced
! Cloud service not configurable
! No test environment for cloud service
! No helpdesk
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
13
Risk group: legislation and regulations
• Violating EU data protection directive
– location, security data
– ownership, agreements with data processors
• Violating EU data retention directive
• Bankruptcy of supplier inhibits keeping obligations
• No grip on what happens to data
– warrant in other country
! Where are my data?
! Conflicting or unclear legislation
! Role of (unreliable) authorities
US: Patriot Act
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
14
Risk group: supplier
• Bankruptcy, conflict
• At the mercy of the supplier
– (pay-per-use) conditions change
– cloud service changes
• Quality not stable, unreliable
• Difficult to switch
– to another supplier
– back
! Vendor lock-in, powerful supplier
! No insight in quality SW development
! Developments (technology, growth, take-overs, …)
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
15
Test Groups – so far
Performance Security Continuity Functionality Maintainability Legislation and regulations Suppliers …
Performance Security Continuity Migration Functionality Maintainability Legislation End-to-end Selection Implementation Operation …
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
16
Test Groups – so far
Performance Security Continuity Functionality Maintainability Legislation and regulations Suppliers …
Performance Security Continuity Migration Functionality Maintainability Legislation End-to-end Selection Implementation Operation …
Testing of Packages
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
17
Test group: performance
• What are the acceptance criteria?
• Load testing
• Stress testing
– not always allowed
– what happens at the boundaries of the “bundle”
• Endurance test, volume test
– restricted possibilities: fair use policy
– monitors
• Elasticity, pay-per-use
– LOAD+PCT+BVA
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
18
Test group: performance
• Test cases based on load profiles
• Load profiles based on operational profiles
• Test environment = production environment
• Testing in real time
– under operating conditions
– with the “cloud shop open”
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
19
Testing Elasticity
100
usage
time
Load profile – „UP‟
99
100
101
Boundary values „UP‟ Load profile – „DOWN‟
Boundary values „UP‟ test case 1: usage=99, paid for 100 test case 2: usage=100, paid for 100 test case 3: usage=101, paid for 200 Boundary values „DOWN‟ test case 1: usage=101, paid for 200 test case 2: usage=100, paid for 100 test case 3: usage=99, paid for 100
200
max=100
want extension?
max=200 200 billed
max=100 100 billed
no
yes
Process Cycle Test
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
20
Test group: security
• Make inventory of security measures
– Internet connection
– Cloud service
– Client
http/ssl vpn wifi/wap data encryption
login identity management autorisation profile
access to building logs
weak passwords
authorisation
pincode mobiles
door closed patch routine
patch routine
social engineering
firewall
firewall
Security measures Authorisation Authentication Technical facilities Security updates Behaviour of people Logging
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
21
Test group: security
• Testing and assessing
– Assessing end-to-end security architecture
– Functional tests
– Tests by specialists
authorisation authentication encryption logs
encryption technique authentication technique
technical infrastructure
physical security
data separation
audit trails patch update routine
hackers test audit
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
22
Test group: continuity
• Testing of redundancy, fall back
• Off line
• Continuous end-to-end regression test
• Measuring the availability
– 99.99….9%
– critical moments
– MTBF, MTTR
• What-if scenarios
– disaster recovery
– internet unavailable
– …
Fail over testing with State Transition Test
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
23
Test group: migration
• Where goes the data?
• To/from/between cloud services
• Data repair: testing data
• Testing the data conversion tool
• Data conversion
– checklist
– performance
– security
CHECKLIST MIGRATION minimal disruption no data loss conversion successfully no hanging transactions no loss due to bad data …
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
24
• Testing SaaS = testing of standard software package
• Testing:
– fit between cloud service and business process
– configuration of the cloud service
– integration of cloud service with other systems
– multi client platforms
– the end-to-end business process
• What is the test basis?
– the old system
– process descriptions, use cases
– (functional) operational profiles
Test group: functionality
Classification Trees
Process Models
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
25
Test group: maintainability
• Test environments
– Public: none, stubs & mocks
– Private: to be negotiated
• Manuals
– Public: instructions for use
– Private: custom manuals, also for maintenance
• Change procedure
– Public: announcements supplier
– Private: to be negotiated
• Helpdesk
– Incident handling
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
26
Test group: legislation and regulation
• Storage and processing of data
– examples…
• Influence of the authorities
– examples…
• How is the test manager supposed to deal with it?
– ensure that it is taken into account
– ensure that lawyers are involved
– bridge between ICT and lawyer
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
27
Broad role of the Test Manager
Implementation testing, testing, testing
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
28
Performance Security Continuity Migration Functionality Maintainability Legislation End-to-end Selection Implementation Operation …
Implementation: what to test?
Risk groups
Test groups
Performance Security Continuity Functionality Maintainability Legislation and regulations Suppliers …
Cloud Service selected!
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
29
Broad role of the Test Manager
Selection
Implementation
risks, criteria, advice, contract
testing, testing, testing
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
30
Selection: the risks
Public SaaS
Intention: introducing Cloud Computing
Cloud Risks
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
31
Selection: criteria
Intention: introducing Cloud Computing
Selection criteria
Cost reduction Business process Performance Scalability New ways of working Continuity Migration Security Integration …
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
32
Broad role of the Test Manager
Selection
Implementation
Operation
risks, criteria, advice, contract
testing, testing, testing
end-to-end regression test, evaluation
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
33
Operation: everything is moving
Operation internet changes
Release Calendar? Change Process?
Continuous End-to-end Test
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
34
Operation, role of the test manager
• Make inventory of cloud continuity risks
– everything is moving!
• Periodic end-to-end testing
– is it still working?
logo van Flair
17-12-2010
Polteq logo_RGB.png
R G B
35 30 96 Blauw
232 62 38 Rood
108 174 68 Groen
35
Cloud & perspective of testing
From Risk To Test
Everything is moving
Broad Role Test Manager
End to End and the rest
logo van Flair
17-12-2010
Polteq logo_RGB.png
Questions?
logo van Flair
17-12-2010
Polteq logo_RGB.png
Thank you!
