Upload
lamdan
View
215
Download
0
Embed Size (px)
Citation preview
Noelia Ortiz, MME, CSSGB, CQA
1
Agenda What is Cloud Computing?
Understanding the Responsibilities of the Test Manager
Performing the Risk Assessments
Testing Cloud
2
Part 1
3
What is Cloud Computing?
Is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.
– NIST
4
Essential Characteristics On-demand self-service
Customers can configure computer facilities without human-interaction with the service supplier.
Services are easily available and can be obtained directly over the Internet.
5
Essential Characteristics Broad network access
Services are offered in a network.
Services can be obtained from different sources (e.g., PCs, laptops, tablets, mobile phones).
6
Essential Characteristics Resource pooling
Multiple customers share the supplier’s infrastructure through a rental model.
Computer resources: storage, computing capacity, memory, network bandwidth, and virtual environments.
7
Essential Characteristics Rapid elasticity
Services can be configured and released quickly and most of the time in an elastic fashion.
Offers the capability of quickly scaling up and down.
Customers can obtain services at any moment and in any desire quantity.
8
Essential Characteristics Measured service
Systems check and optimize the use of the underlying infrastructure.
The usage of the following is measured:
Storage Capacity
Computing Capacity
Bandwidth
Active user accounts
Provides transparency for the supplier and customer.
9
Benefits of Cloud Computing
10
Risks of Cloud Computing
CLOUD RISKS
Vendor Longevity
Support
Change Control
Security Privacy
Availability
Regulatory Compliance
Data Mobility
11
Types of Service Models
12
Types of Service Models Layers of the service growth model:
Hardware: Equipment (servers, network resources, storage).
Virtualization: Software that makes it possible to create multiple or different environments, based on the hardware. The software runs in an environment that is not physically present.
Platform: Runtime environment in which software can run (.NET, PHP, Apache)
Application: The software for the customer.
13
Types of Service Models No service purchased
In-House Outsourced
Application
Platform
Virtualization
Hardware
14
Types of Service Models IaaS
The customer gets access to computing power, storage, networks and other basic computer facilities and puts together their own infrastructure.
The customer does not manage the cloud but does decide which operating system run on it, the amount of storage, and the applications that are rolled out on it.
15
Types of Service Models IaaS
In-House Outsourced
Application
Platform
Virtualization
Hardware
16
Types of Service Models PaaS
The customer uses the programming languages and tools that are supported by the PaaS supplier.
The customer has no control or management over the infrastructure but does have complete control over the applications and some of the configuration of the platform environment and makes their choices for developing or buying them.
17
Types of Service Models PaaS
In-House Outsourced
Application
Platform
Virtualization
Hardware
18
Types of Service Models SaaS
Customer uses applications from a supplier that are running on cloud infrastructure.
These applications are accessible through a simple interface - such a web browser - for everybody, at any time, in any place and with any device.
The customer has limited influence to the configuration.
19
Types of Service Models SaaS
In-House Outsourced
Application
Platform
Virtualization
Hardware
20
Types of Service Models
21
Implementation Models
Public Cloud
Community Cloud
Private Cloud Hybrid Cloud
22
Implementation Models Private Cloud
Exclusively used by one customer.
The cloud can be managed by the customer or by an external party and can be located in-house or in an external location.
23
Implementation Models Community Cloud
Shared by a number of customers and supports a specific community that has shared interests (security requirements, policies, standards).
Management is done by the customers themselves or is outsourced.
The infrastructure can be located in-house at one of the customer’s premises or at an external location.
24
Implementation Models Public Cloud
The cloud infrastructure is publicity accessible.
The owner is the organization that provides the service.
Hybrid Cloud
Consists of two or more independently operating clouds (private, community, public) that are combined to make the exchange of data and applications possible.
25
Part 2
26
Risk Assessment With the different stakeholders, the chance of
something going wrong with the information system is discussed along with the impact this can have.
A large likelihood of failure with significant consequences represents a high risk and requires a lot of test effort.
27
Risk Assessment
System
ComponentRisk Group Risk
Classification
Failure Rate Impact Risk Category
Security
Performance
Manageability
Legislation and
regulation
28
Obtaining Agreement and Information from the Supplier At this stage, a lot of testing activities are deployed
from which information and support from the supplier is needed.
Additional agreement with the supplier in those cases where the supplier’s standard terms and conditions do not fully provide them.
Service specifications
Test environments
Guidelines for logging issues
29
Obtaining Agreement and Information from the Supplier Test Manager needs to watch out:
Where are the infrastructure configuration?
Where can the manuals be
found?
What are the guarantees on availability?
How is the customer
informed about changes in the
service?
How are issues logged?
What are the guarantees on
security?
30
Obtaining Agreement and Information from the Supplier Additional terms and conditions:
Supplier Audits (security or
process quality)
Support with implementing
the service
Periodic supplier evaluations and
reviews of service
31
Test Manager Responsibilities Selection
To determine the suitability of a service the test manager should perform the following:
Put forward cloud related aspects
Determine completeness and controllability of the selection criteria
Assess services and suppliers
Issue a selection advice
32
Test Manager Responsibilities Implementation
Implementation consists of all activities that are needed to take the service into production.
The steps for the implementation consist on the following:
Executing the risk assessment
Setting up the test strategy
Executing the test measure
33
Test Manager Responsibilities Production
The tasks of the test manager continues after a successful implementation.
The test manager is responsible to perform the following tasks once the Cloud is in production:
Ensure continuity as changes are implemented
Monitor the quality of the service
Review the original selection criteria
34
Testing with the help of Cloud Many testing services are offered under the name
“Testing as a Service” (TaaS).
TaaS = Test outsourcing
Test outsourcing can be handled with the use of the “Test Outsourcing Governance Approach (TOGA®).
35
Part 3
36
Performance Risks With cloud computing, more communication is done
over the Internet, which makes performance risk more significant.
37
Performance Risks Example:
Performance of a travel-booking website that is hosted in the cloud
What if the clients have to wait too long to book a flight?
An important performance risk would be the Internet connection.
38
Performance RisksRisks
Response times result in problems Performance decreases due to changes by supplier
Processing capacity (throughput) is insufficient
Scaling (up/down) does not suffice
Upload/download speed (bandwidth) is insufficient
User experiences quality of video or audio as low
Other customers affect performance User requirements for performance are not clear
Performance is not sufficient on all types of devices
Executing performance tests is not possible
Performance varies due to Internetconnection
Performance test are not representative
Performance deteriorate over time User demands change over time39
Security Risks There is a lot of regulation on information security and
privacy.
Sharing IT resources with other customers produces security issues.
One of the advantages of the cloud is that the services can be obtained everywhere and on all kinds of platforms. This results in a new vulnerability.
40
Security RisksRisks
Building insufficiently protected against break-in
Service is insufficiently robust against attacks by hackers
Authentication is insufficient:-Other customers gain access-Unauthorized people gain access-Customer gains access to other customers’ data
Data is lost:-Storage device error-Scaling-Procedural errors-Software errors
Authorization is insufficient:-Administrators on the supplier’s are not sufficiently restricted from accessing client data
No access to data because of a business incident on the supplier side
There are too many people that can access everything (on the supplier side)
Security is not up to date
41
Availability and Continuity Risks When a service is not available at a given moment, it
immediately has a disruptive effect on the business processes that use it. (i.e. Losing Internet connection, a device error in the service itself)
Depending on the risk, specific test measures are taken for risk mitigation.
Dependency on the Internet is an important risk.
42
Availability and Continuity RisksRisks
Connection to the Internet is disrupted:-At the supplier-At the interfacing systems (customer)-At the user devices (customer)
Bankruptcy of the service supplier threatens the continuity of business processes
The Internet connection is disrupted at other locations around the world
There is no backup plan
The business process is disrupted by problems with migration:-Missing data after migration-Data changed during migration-Transaction lost during migration
Responsibility for continuity failure is not clear because multiple suppliers are involved
Data has become unreadable:-Because of hardware failure-Because of loss of encryption key
There are insufficient agreements in place about availability
43
Functionality Risks Choosing a SaaS product is very similar like selecting
an “off-the-shelf” software.
Therefore, the risks are much like the risks found in selecting and implementing an “off-the-shelf” software.
A service often has to integrate with other systems. The risk is that the business processes do not function properly anymore.
44
Functionality RisksRisks
Service and business processes do not align:-Service does not meet all requirements of the business processes-Using the service results in the need for business process changes-Service provides insufficient means for configuration
Customization:-Is needed but not possible-Does not function properly on supplier side-Does not function properly on customer side-Is not robust when changes are not implemented
Quality of the service is inadequate(bugs)
Services do not match the technical infrastructure of the customer
Service is not according to the supplier’s description
Migration problems
Repairing user errors is not possible The manuals are inadequate
Configuration is not done correctly Changes are made to the service 45
Maintainability Risks The supplier performs different forms of service
maintenance. As a result, the customer has no control over maintenance.
When customers are not informed of changes to the service, it imposes a maintainability risk.
46
Maintainability RisksRisks
Documentation from supplier is no longer correct after the service is modified
There is insufficient support from the supplier’s help desk
Business process descriptions are not up to date
There are problems with repairing error situations
Test documentation from supplier is insufficient
Environments are not available for the following purposes:- Testing the service- Testing performance- Testing the E2E business processes
There is insufficient experience with cloud computing
IT landscape is not fully under in-house control
There is no change procedure
47
Legislation and Regulations Risks Cloud computing is a worldwide market that does not
consider national borders, and as a result resides under more than one jurisdiction.
The user of the services will have to obey different regulations.
48
Legislation and Regulations RisksRisks
Processing and storing data do not comply with some laws:-Laws in home country-Laws in other countries in which the data can reside
There is no agreement with the service provider about safe handling of data
Countries have conflicting laws Reliability of a regime in a country is questionable
Legislation is unclear Third parties gain access to company data by citation or investigation (jurisdiction)
It is not clear where data is stored, which causes uncertainty about legal risks
Legal issues arise when the service is down
Legislation changes49
Suppliers and Outsourcing Risks These risks mostly are related to supplier dependency.
It is important to determine up front what happens to the continuity of the service in the case of a business conflict.
50
Suppliers and Outsourcing RisksRisks
Bankruptcy of supplier threatens continuity of business process
There are multiple suppliers (multivendor) who shrink responsibilities toward each other
Supplier shut downs the service Supplier does not support the correct functioning of the service with test results
There is ambiguity in the contract about the following:-Continuity-Performance-Security-Defects-End of contract
There is no test set available to determine the correct functioning of the service
Customer is dependent on one supplier Service ownership is handed over to another party 51
Part 4
52
Test Measures In order to address the risk, test measures are needed.
Test Management
Test Specifications
Test Execution
Testing cloud services is a pick and mix of test measures depending on the risks.
The testing approach varies between service models (IaaS, PaaS, SaaS).
53
Test Measures The starting point is the outcome of the product risk
assessment of the service.
The result is a list of relevant risks with an assigned classification (high/medium/low).
This list aids the test manager to identify the necessary test measures.
54
Testing during selection The supplier selection process is divided into the
following steps:
Include cloud-related aspects
Determine completeness and controllability of selection criteria
Assess services and suppliers
Issue selection advise
55
Testing during selection Include cloud-related aspects
When the decision is made to go into the cloud, new risks are introduced.
This may lead to certain choices in the selection process such as choosing a private service rather than a public service.
The decision to not choose a cloud solution can be also be drawn after the risks are assessed.
56
Testing during selection Determine completeness and controllability of
selection criteria
Selecting a cloud service is very similar to selecting a software package.
Evaluate service capabilities
Evaluate the supplier
Criteria need to be defined for the selection process.
Criteria must be assessable or testable.
57
Testing during selection Assess services and suppliers
Selection criteria for a service
The required quality needs to be known.
How business requirements are translated into requirements and acceptance criteria for the cloud service?
Selection criteria for a supplier
Supplier properties need to be assessed.
To what extent is the supplier of a candidate service reliable?
58
Testing during selection Assess services and suppliers
Inspect specifications and terms.
Ask for references and certificates.
Perform audits and inspections.
Proof of concept (dynamic testing).
Simulate E2E business processes.
59
Testing during selection Issue selection advise
During the selection process, not all risks identified are related to IT.
Risks related to cost and legal can also be present.
60
Testing during selection Issue selection advise
During the selection advise the following questions are addressed:
What are the risks with implementing the intended cloud solution?
To what extent are business processes supported?
What are the risks in choosing a particular service?
How much customization is needed to implement this particular service and which risks occur as a result?
What are the risks in choosing a particular supplier?
61
End-to-End Testing (E2E) E2E is the broadest possible test in which the end
results are taken as a starting point.
Is a system test of more that one system.
Ensures data integrity is maintained between various system components and systems.
62
End-to-End Testing (E2E) System Integration Test vs. E2E
The E2E testing focuses on: Business processes
“System picture” (how the systems and services are connected and information flow)
Exact functionality of the systems and services
Integration Test E2E
Mainly focuses on interfaces between systems
Tests the total combination of systems and business processes
63
End-to-End Testing (E2E) Creating E2E test cases
Describe all test actions in the test case. Example: A test starts in a certain system and after passing through a
number of systems and services ends in another system with a certain result.
Determine the data that represents the test case and the expected result of the test case.
Determine the configuration and base data needed in all systems and services through which the E2E test case passes.
Describe the expected intermediate results of the test case on all interfaces between the systems and services.
64
Testing Performance Known performance tests could be applied to test the
cloud but not all of them.
For example, stress test is not possible in most cases as it jeopardizes the stability of the service for the other customers.
To determine whether a service meets the performance requirements, acceptance criteria need to be defined.
65
Testing Performance Load Test
The tester performs actions on the service as a user would and determines whether the service reacts with sufficient speed and whether errors do or do not occur.
Possible objectives to perform the load test. Average load (number of users)
Peak load
Many users logging simultaneously
Load during working hours
Increased load on the Internet
66
Testing Performance Load Test
The fact that the customers do not know about each other’s load on the service is a complicating factor.
The load on the service by another customer can lead to different metrics from the same test at different times.
It is recommended: Obtain information from the supplier about the load statistics
of other customers.
Execute the test on real time.
67
Testing Performance Stress Test
A stress test determines the behavior of a service beyond the peak load.
This test will show what happens when the service load increases more quickly or when a sudden, unexpected peak occurs.
The stress is conducted by increasing the load until the moment something happens.
68
Testing Performance Stress Test
During the stress test execution, the tester needs to stay within the service’s terms of use to avoid impacting other customers.
It is recommended to test the system at the edge of the guaranteed load to determine what would happen with the service.
69
Testing Performance Endurance Test/Volume Test
Some of the causes for performance decreasing can often result from memory overflow (long files, memory leaks) or files becoming fragmented.
After a certain volume is processed, performance drops or stops completely.
This test is performed by inputting as much volume as possible in a short period of time to get results quickly.
70
Testing Performance Endurance Test/Volume Test
This testing approach brings risks as customer can easily move beyond the officially permitted load.
71
Testing Performance Testing Elasticity and Manual Scalability
Testing elasticity is a new aspect of performance testing.
The objectives of these tests are to determine whether the performance of the service meets the requirements across the entire load spectrum and whether the appointed service capacity scales with the service load.
72
Testing Performance Testing Elasticity and Manual Scalability
This test consists on executing a load test with the load increasing beyond the scaling boundary (scaling up) and then the load decreasing below the scaling boundary (scaling down).
The following errors could possibly emerge in testing elasticity: Automatic scaling up does not perform as required
Automatic scaling down does not perform as required
During scaling functional problems occur
73
Testing Security Security is predominantly about information security.
ISO 27001 empathizes three aspects of information security:
Data confidentiality
Integrity of data
Availability of data
74
Testing Security Testing Encryption
This test consists on verifying whether or not encryption is activated.
Most modern systems tests tools are able to test messages with and without encryption, and by comparing them, can determine if the data stream is encrypted.
75
Testing Security Testing Authorization
Functional tests are used to test authorization.
The test basis is an authorization table in which it is shown what someone with a specific role is and is not allowed to do.
User GroupAdding Users
Deleting Users
Manage Projects
Filling In Time
Sheets
Prove Time Sheets
Generate Monthly
Totals
Administration X X X X X X
ProjectManager
X X X X X
Standard User X X
76
Testing Security Testing Authorization
Test the cells by performing the actions with the specific role and checking whether it is or is not possible.
Try to gain access to data that only people with the highest authorization are permitted to access.
Try to gain access to data that only people with administrator permission are permitted to access.
77
Testing Security Testing security robustness against Internet attacks
Security risks are often exposed by hackers through directed attacks. Directory traversal – Read and/or write in directories other than
those allowed.
XML external entity attack – Include extra (bad) data in an XML file
SQL injection – Request and/or change data by manipulating SQL queries.
Testing robustness against attacks requires in-depth knowledge of and experience with test tools.
78
Testing Security Testing log files and audit trails
An important security measure is to track all changes in data and attempts to access data.
Testing is done using functional tests.
Execute changes and check whether they are registered in the log files or audit trail.
79
Testing Security Testing security patch routines
Security systems and other software need to be up to date with the latest security patches.
Testing these routines requires checking whether the proper procedures are in place and whether there is evidence that these procedures are followed.
80
Testing for Manageability The essence of manageability is how to keep the
service and the processes around it working together.
Manageability and maintainability are closely related and is about the ease that a service can be modified to keep it in production.
81
Testing for Manageability Manageability is tested using a checklist.
Manageability checks for documentation are as follows: Is documentation available?
Is documentation kept up to date?
Are changed versions sent to the current stakeholders?
Various specifications are part of manageability and are subject to testing.
82
Testing for Manageability Specifications on the supplier side
Interface specifications
Specifications for customer resources
Platform specifications
Infrastructure specifications
Specifications on the customer side Infrastructure specifications (IaaS)
Systems specifications
Architecture documentation
83
Testing for Manageability User documentation
An instruction manual is needed.
Documentation is needed on how the service needs to be configured, used, and maintained by the customer.
Test environments availability
It is important to have the availability of test environment in order to test the service before it goes live.
84
Testing for Manageability Test documentation
There are two categories:
Customer test documentation
Supplier test documentation
Test documentation is mostly used, for example, to execute regression testing and retesting for patches and infrastructural changes.
85
Testing for Manageability Incident management procedure
Preferably the supplier has an incident management procedure in which both the supplier and customer agree if and when the incident is resolved.
Certain incidents are resolved by the customer.
For example, changing the service configurations.
86
Testing for Manageability Change procedure and version control
The customer needs to set up a change control procedure.
Some changes can be performed by the customers themselves and others the supplier needs to be involved.
Continuous integration requires well-organized version control at the supplier side as well as at the customer side.
87
Testing for Manageability Change procedure and version control
The supplier is responsible to notify the customer which versions of documentation are active for which version of the service.
The customer needs to control the version of all interfacing systems along with all documentation.
It is important to know which software version and which test basis version is used.
88
Testing for Manageability Maintainability of software
For IaaS and PaaS, the customer is responsible for the maintainability of the software.
Poor maintainability results in recurring errors and parts of the service not functioning after changes and patches.
89
Testing for Availability/Continuity Availability is about a part of the IT landscape (such as
a service).
Continuity is about the process that uses it.
The continuity of a service is key:
How often does a disruption occur?
How fast it is resolved?
What damage has this disruption caused?
90
Testing for Availability/Continuity Testing failover
When executing a failover test, the worst-case scenario is assumed.
The failover test shows that the interruption of a service stays within acceptable or agreed-upon boundaries.
The service needs to be fully running to mimic a realistic situation.
91
Testing for Availability/Continuity Testing failover
For a failover test to be successful a number of things need to work properly:
IP addresses are switched
Load balancers stop trying to connect to the failed process
Backups are restored
Machines can be restarted after a power cut
Networks are up and running after a failure
Data is not lost or corrupted
92
Testing for Availability/Continuity Testing failover
Checklist specifically for SaaS continuity after failover switch:
Has the configuration been disturbed?
Is the failure even noticed?
Does the automatic failover start to work?
Are there any transactions lost?
Is there any data lost?
Does the audit trail work properly?
Is performance back to normal?
93
Functionality Testing Some of the functional test objectives are:
Does the service fit the business processes?
Do the business processes fit the service?
Is the service quality sufficient?
Is the service user friendly?
Is the service configuration done correctly?
Do interfaces between the service and other systems work properly?
Does everything work after changes?
94
Functionality Testing Testing service quality
The customer can perform an acceptance test on the service to get an impression of service quality.
Test scenarios are designed and executed for the most important processes for which the service will be used.
95
Functionality Testing Testing user-friendliness
How many screens does a user have to go through to make an appointment?
Is the tab order intuitive? Is there sufficient help functionality?
These can be tested implicitly or explicitly in the functional tests activities.
A manual or work instruction can act as a test basis and at the same time be a test object.
End users can work with the service in the development environment to learn how it works.
96
Functionality Testing Testing interfaces to other systems
Four aspects are differentiated when testing interfaces: Technique – How is data exchange organized physically?
Syntax – How is data structured and what are the syntax rules?
Semantics – Are uploaded or downloaded data processed correctly by the receiving service or receiving system?
Nonfunctional aspects (security, performance) – Any problem in these areas?
97
Functionality Testing Testing service configuration
Several things can go wrong with the configuration:
Design of the configuration is incorrect
Configuration errors
Service configuration does not work properly
Peer review is the most common method to verify the configuration.
Configuration can be tested using a functional test.
98
Functionality Testing Multi-platform testing
One of the advantages working in the cloud is the ability to access the service using different resources (laptop, tablet, mobile).
A complete functional test of the platform is not feasible.
Choices can be made based on:
What platforms and versions are used the most?
New platforms
Focus on the most-used paths through the service
99
Functionality Testing Testing for regression
When something changes in the IT landscape (the service or something else) there is always the chance that not everything will work correctly.
When changes are performed, it is important to determine where and how extensive the regression testing should be. A risk analysis will determine the depth of the testing.
Regression testing is most likely to be performed on functionality.
100
Testing Migration Different migration scenarios are:
Transfer into the cloud, where the applications remain the same. The data is only moved to another location.
Transfer to SaaS. Data from the existing application needs to be migrated to the new service.
Transfer from one SaaS to another SaaS.
Transfer out of the cloud.
101
Testing Migration The following general acceptance criteria will serve as
a basis for the migration test objectives:
Interruption to business process is minimal.
All migrated data can be tracked.
All data is converted correctly.
Defects in data before migration do not cause problems during migration.
No unnecessary data is migrated to the service.
102
Testing Migration The test robustness depends on the following factors:
How robust is the service for unexpected contents in the imported data?
How important is the migrated data for the proper functioning of the service?
How extensive are the changes in the way in which data is used?
103
Testing Migration Migration testing must contain the following:
The most important business processes. Process-oriented test, turned to the (possibly altered) user
process.
The correct import of data (is the data correct?).
The potentially changed way in which data is used.
The correct build-up of data history (correct order, with correct date and time).
104
Testing MigrationData Migration in IaaS and PaaS Data Migration in SaaS
1. There is the possibility that the data is moved with the hardware (such as setting a private cloud). No data migration is occurring. However, E2E and regression tests are needed.
2. When there is a new infrastructure and the data is copied to the cloud, a test needs to be executed to make sure the data transfers correctly.
1. When migrating data to SaaS, a data conversion is needed.
2. It can be performed by fully rebuilding the data by using the new (SaaS) application to manually enter the data. This method takes a lot of time.
3. A conversion software tool can be used.
105
Testing Migration The migration to a new test environment should be
also considered:
Transfer and migrate test data (the same steps as with production data).
Configure interfaces to other test systems.
Organize access to the new environments.
Configure administration for the new test environments (performed by supplier).
Test the test environment.
106
Testing due to legislation and regulations Organizations need to take legislation and regulations into
account when configuring the IT landscape.
There are legislations for:
Financial accountability
Pharmaceuticals
Medical care
Telecoms
On the most common legislations is Privacy legislation.
107
Testing in production Continuity in production in the case of changes
Services and services environments are always subject to change.
The following changes can occur once the service is live:
Changes in the service:
Announced changes – E2E regression test on the high-risk parts of the service
Retrospective information about changes – E2E regression test on the high-risk parts of the service
No information – Regularly perform the E2E regression testing on the parts with the highest risk
108
Testing in production Changes in other systems:
E2E regression test
Regression test targeting the changes
Changes at the supplier:
Reevaluate the supplier against selection criteria
Changes in the business processes:
E2E test with revised test cases to take into account the new procedures
Changes in connected resources:
E2E test can uncover problems in the interaction of the resources with the service in a test environment
109
Thank You!
110