Service Overview - HUAWEI CLOUD

Cloud Performance Test Service

Service Overview

Issue 01

Date 2021-04-29

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2021. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without priorwritten consent of Huawei Technologies Co., Ltd. Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei andthe customer. All or part of the products, services and features described in this document may not bewithin the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,information, and recommendations in this document are provided "AS IS" without warranties, guaranteesor representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

Issue 01 (2021-04-29) Copyright © Huawei Technologies Co., Ltd. i

Contents

1 What Is Cloud Performance Test Service?......................................................................... 1

2 Advantages............................................................................................................................... 4

3 Application Scenarios............................................................................................................. 6

4 Notes and Constraints............................................................................................................ 9

5 Basic Concepts........................................................................................................................12

6 Related Services.....................................................................................................................15

7 Billing....................................................................................................................................... 18

Cloud Performance Test ServiceService Overview Contents

Issue 01 (2021-04-29) Copyright © Huawei Technologies Co., Ltd. ii

1 What Is Cloud Performance Test Service?

Distributed architecture and microservice technologies have caused applications tobecome increasingly complex. This complexity results in architecturedeconstruction and performance improvements. However, it also brings thechallenges of difficulty locating performance problems in the productionenvironment and long repair periods. Performance tests in advance of applicationlaunches are therefore necessary.

Cloud Performance Test Service (CPTS) provides performance test services forcloud applications using HTTP, HTTPS, TCP, or UDP. CPTS performs rapidsimulation of service peaks with large numbers of concurrent users. It allows youto define the contents and time sequences of packets and supports differentcombinations of multiple transactions for complex scenario tests. CPTS simplifiesperformance pressure tests, focuses on services and performance problems,reduces costs, improves stability, optimizes user experience, and maximizes thebusiness value of enterprises.

Cloud Performance Test ServiceService Overview 1 What Is Cloud Performance Test Service?

Issue 01 (2021-04-29) Copyright © Huawei Technologies Co., Ltd. 1

Figure 1-1 Cloud Performance Test Service (CPTS)

FunctionsCPTS provides tests for HTTP/HTTPS/TCP/UDP-based applications with high userconcurrency. It allows you to flexibly define multi-protocol packet contents,transactions, and test task models. CPTS also allows you to view performancestatistics, such as concurrency, transactions per second (TPS), and response latencyin real time or in offline mode. You can also create private test clusters or scale inor out resource groups for performance tests of different scales.

Multi-protocol and high-concurrency performance tests

● Quickly define standard HTTP/HTTPS/TCP/UDP packet contents. You can sendpressure test traffic to different tested applications through simpleadjustment.Based on the service requirements of tested applications, CPTS allows you todefine any fields in HTTP/HTTPS/TCP/UDP protocol packets, for example,configuring and editing the GET and POST methods of HTTP, as well as theuniform resource locator (URL), header, and body fields.

● Define the behavior of virtual users for different test scenarios.Specify the interval for sending requests of the same user by setting the thinktime, or define multiple request packets in a transaction to set the number ofrequests initiated by each user per second.

● Customize the response result verification to provide more accurate standardsfor determining successful requests.CPTS allows you to configure checkpoints based on your service requests.After obtaining response packets, CPTS verifies their response code andheader fields. Only response packets meeting the specified conditions areconsidered normal responses.



Defining test task models for complex scenario tests

● With multiple flexible combinations of transaction elements and test taskphases, CPTS allows you to test application performance in scenarios withdifferent user operation types and concurrent user operations.

● A transaction can be used by multiple test tasks, and multiple test phases canbe defined for a transaction. In each test phase, you can define the testduration, number of concurrent users and tests, as well as simulate complexscenarios with different traffic peaks and troughs.

Figure 1-2 Complex scenario tests

Providing professional performance test reports for easy understanding ofapplication performance

● CPTS collects application performance statistics from multiple aspects, such asTPS, number of concurrent users, response latency, number of accumulatedaccess requests, response verification results, and response timeout interval.

● CPTS provides real-time and offline test reports for you to view and analyzetest data at any time.

Managing private test clusters, isolating tenants' traffic, and providing on-demand use

● Using CPTS, you can create test clusters as required to isolate the traffic ofdifferent users and perform internal and external network pressure tests ofapplications. After the tests are complete, the test clusters can be deleted atany time.

● CPTS supports real-time scale-ins, scale-outs, and upgrades of test clusters.



2 Advantages

CPTS provides a one-stop performance test solution, helping you identifyperformance bottlenecks of applications in advance.

Cost-Efficient Simulation of Ultra-High Concurrency● CPTS provides you with private test clusters. In such a test cluster, a single

execution node can simulate tens of thousands of virtual users, and the entiretest cluster can simulate millions of virtual users.

● CPTS simulates millions of instantaneous concurrent requests. In this way, youcan identify application performance bottlenecks in high concurrencyscenarios and prevent system breakdown caused by large numbers of accessrequests. CPTS is easy to operate and greatly reduces test time.

● CPTS supports execution of multiple concurrent tasks. It enables you to testthe performance of multiple applications at the same time, greatly improvingtest efficiency.

Flexible and Fast Performance Testing, Achieving Quick Application Rollout● Flexible protocol customization: HTTP/HTTPS tests are used to test the

performance of various applications and microservice interfaces developedbased on the HTTP/HTTPS protocol. TCP/UDP tests support the string andhexadecimal code stream modes, which meet the data constructionrequirements of various non-HTTP protocols.

● Flexible combination of multiple transaction elements and test task phases:CPTS provides flexible definition of data packets and transactions, as well assimulates scenarios where multiple users perform transaction operationsduring traffic peaks and troughs of test tasks. All of these features make CPTSideal for complex scenario tests. In addition, CPTS allows you to specify thenumber of concurrent users for each transaction at each period and simulatesinstantaneous service traffic.

On-demand Use of Resources in Performance Tests● Cloud Container Engine (CCE) resource group: Create test clusters as required

to isolate the traffic between tenants and complete pressure tests on theinternal or external network. After tests are complete, you can delete clustersat any time. CPTS supports real-time scale-ins, scale-outs, and upgrades oftest clusters.

Cloud Performance Test ServiceService Overview 2 Advantages


● Shared (external network) resource group: Use shared resource groups fordebugging or small scale concurrent pressure tests.

Quick Location of Performance Bottlenecks● CPTS provides professional performance test reports to show metrics such as

transaction concurrency, TPS, throughput, and response latency, metrics thatworth attention in order to provide a pleasant use experience. CPTS providesonline and offline reports, allowing for analysis of test data at any time.

● CPTS seamlessly interconnects with Application Performance Management(APM) and Application Operations Management (AOM). Using intelligentanalysis, CPTS associates multiple monitored objects, and displays resourceusage of applications, application tracing, and full-link topology. You can viewthe running statuses of applications in real time, and quickly locateperformance bottlenecks.

Cloud Performance Test ServiceService Overview 2 Advantages


3 Application Scenarios

CPTS provides distributed pressure tests and is widely used in various industries,such as the Internet, digital marketing, Internet of Vehicles (IoV), and finance.

E-Commerce Flash Sale TestsE-commerce flash sales are characterized by large-scale user concurrency, multipleburst requests, and repeated access attempts. Guaranteeing the availability ofwebsites under heavy load is key.

Advantages

● Scenario simulation: CPTS simulates millions of instantaneous concurrentrequests in seconds, and simulates a heavy-load website in a test model.

● Professional test report: CPTS provides statistics on the response latency rangethat accurately reflect user experiences.

● Retry for failed users: User-defined comparison of results calculated usingexpressions allows users who failed to log in to retry.

Figure 3-1 E-commerce flash sale tests

Cloud Performance Test ServiceService Overview 3 Application Scenarios


Game Peak Tests

Game services feature auto scaling in peak and off-peak scenarios. You can verifyif auto scaling of games is normal, and if KPIs meet requirements in burst trafficscenarios.

Advantages

● Multi-scenario combination simulation: CPTS simulates real scenarios bycombining multiple transactions, which include diverse elements, andcustomizing packets.

● Peak and off-peak scenario simulation: CPTS develops a pressure test curvefor each transaction within a defined period to simulate peak and off-peakscenarios.

● KPI measurement: You can verify game KPIs in a peak scenario based on acustomized response timeout interval.

Figure 3-2 Game peak tests

Complex Scenarios Support

CPTS simulates all the complexities of real traffic: A user's access may involvemultiple HTTP requests; different users perform different transaction operations;user access fluctuates with transactions, displaying a sharp peak and off-peaktrend; there may be instantaneous concurrent users. Performance tests must beperformed on services to identify performance bottlenecks in advance.

Advantages

● Flexible model customization: CPTS supports multi-transaction tests in whichmultiple users perform multiple operations.

● Burst traffic: CPTS allows you to specify the number of concurrent users foreach transaction at each period and simulates instantaneous service traffic.

● Result verification: Customized comparison of results calculated usingexpressions helps you customize transaction standards.



Figure 3-3 Complex scenarios support

Application Performance OptimizationCPTS allows you to build performance test models, transmit simulated traffic toapplications with CPTS executors, view resource monitoring and tracing status ofapplications, and monitor concurrent processing capabilities to facilitateperformance optimization.

Advantages

● Flexible expansion: CPTS supports multiple performance tests and on-demandexpansion of executor clusters.

● One-stop solution: CPTS can record and report concurrency capabilities,response latency, CPU or memory usage, and microservice processing latency.

● Pay-per-use: You are charged based on the duration of your performance testand the scale of concurrency.



4 Notes and Constraints

Before using test clusters, you must create the resources required for deployingtest clusters through CCE. You are advised to create nodes of the correspondingspecifications in CCE based on the number of concurrent users required for apressure test. At least two nodes must be created for a CCE cluster. One is themanagement node (node used to manage execution nodes). The other is theexecution node (the target machine that a pressure test will be performed on andcan provide performance data during testing).

NO TE

For a CPTS test project, if the number of concurrent users is less than 1000, you can use ashared resource group for testing and do not have to create a resource group. For a JMetertest project, you need to create a resource group regardless of the number of concurrentusers.

Recommended Node SpecificationsCreate nodes of the corresponding specifications in CCE based on the number ofconcurrent users required in a pressure test. For details about specifications, seeTable 4-1.

Table 4-1 Recommended node specifications

Number ofConcurrent Users

Specifications Quantity

0-5000 Management node: 4vCPUs | 8 GB

1

Execution node: 4 vCPUs| 8 GB

1

5001-10,000 Management node: 4vCPUs | 8 GB

1


1

Cloud Performance Test ServiceService Overview 4 Notes and Constraints


https://www.huaweicloud.com/intl/en-us/product/cce.html

Number ofConcurrent Users

Specifications Quantity

10,001-20,000 Management node: 4vCPUs | 8 GB

1


2


1


3


1


4


1


5

More than 50,001 Management node: 4vCPUs | 8 GB

1


nNOTE

Each execution node (8 vCPUs |16 GB) supports 10,000concurrent users.

RestrictionsWhen using CPTS, note the following quota restrictions. For details, see Table 4-2.For details about quotas, see Quotas.

Table 4-2 Restrictions

Parameter Description Default Value

Maximum number ofconcurrent users in atask

Restriction on thenumber of concurrentusers in a task

1000000

Quota for the number ofinstance resource groups

Restriction on thenumber of instanceresource groups

5



https://support.huaweicloud.com/intl/en-us/usermanual-iaas/en-us_topic_0040259342.html

Parameter Description Default Value

Quota for the number ofproject transactions

Restriction on thenumber of transactionsin a project

100

Quota for the number oftransaction elements

Restriction on thenumber of elements in atransaction

40

Quota for the number ofprojects

Restriction on thenumber of tenantprojects

100

Quota for the number oftasks

Restriction on thenumber of tasks in aproject

200

Quota for the number oftasks in a sharedresource group

Restriction on thenumber of tasks in ashared resource group

1000

Quota for the number ofrunning tasks in a sharedresource group

Restriction on thenumber of running tasksin a shared resourcegroup

2

Quota for the runningduration of a task in ashared resource group

Restriction on therunning duration of atask in a shared resourcegroup

3600

Quota for the number offile variables

Restriction on thenumber of file variables

100



5 Basic Concepts

Execution NodeAn execution node is a pressure target machine that provides performance dataduring a pressure test.

Management NodeA management node is used to manage execution nodes.

Test ProjectCPTS manages test projects created by users. Test transactions, pressure test tasks,and test reports are shared in a test project. Users can create test projects fordifferent test programs.

Figure 5-1 Creating a test project

TransactionA transaction indicates a user-defined test operation model, which includes fourparts: think time, packet, response extraction, and checkpoint. HTTP, HTTPS, TCP,or UDP packets are supported.

PacketPackets are data blocks transmitted between HTTP-based applications. These datablocks start with text metadata that describes the packet content and meaning.

Cloud Performance Test ServiceService Overview 5 Basic Concepts


Optional packet data follows the text metadata. Packets are transmitted betweenthe client, server, and agent.

Think TimeTo better simulate user behavior, insert a waiting time between differentoperations. For example, when a user receives data from the server, the user maywait several seconds before viewing data and providing responses. This period oftime is called think time.

Response ExtractionIf a transaction contains multiple packets, the output of the previous packet,which is extracted by a regular expression, is used as the input of the next packet.

CheckpointCheckpoints are where you define the verification information to determinewhether the contents returned by the server are correct.

Test TaskA test task is a performance test activity initiated based on a defined test model.

Test ReportAfter a test task is complete, CPTS generates a test report, which consists of fiveparts: TPS of each metric, total number of each metric, bandwidth, responselatency ratios, and TP response latency.

Number of Concurrent UsersThe number of concurrent users refers to the number of users performing serviceoperations on the system at the same time. In CPTS, it is the number of virtualusers set when you define test task phases.

TPSTPS refers to the real-time statistics of case responses obtained by CPTS from thetested server every second of the statistical period. TPS = Number of normalreturns in the statistical period/statistical period.

BandwidthRecords the real-time bandwidth usage during the running of the pressure testtask. Uplink bandwidth refers to the speed at which the CPTS execution nodesends out data. Downlink bandwidth refers to the speed at which the CPTSexecution node receives data.

Response TimeResponse time indicates the duration from the time when a client sends a requestto the time when the client receives a response from the server.



Response TimeoutIf the corresponding TCP connection does not return the response data within theset response timeout (5 seconds by default), the transaction request is considereda response timeout. Possible causes are: the tested server is busy, in crashes, or thenetwork bandwidth is fully occupied.

Verification FailureThe response packet content and response code returned from the server do notmeet the expectation (the default expected response code of HTTP/HTTPS is 200),such as code 404 or 502. A possible cause is that the tested service cannot beprocessed normally in scenarios with a large number of concurrent users. Forexample, a database bottleneck occurs in the distributed system or the backendapplication returns an error.

Resolution FailureAll response packets are received, but some packets are lost. As a result, the entiretransaction response is incomplete. In this case, network packet loss may be thecause.



6 Related Services

CPTS works with other cloud services to provide you with performance tests.Relationship with other cloud services shows the relationship between CPTS andother cloud services.

Figure 6-1 Relationship with other cloud services

CCE

CCE provides highly scalable, high-performance, enterprise-class Kubernetesclusters and supports Docker containers. With CCE, you can easily deploy, manage,and scale containerized applications on HUAWEI CLOUD.

CPTS uses CCE to create nodes that serve as the management and executionnodes in CPTS pressure test resource groups.

Cloud Performance Test ServiceService Overview 6 Related Services


https://support.huaweicloud.com/intl/en-us/cce/index.html

AOMAOM is a one-stop, multi-dimensional O&M management platform for cloudapplications. It monitors applications and related cloud resources in real time.AOM collects and associates resource metrics, logs, and events to analyzeapplication health statuses, as well as provides flexible alarm reporting and datavisualization. This helps you detect faults in a timely manner and monitor runningstatuses of applications, resources, and services in real time.

You can use AOM to comprehensively monitor and uniformly manage servers,storage devices, networks, web containers, and applications hosted in Docker andKubernetes. This prevents problems effectively and helps O&M personnel locatefaults in minutes, reducing O&M costs. In addition, AOM supports interconnectionwith self-developed monitoring systems or report systems through unified openAPIs. Unlike traditional monitoring systems, AOM monitors applications to reflectservice statuses. This meets enterprise requirements for efficiency and fastiteration, helping enterprises support services through IT, achieve strategicobjectives, and protect and optimize IT assets.

CPTS uses the resource monitoring provided by AOM, which is useful forintelligent analysis.

CTSCloud Trace Service (CTS) is a log audit service for cloud security. It allows you tocollect, store, and query operation records of cloud resources for security analysis,compliance auditing, resource tracking, and fault locating.

CTS can record operations associated with CPTS for future query, audit, andbacktracking.

AOSApplication Orchestration Service (AOS) enables you to deploy your applicationson HUAWEI CLOUD with a few clicks, simplifying cloud service management.Using templates to describe and orchestrate applications and related cloudservices, AOS facilitates automatic application deployment, cloud service creation,and E2E application lifecycle management.

After a resource group is created in CPTS, the cpts-stack stack is generated inAOS.

APMAPM monitors and manages the performance of cloud applications in real time.APM provides performance analysis of distributed applications, helping O&Mpersonnel quickly locate and resolve faults and performance bottlenecks.

APM is a cloud application diagnosis service with powerful analysis tools. Itdisplays the application statuses, call processes, and user operations throughtopologies, tracing, and transactions, allowing you to quickly locate and resolvefaults and performance bottlenecks.

CPTS leverages APM to trace and monitor applications deployed on APM. On theCall Chain page of APM, you can view the call chain information of applications.Red indicates abnormal, yellow indicates alarm, and green indicates normal.



https://support.huaweicloud.com/intl/en-us/aom/index.html

https://support.huaweicloud.com/intl/en-us/cts/index.html

https://support.huaweicloud.com/intl/en-us/aos/index.html

https://support.huaweicloud.com/intl/en-us/apm/index.html

OBSObject Storage Service (OBS) is a secure, reliable, and cost-effective cloudstorage service. With OBS, you can easily create, modify, and delete buckets, aswell as upload, download, and delete objects.

The file variables of CPTS can be stored in OBS.

SWRSoftware Repository for Container (SWR) provides easy, secure, and reliablemanagement over container images throughout their lifecycles, facilitating thedeployment of containerized services.

In CPTS, SWR is used to obtain the images used for pressure tests in clusters.



https://support.huaweicloud.com/intl/en-us/obs/index.html

https://support.huaweicloud.com/intl/en-us/swr/index.html

7 Billing

Billing ItemTwo types of costs will be generated when you run performance tests in CPTS: thecost for using CPTS and the cost for using resources of other cloud services, suchas Elastic Cloud Server (ECS).

● The resources are used to deploy clusters and are billed by ECS.Nodes of a pressure test resource group need to be created on CCE inadvance. For details, see Buying a Node. Nodes in a resource group are billedby the corresponding ECS.

● CPTS is billed based on the virtual user minutes (VUMs) consumed in pressuretests. A VUM is the number of virtual users multiplied by the number ofminutes in pressure tests. For details, see Pricing Details.

Billing ModePay per use

In the pay-per-use billing mode, you pay only for what you use in pressure tests.

Note:

To perform a pressure test with a peak concurrency larger than 100,000, you needto submit a service ticket.

1. Access Support Center.2. In the navigation pane on the left, choose Service Tickets > Create Service

Ticket.3. Click here, enter the service ticket information, and click Submit.

Cloud Performance Test ServiceService Overview 7 Billing


https://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_01_0033.html

https://www.huaweicloud.com/intl/en-us/pricing/index.html?tab=detail#/cpts

https://console-intl.huaweicloud.com/ticket/?agencyId=06290025418026bb1f64c01882f0c4d0&locale=en-us#/ticketindex/createIndex

Figure 7-1 Creating a service ticket

ArrearsIf your account is in arrears after a pressure test task is executed, you cannot useCPTS. If you top up your account to repay the amount overdue within theretention period (15 days), the offline report data can be retained. Otherwise, thedata will be deleted. Top up your account in time to prevent the data from beingdeleted.

Cloud Performance Test ServiceService Overview 7 Billing


Documents

Service Overview - HUAWEI CLOUD