Upload
qualitest-group
View
166
Download
1
Tags:
Embed Size (px)
Citation preview
Testing - The Gatekeeper for Quality in NFV
2
Bryon LowenVice-President Telco at
QualiTest
Benny Sand NFV Strategy Lead at
QualiTest
Webinar Presenters
3
Agenda
NFV – Present & Future Risks and Challenges in NFV Testing
Summary & Questions Creating an NFV test strategy
Building SDN / NFV Assurance Process
Introduction
4
Agenda
NFV – Present & Future Risks and Challenges in NFV Testing
Summary & Questions Creating an NFV test strategy
Building SDN / NFV Assurance Process
Introduction
5
About QualiTest Group
| World’s 2nd Largest Independent Testing Company
| QualiTest was recognized as the only visionary pure play QA and testing Service Company in the “ Magic Quadrant”
6
Partial list of Telecom Customers
7
Agenda
NFV – Present & Future Risks and Challenges in NFV Testing
Summary & Questions Creating an NFV test strategy
Building SDN / NFV Assurance Process
Introduction
8
OPEX impact
| Shared computing resources
| Reduced power, space, cooling requirements
| Executing trials of new services without incurring much risk
| Dynamic recovery from failures
| Proof of Concepts (POCs) and trials can be run faster
9
CAPEX impact
| The use of commodity servers reduces hardware costs
| No dependence on specialized hardware
| A single, common server architecture can be used within the data center environment
| Migration to a distributed cloud architecture allows operators to deploy backup in a cost-effective way
| Cycling the hardware more frequently to improve the overall performance of the network
10
Predications
| NFV planning phase is shifting gears
| Operators are moving along the NFV curve
| Finical forecasts indicate a significant growth in NFV investments
| NFV will start to influence budgets and purchasing decisions
| NFV Adaption pace is progressing faster then expected
11
Agenda
NFV – Present & Future Risks and Challenges in NFV Testing
Summary & Questions Creating an NFV test strategy
Building SDN / NFV Assurance Process
Introduction
12
Strategic Challenges
| NFV enables accelerated time-to-market
| NFV enables organizations to implement services anywhere in the world at any time
| NFV involves very complex integration of various software elements
| Everything known becomes unknown
| NFV is more than a mere shift of technologies; it’s a cultural transformation in Telecom organizations
13
Testing Challenges
| Network testing expertise versus Virtualization and Software testing
| Agile orientation will be introduced in Network testing
| Variety of NFV infrastructures requires more testing from the carriers
| Frequent updates in the Network services will require more management and orchestration testing
| More test automation is required to support agility, impact testing and regression testing
| NFV test strategies are needed to accelerate and ensure the delivery of NFV solutions with guaranteed quality
14
Agenda
NFV – Present & FutureRisks and Challenges in
NFV Testing
Summary & Questions Creating an NFV test strategy
Building SDN / NFV Assurance Process
Introduction
15
Building SDN / NFV assurance process
Build strong assurance process to capitalize SDN / NFV potential
| Renegotiate contracts
| Right Price
| Comprehensive Service Level Agreements (SLA)
| Increasingly Complex
| Not affect Quality of Service (QOS)
16
QualiTest NFV validation process
QualiTest aims to create a unified end-to-end approach for service assurance and network
performance
NFV Management
Orchestration Infrastructure Migration
17
QualiTest NFV validation process
Assure full compliance to ETSI ISG Standards.
•Define NFV Testing process
•Test Requirements•NFV Test Environment
•Risk Assessment•Roles and Responsibilities
•Team Skills and Training
•Test Inputs / Basis
Assure full compliance to NFV requirements for any network equipment provider and streamline the ability of vendors and users to be able to implement compatible NFV solutions.
Test Environment: simulation tools, network analytics, QoS and QoE probes.Automated Certification Process for each VNF before it is introduced to the network
Test management and Orchestration prior to VNF migration, including each component (e.g. Orchestrator, Vnfm, VIM).
Test the VNF architecture and implementation to assure smooth transition.
NFV Test Policy
ETSI NFV ISG Compliance
(NFV-Co)
Selecting Mano Vendors
(NFV-Ven)
NFV Staging Environment
(NFV-SE)
MANO Testing (MANO-T)
VNF Migration (VNFM-T)
18
QualiTest NFV validation process
Assure full compliance to ESTI ISG Standards
ETSI ISG Standards Test procedure
Assuring compliance aims to find the deviations from standards• MANO• VNF/NE
• Analyse ISG current standards • Prepare compliance requirements • Analyse each NFV component / int • Define gap points and analyse • Repeat steps 1-6 for each VNF and NE
NFV Test Policy
ESTI NFV ISG Compliance
(NFV-Co)
Selecting Mano Vendors
(NFV-Ven)
NFV Test Environment
(NFV-TE)
MANO Testing (MANO-T)
VNF Migration (VNFM-T)
19
QualiTest NFV validation process
Assure full compliance to NFV requirements for any network equipment provider and streamline the ability of vendors and users to be able to implement compatible NFV solutions.
NFV Test Policy
ESTI NFV ISG Compliance
(NFV-Co)
Selecting Mano Vendors
(NFV-Ven)
NFV Test Environment
(NFV-TE)
MANO Testing (MANO-T)
VNF Migration (VNFM-T)
ETSI specifications on Network Functions Virtualization
• NFV Infrastructure; Hypervisor Domain• NFV Resiliency Requirements• NFV Architectural Framework• NFV Terminology for Main Concepts in NFV• NFV Infrastructure; Compute Domain• NFV Infrastructure; Network Domain• NFV Service Quality Metrics• NFV Management and Orchestration
• NFV Virtual Network Functions Architecture• NFV Security; Security and Trust Guidance• NFV Performance & Portability Best
Practices• NFV Proofs of Concept; Framework• NFV Infrastructure; Methodology to
describe Interfaces and Abstractions• NFV Security; Problem Statement• NFV Use Cases• NFV Virtualization Requirements
20
QualiTest NFV validation process
Build a comprehensive Test Environment, including Control plane and Data plane workload simulation tools, network analytics, QoS and QoE monitoring probes.
NFV Test Policy
ESTI NFV ISG Compliance
(NFV-Co)
Selecting Mano Vendors
(NFV-Ven)
NFV Test Environment
(NFV-TE)
MANO Testing (MANO-T)
VNF Migration (VNFM-T)
NFV provides great flexibility:
• Build parallel test environments
• Scale up and down compute resources
• Copy any existing environment
Environment build guidelines:
• Management and Orchestration should be similar to the production
• NS and VNF Instantiation should be similar to production environment
• Same level of resources should be allocated.
Test Appliances:
• Simulate workload traffic on the client / server
• Simulate both data plane & control plane traffic
• Measure key metrics data & control plane
Virtual Appliances vs. Physical Test
Appliances:• Virtual test appliances
are more cost effective • Physical test appliances
are recommended for highest levels of data-plane performance or microsecond-level timing accuracy
21
QualiTest NFV validation process
Test management and Orchestration prior to VNF migration, including each component (e.g. Orchestrator, VNFM, VIM).
NFV Test Policy
ESTI NFV ISG Compliance
(NFV-Co)
Selecting Mano Vendors
(NFV-Ven)
NFV Test Environment
(NFV-TE)
MANO Testing (MANO-T)
VNF Migration (VNFM-T)
The first step is to test NFV Management and Orchestration Architectural Framework
NFV Orchestrator
VNF Manager
(VIM) and NFV Infrastructure (NFVI)
VIM– VNF Manager (Vi-Vnfm)
Orchestrator – VIM (Or-Vi)
The VNF is deployed as one
or more Virtual Machines;
src ETSI GS NFV 002 v1.2.1
22
QualiTest NFV validation process
Test the VNF architecture and implementation to assure smooth transition.
NFV Test Policy
ESTI NFV ISG Compliance
(NFV-Co)
Selecting Mano Vendors
(NFV-Ven)
NFV Test Environment
(NFV-TE)
MANO Testing (MANO-T)
VNF Migration (VNFM-T)
Static testing VNF-Mano Integration
Conformance testing VNF Internal functionality
Scalability/Performance End-to-end/system test
23
Agenda
Risks in testing NFV & SDN
Challenges with NFV – what, when and why to verify with SDN/NFV?
Summary & Questions Creating an NFV test strategy
Building SDN / NFV assurance process
Introduction
24
NFV framework – Test areas
Overall, the NFV framework consists of three main components.
1.Virtualized Network Functions (VNF)
2.NFV Infrastructure (NFVI)
3.Network Functions Virtualization Management and Orchestration Architectural Framework (NFV-MANO Architectural Framework)
25
Example:MMS VNF Test Strategy
Multimedia Messaging Service (MMS) is similar to the Short Message Service (SMS) in that a message is passed between an Originating Party (OP) and a Destination Party or Parties (DP).
The core Network Element is the Multimedia Messaging Service Centre (MMSC) which acts as a central store and forward server between the OP and DP, as shown below.
26
MMS VNF Test strategy - Approach
| Static testing (drive defect curve)
| Scope (requirements/functionality of VNF)
| Shape test (risk based / ISO9126)
| Test environment (NFV-TE):| Simulation tools
| Network analytics
| QoS / QoE probes
| Onboarding Automated Certification Process.
27
Test Phases – FAT and NFVI int
Factory Acceptance TestingCompliance testing for each VNF introduced to the network:
• Step 1: Analyze ISG current standards• Step 2: Prepare compliance requirements• Step 3: Analyze each NFV component and interfaces• Step 4: Define gap points and analyze• Step 5: Prepare correction plan• Step 6: Monitor correction plan execution• Step 7: Repeat steps 1-6 for each VNF and NE
Requirements coverage
NFVI –test facility integrationEnsure full validation against NFVI in labs
VNF compliance and End2end testing
VNF compliance testing:VNF Compliance Criteria – functional criteria: • VNF Design Patterns• VNF Update and Upgrade• VNF's Properties• VNF Topological Characteristics• VNF States and Transitions• VNF Fault Management.
End to end testing:• Test service end to end to ensure full integration• Testing VNF performance together with other VNFs & non VNF• Validate that introducing the new VNF doesn’t affect other VNF’s
functionality and performance.
29
MMS VNF Test strategy – example Metrics1 NFV Service Metrics
VM Provisioning Latency, instantiation latency. Time between VM instantiation and first available packetVM Stall (event duration and frequency)VM Scheduling Latency
2 QOS and Data-plane metrics:Latency on each of tens of thousands of data streamsThroughput and forwarding rateFrame loss ratePacket-delay variation and short-term average latencyDropped frames and error framesService Disruption Time for Fail-over Convergence
3 QoE and Control-plane metrics:HTTP: page load time, load time varianceVideo: MOS-AV score, range = 2-5 with 5 being the bestHTML5 video - AS score, 100 % score as the maximumDirect metrics:
Peak Signal to Noise Ratio (PSNR)Structural Similarity (SSIM) - compare the original image with the received imageVideo Quality Metric (VQM)Mean Opinion Score (MOS) - This metric combines delays, perceived jitter at application layer, codec used for communication and packet loss at application layer
Indirect metrics:Startup time: Time difference between sending the request for content and the time when the user actually received the contentDelivery synchronization - In a multicast many-to-many scenario it is important that the content is received by all participants at the same time. Freshness: The time difference between the time when the content is actually generated and the time when the users receives it.Blocking: When the buffers on the receiver are empty and the user has to wait for content.
Connections establishment rate, and transactions per secondTotal number of connections, round trip time and goodput
30
Non Functional -Load and Migration
Load / Performance
• Ensure auto-scale works properly
• Ensure resources consumption is efficient
• Measure performance meets SLA in high performance workloads
Failover Convergence testing• Convergence measurement involves the measurement of
processing time of the trigger event in the control plane and the traffic switchover time.
• It is important in a multiple VNF deployment scenario that the convergence time of any VNF is not impacted by the other VNFs on the same physical server.
Level % of BHTR Used forLow 5% Functional Testing
Average 50% Scalability, on-going, fail-over
Busy 100% PerformanceStress 150% Stress Testing, auto-scaling
31
Test strategy goals
| Build an NFV ecosystem
| Drive quality and governance
| Continuous integration model
| Reduce cost of integration
32
Agenda
NFV – Present & Future Risks and Challenges in NFV Testing
Summary & Questions Creating an NFV test strategy
Building SDN / NFV Assurance Process
Introduction
33
Summary
| NFV allows agile real-time shifting of assets to where they are needed
| Testing, in the new world of NFV, changes dramatically and becomes a core discipline that is central to the overall goal of pleasing customers in a cost-effective way
| NFV –testing is the change agent from the physical networks to the virtualized ones
| NFV – Testing challenges deep interdependencies and lightning fast technology introductions; hence it becomes the quality gatekeeper for NFV
| If an operator is to find benefits in the new technology, they primarily want reliable services, at the right price, supported by strong SLAs.
| The move to a software-centric architecture means becoming more software-focused (skills pivot)
| Move towards network DevOps
34
Q&A
www.QualiTestGroup.com