CLOT KPI Analysis Guideline(RJIL)_v1.0

8/16/2019 CLOT KPI Analysis Guideline(RJIL)_v1.0

1/135

CLOT KPI Analysis Guideline

(RJIL)

March, 2015

Solution Lab.


2/135

Responsible for this document:

배경

• 본 문서를 통해, RJIL CLOT 시험결과에 대한 원활한 KPI Pass/Fail 분석 업무 수행을 위해초기 KPI Fail 사례 등을 바탕으로 작성 하였음

• 작성시점 기준 ActixOne을 활용한 분석이 용이하지 않아, 향후 AxtixOne을 활용한 분석방법 기술 필요(VoLTE 관련항목등)

• 본 문서는 프로젝트 목표 달성을 위해 관련자 모두에게 공유 가능하며, 수정사항 필요시수정및추가가능함

작성범위• 본 문서는 KPI 기준 및 ActixOne/XCAP/Anyplan-Optimizer활용 분석 방안을 제시 함

• Reference CLOT 진행시발생한 KPI Fail 사례를 통해 이슈를 정의 함

Change History:

1© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

Changes Author Date Version

Initial Version Joohyun Kim/Jaewook Jung 29.03.2015 1.0


3/135

Table of contents CLOT KPIs

How to Analyze KPI Items

• RSRP

• SINR (Info)

• Overlapping Servers within 5dB

• PCI for all Sectors Observed (Info)

• Mac DL Throughput >= 2Mbps

• Mac UL Throughput >= 256kbps

• Average Mac DL Throughput

• Average Mac UL Throughput

• Call Drop

• TAU Success Rate (Info)

• Handover Success Rate

• Handover Interruption Time < 100ms

2


4/135

Table of contents How to Analyze KPI Items

• RACH Success Rate

• Attach Success Rate

• VoLTE Setup Success Rate

• Attach Setup Time < 1.5s

• IMS Registration Time < 1.5s (Report Only)

• VoLTE Setup Time < 1.5s

• VoLTE Jitter < 40ms

• VoLTE RTP Packet Loss Rate

• Average BLER DL (Info)

•Average BLER UL (Info)

Appendix

3


5/135

CLOT KPIs

4

No KPIs Call Model CategoryTarget

Value

Non-

ExclusionExclusion

1 RSRP LDL1, LUL KPI > 95 % Don’t Consider

2 SINR >= -2 dB LDL1, LUL Info - Don’t Consider

3 Overlapping Servers within 5 dB 60 % Don’t Consider

4 Overlapping Servers within 5 dB 90 % Don’t Consider

5 PCI for all Sectors Observed LDL1, LUL Info - Don’t Consider

6 MAC Throughput DL >= 2 Mbps LDL KPI > 95 % Don’t Consider

7 MAC Throughput UL >= 256 kbps LUL KPI > 95 % Don’t Consider

8 Average MAC Throughput DL (kbps) LDL KPI > 16,500 Don’t Consider

9 Average MAC Throughput UL (kbps) LUL KPI > 2,500 Don’t Consider

10 Call Drop Rate LDL1, LUL, VoL KPI < 1 % Don’t Consider

11 TAU Success Rate LDL1, LUL, VoL Info - Don’t Consider

12 Handover Success Rate LDL1, LUL, VoL KPI > 98 % Don’t Consider

13 Handover Interruption Time < 100ms LDL1, LUL, VoL KPI > 95 % Don’t Consider

14 RACH Success Rate VoS KPI > 99 % Don’t Consider

15 Attach Success Rate VoS KPI > 99 % Don’t Consider

16 VoLTE Setup Success Rate VoS KPI > 99 % Don’t Consider

17 Attach Setup Time < 1.5 s VoS KPI > 95% Don’t Consider18 IMS Registration Setup Time < 1.5 s VoS Info > 95% Don’t Consider

19 VoLTE Setup Time < 1.5 s VoS KPI > 95% Don’t Consider

20 VoLTE Jitter < 40 ms VoL KPI > 95 % Don’t Consider

21 VoLTE Latency < 100 ms VoL KPI > 95 % Don’t Consider

22 VoLTE RTP Packet Loss Rate VoL KPI


6/135

How to Analyze KPI Items

5


7/135

1. RSRP

Description

• Ratio of RSRP which meet ‘Coverage Objective RSRP Threshold’ within exclusion zone

— ‘Coverage Objective RSRP Threshold’ value for each sector is referred from Cell Ref File

※ Sectors with the same clutter category could have different threshold values

— BSP for the on-air sites would be only considered as exclusion zone


Clutter Category Coverage Objective RSRP Threshold

DU -106.8

DU -107.4

DU -108.2DU -109.2

DU -110.5

MU -108.8

MU -107.2

MU -110.2

MU -110.3

MU -110.4

RU -113.6

RU -114.8

RU -116.1

RU -116.2

SU -111.8

SU -112.2

SU -113.2

SU -113.3

SU -113.4

SU -116.3

[Coverage Outage Threshold according to Clutter Category] [BSP as Coverage Polygon]


8/135

1. RSRP

Target Value

• > 95%

Used Call Model

• LDL1, LUL (Merge)

Formula

• (The number of passing RSRP samples / The total number of RSRP samples)

within exclusion zone

— Passing RSRP samples : the binned RSRP samples which meet ‘Coverage Objective RSRP Threshold’


— The total number of RSRP samples : the all binned RSRP samples within exclusion zone



9/135

1. RSRP

Analysis using ActixOne and AnyPlan-Optimizer

• Check KPI value in ActixOne Report

• Find poor RSRP area using ActixOne and AnyPlan-Optimizer

— Check RSRP plots with BSP in AnyPlan-Optimizer

※ Poor RSRP area outside BSP does not have to be considered in the analysis

•

Find the issue eNBs related with poor RSRP area— Check PCI plots in ActixOne (or AnyPlan-Optimizer)



10/135

1. RSRP


• Check the coverage of the issue eNBs

— Check PCI plots in ActixOne(or AnyPlan-Optimizer)

— Execute ‘Sector Swapped Site’ and ‘Tx Off Site’ in AnyPlan-Optimizer

— Check ‘Sector Plot’ for each issue eNB in AnyPlan-Optimizer



11/135

1. RSRP


• Check the issue eNBs alarm status during drive test from LSM

— Check the alarms such as RRH/Site/SNMP Down, ECP/UMP/GPS Function Fail , VSWR etc.


I-OR-SUNA-ENB-9005 Alarm History in eNB

No. Severity Code ProbableCause Alarm time Clear time Clear type

541 Critical 2007Protocol Status Alarm

( Communication Fail : PING )2015-02-03 12:37 2015-02-03 12:43 Auto

556 Critical 2007Protocol Status Alarm

( Communication Fail : PING )2015-02-03 11:49 2015-02-03 11:49 Auto


12/135

1. RSRP


• Execute ‘AOP’ in AnyPlan-Optimizer

• Fine-tune ‘AOP’ recommendation manually if necessary

• Check the prediction results

— RSRP as well SINR and Serving Sector area should be also checked together

※ Refer to ‘AnyPlan-Optimizer User Guideline’ for the details

※ If RSRP KPI is not improved after finishing ‘AOP’/’Modify Sector Parameter’,

it should be checked UE’s normality or the measurement environments such as obstacles


Baseline

Prediction


13/135


14/135

Description

• Ratio of SINR that meets ‘SINR Outage Threshold’ value (-2dB) within exclusion zone

Target Value

• -

Used Call Model


Formula

• (The number of passing SINR samples / The total number of SINR samples)


— Passing SINR sample : the binned SINR samples which meet ‘SINR Outage Threshold’ value (-2dB)’


— The total number of SINR samples : the all binned SINR samples within exclusion zone

2. SINR (Info)



15/135



• Analyze poor SINR area

: Analysis methodology would be the similar with RSRP analysis

※ Refer to ‘RSRP analysis’ and ‘AnyPlan-Optimizer User Guideline’ for the details

— Find Poor SINR area— Find issue eNBs and check alarms

— Execute ‘AOP’/’Modify Sector Parameters’ in AnyPlan-Optimizer etc.

ex) removing dominant interference or enhancing the serving signal

— Check the prediction results

Analysis Results

• Antenna Recommendation

— Azimuth, M-Tilt and E-Tilt

• The issue eNB lists

※ All alarms not only for issue eNBs but also for all eNBs in the cluster should be

checked and cleared before drive test

2. SINR (Info)



16/135

Improve Point

• Even through RSRP is good, SINR was very low due to missing neighbor

▶ Check Neighbor relation between two cells

2. SINR (Info)

SINR

RSRP

Despite of relatively high RSRP, SINR is

very low due to missing neighbor



17/135

Description

• The average number of sectors having RSRP difference with serving best server within 5 dB,

including best server signal within exclusion zone

※ The number of overlapping sectors is calculated first, then it is averaged by binning

Target Value

• > {60% | 90%} for 1 server and 2 servers respectively

Used Call Model• LDL1, LUL (Merge)

Formula

• (The number of passing overlapping samples / The total number of bin samples)

within exclusion zone— Passing overlapping samples : the binned samples which meet the average number of overlapping

sectors within 5 dB


18/135



• Find overlapping area in ActixOne and AnyPlan-Optimizer

— Check Serving and Neighbor RSRP in ActixOne (or AnyPlan-Optimizer)

— Check ‘Pollution Plots’ in AnyPlan-Optimizer※ If ‘AOP’ recommendation was already derived, check ‘Pollution Plots’ in Prediction layer

3. Overlapping Servers within 5 dB



19/135


• Analyze overlapping area

: Analysis methodology would be the similar with RSRP analysis


— Find Overlapping area

— Find issue eNBs and check alarms


ex) removing dominant interference or enhancing the serving signal

— Check the prediction results

Analysis Results


— Azimuth, M-Tilt and E-Tilt







20/135

Analysis using XCAP

• Analyze overlapping area from XCAP

• Find issue overlapping area

— By XCAP Analyze, try to find the overlapping area



5dB


21/135

Description

• The number of observed PCIs during drive test compared with the number of planned PCIs

for the particular cluster

Target Value

• -

Used Call Model


Formula

• The number of observed PCI during drive test / The number of planned PCIs for the

particular cluster

4. PCI for all Sectors Observed (Info)



22/135



• Check measured PCIs in ActixOne (or AnyPlan-Optimizer)

— Check PCI Plots in ActixOne (or AnyPlan-Optimzier)

— Check ‘Sector Plot’ and ‘Display Type’ function in AnyPlan-Optimizer

• Check the un-measured eNBs alarm status during drive test from LSM


Analysis Results

• The unmeasured on-air eNB lists




4. PCI for all Sectors Observed (Info)



23/135

Description

• Ratio of summed MAC DL Throughput from device1 and device2 which meets the criteria

(2Mbps) within exclusion zone

Target Value

• > 95%

Used Call Model

• LDL1, LDL2 (Separate)

Formula

• (The number of passing DL Tput samples (LDL1) + The number of passing DL Tput samples (LDL2)) /

(The total number of DL Tput samples (LDL1) + The total number of DL Tput samples (LDL2))

within exclusion zone— Passing DL Tput samples : the binned MAC DL Throughput samples which meet 1Mbps within

exclusion zone

— The total number of DL Tput samples : the all binned MAC DL Throughput samples within exclusion

zone

5. DL MAC Throughput >= 2Mbps



24/135


25/135

Analysis using ActixOne, AnyPlan-Optimizer and XCAP

• Find the issue eNBs related with poor DL Throughput area

— Check PCI plots in ActixOne (or AnyPlan-Optimizer)

• Check the issue eNBs alarm status during drive test from LSM


※ Refer to ‘RSRP analysis’ for the details

• Modify Antenna Parameters in AnyPlan-Optimzier (Improve DL Edge SINR)

— Execute ‘AOP’ in AnyPlan-Optimzier

— Execute ‘Modify Sector Parameters’ in AnyPlan-Optimzier if necessary




I-OR-SUNA-ENB-9005 Alarm History in eNB

No. Severity Code ProbableCause Alarm time Clear timeClear typ

e

541 Critical 2007

Protocol Status Alarm

( Communication Fail :

PING )

2015-02-03

12:37

2015-02-03

12:43Auto

556 Critical 2007

Protocol Status Alarm

( Communication Fail :

PING )

2015-02-03

08:53

2015-02-03

11:49Auto

h h b


26/135

• Check other issues in XCAP

— Check missing Neighbors (No HO after multiple MR transmission)

— Check Backhaul issues

‐ Server Drop (Error message from FTP server)‐ Socket Close

‐ Initial attach or IdletoActive fails due to no receiving ‘dlInformationTransfer ’ message

— Check FTP Server issues

Analysis Results


— Azimuth, M-Tilt and E-Tilt• The issue eNB lists



• Missing NBR list etc.



h h b


27/135

Fail Cases

• Poor RF (SINR)

— DL MAC Throughput was low due to the low SINR

▶ RF Optimization



DL MAC Throughput fail

5 DL MAC Th h 2Mb


28/135

Fail Cases

• Missing Neighbors

— DL MAC Throughput was low due to no HO procedure

‐ No HO procedure after multiple MR transmission due to missing Neighbors

‐ RACH procedure by ‘UL Data arrival’ after multiple MR transmission due to missing

Neighbors

▶ Check Neighbor list in LSM (check missing neighbor)



5 DL MAC Th h 2Mb


29/135

Fail Cases

• Interference issue

— DL MAC Throughput was low due to UL interference

‐ eNB didn’t receive ACK/NACK for FTP DL data on PUSCH due to high IoT in this eNB

▶ Check Interference


© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

LDL Low DL Throughput

LDL BLER

LDL PUSCH Tx Power

LDL Power Headroom


5 DL MAC Th h 2Mb


30/135

Fail Cases

• Late Service Request Triggering

— UE triggered service request too late after RRC Connection Release, so there happened server drop

▶ Check Backhaul and S1 interface



Release

Server Drop

DL Low Throughput

6 UL MAC Th h t 256kb


31/135

Description

• Ratio of MAC UL Throughput which meets the criteria (256kbps) within exclusion zone

Target Value

• > 95%

Used Call Model

• LUL

Formula

• (The number of passing UL Tput samples / The total number of UL Tput samples)


— Passing UL Tput samples : the binned MAC UL Throughput samples which meet 256kbps within

exclusion zone

— The total number of UL Tput samples : the all binned MAC UL Throughput samples within exclusion

zone

6. UL MAC Throughput >= 256kbps


6 UL MAC Th h t 256kb


32/135



• Analyze poor UL Throughput area

: Analysis methodology would be the similar with DL Edge Throughput analysis

— Find poor UL Throughput area



— Check other issues such as missing neighbor, FTP Server issues in XCAP



BSP

6 UL MAC Th h t > 256kb


33/135

Fail Cases

• Poor RF (RSRP)

— UL MAC Throughput was low due to the low RSRP

▶ RF Optimization



UL MAC Throughput fail



34/135

Fail Cases

• Missing Neighbors

— UL MAC Throughput was low due to no HO procedure

‐ After multiple MR transmission, UE didn’t start HO procedure due to missing Neighbors

‐ After multiple MR transmission, UE started RACH procedure by ‘UL Data arrival’






35/135

Fail Cases

• Backhaul issue

— UL MAC Throughput was low due to several backhaul related issues

‐ FTP Server Drop

‐ Socket Close

‐ Small Buffer Size of UE due to small TCP window size of eNB

‐ Late service request after RRC Connection Release

‐ No receiving ‘Security protected NAS message’ message from MME after RRC connection

setup complete in normal RF environment

▶ Check Backhaul and S1 interface



6 UL MAC Throughput > 256kbps


36/135

Fail Cases

• Inherence issue

— UL MAC Throughput was low due to interference(Tx Power : Max, Power Headroom : low, RF

environment : Not bad)

▶ Check interference



UL Throughput

7 Average DL MAC Throughput


37/135

Description

• Summation of the averaged MAL DL Throughput from device 1 and device 2 within

exclusion zone

Target Value

• > 16,500kbps

Used Call Model

• LDL1, LDL2 (Separate)

Formula

• (Average MAC DL Throughput for LDL1 + Average MAC DL Throughput for LDL2) within

exclusion zone

— Average MAC DL Throughput : (summation of the binned MAC DL Throughput samples / the total

number of MAC DL Throughput samples) within exclusion zone

7. Average DL MAC Throughput


7 Average DL MAC Throughput


38/135



• Analyze poor DL Throughput area






7. Average DL MAC Throughput


8 Average UL MAC Throughput


39/135

Description

• The averaged MAL UL Throughput within exclusion zone

Target Value

• > 2,500kbps

Used Call Model

• LUL

Formula

• Average MAC UL Throughput for LUL within exclusion zone

— Average MAC UL Throughput : (summation of the binned MAC UL Throughput samples) / (the total

number of MAC UL Throughput samples) within exclusion zone

8. Average UL MAC Throughput


8 Average UL MAC Throughput


40/135



• Analyze poor UL Throughput area






8. Average UL MAC Throughput


BSP

9 Call Drop Rate


41/135

Description

• Ratio of Dropped VoLTE Calls and Abnormal Release of Data Call

Target Value

•


42/135

Analysis using ActixOne and XCAP


• Check call drop events in ActixOne

— ActixOne does not provide the items which can check VoLTE Call Drop and RRC Drop yet

9. Call Drop Rate


9 Call Drop Rate


43/135


• Check Call drop events in XCAP

— Check the event times for all call drops

• Check the detailed messages in XCAP for the fail events

9. Call Drop Rate


9 Call Drop Rate


44/135

Fail Cases

• Poor RF Environments

— UE didn’t received RTP packets due to poor RF quality

▶ Check RF environment (RSRP/SINR)

9. Call Drop Rate


9 Call Drop Rate


45/135

Fail Cases

• Service Reject

— UE didn’t received RTP packets due to service reject from core network

▶ Check Core network (including IMS Server)

9. Call Drop Rate


Service Reject

9 Call Drop Rate


46/135

Fail Cases


— UE was released from eNB without ‘rrcConnectionRelease’ message due to Out-of-Sync.


9. Call Drop Rate


9 Call Drop Rate


47/135

Fail Cases

• Abnormal release by S1 Interface issue

— UE was released from eNB without ‘rrcConnectionRelease’ message due to S1 interface

▶ Check Backhaul and S1 Interface

9. Call Drop Rate


No ‘rrcConnectionRelease’ message

10 TAU Success Rate (Info)


48/135

Description

• Ratio of Tracking Area Update (TAU) Success

— TAU procedure : from Tracking Area Update Request to Tracking Area Update Success in NAS L3

message

Target Value

• -

Used Call Model• LDL1, LUL and VoL (Separate)

Formula

• The number of TAU Success / The number of TAU Attempts

— TAU Success : Receiving ‘Tracking area update accept’ (DL EPS MM)

— TAU Attempt : Sending ‘Tracking area update request’ (UL EPS MM)

10. TAU Success Rate (Info)




49/135

Message from XCAP

• From Tracking area update request (UL EPS MM) to Tracking area update success (DL EPS MM)



Complete

Attempts



50/135



• Check TAU fail events in ActixOne

— ActixOne does not provide the items which can check VoLTE Call Drop and RRC Drop yet

• Check TAU fail events in XCAP

— Check the specific event time for TAU fails





51/135







52/135

Fail Cases

• No TAU Update Message

— After sending the TAU Message, UE didn’t receive the TAU accept

▶ Check Core network (including MME)


51


TAU Update Fail


11. Handover Success Rate


53/135

Description

• Ratio of Handover Success for the intra-frequency LTE Handover Procedure

— Handover procedure : from RRC Connection Reconfiguration (contains HO command)

to LTE MAC RACH Attempt Success for RRC Connection Reconfiguration Complete

Target Value

• >98%

Used Call Model• LDL1, LUL and VoL (Separate)

Formula

• The number of Handover Success / The number of Handover Attempts

— Handover Success : LTE MAC RACH Attempt Success for ‘rrcConnectionReconfigurationComplete’

— Handover Attempts : Sending ‘rrcConnectionReconfiguration’ (contains HO command)





54/135

Message from XCAP

• From ‘rrcConnectionReconfiguration’ message (with HO Command) to LTE MAC RACH

attempt Success



Complete

Attempts



55/135



• Check ‘LTE Handover Fail’ events in ActixOne





56/135


• Check Handover fail events in Macro and XCAP

— Check the specific event time for HO fail






57/135

Fail Cases

• Failure at MSG2

— UE didn’t receive RAR(MSG2) from eNB due to poor RF environments






58/135

Fail Cases

• RLF during HO

— UE didn’t succeed Handover due to Radio link failure during HO procedure

▶ Check RF Environments (RSRP/SINR)

a do e Success ate




59/135

Fail Cases

• PCI Duplication

— UE didn’t receive RAR(MSG2) due to referring the wrong cell

▶ Check PCI Collision and Confusion in NBR list from LSM


RACH info

SIB2



60/135

Fail Cases

• No Measurement Report

— Measurement parameters are not configured in eNB

▶ Check eNB Parameter or configuration


Measurement parameters are not

configured

12. Handover Interruption Time < 100ms


61/135

Description

• Ratio of Handover Success which meets the criteria (100ms) among the intra-frequency LTE

Handover Success Procedure

— Handover procedure : from RRC Connection Reconfiguration (contains HO command)to LTE MAC RACH Attempt Success for RRC Connection Reconfiguration Complete

Target Value

• >95%

Used Call Model

• LDL1, LUL and VoL (Separate)

Formula

• The number of Handover Success within 100ms / The number of Handover Success

— Handover Success within 100ms : LTE MAC RACH Attempt Success for

‘rrcConnectionReconfigurationComplete’ within 100ms

— Handover Success : LTE MAC RACH Attempt Success for ‘rrcConnectionReconfigurationComplete’

p




62/135

Message from XCAP

• From ‘rrcConnectionReconfiguration’ message (with HO Command) to LTE MAC RACH

attempt Success

p


Complete

Attempts



63/135



• Check ‘LTE Slow Handover’ events in ActixOne

※ Handover Interruption Time fail could be different with ‘LTE Slow Handover’ events in ActixOne

p




64/135


• Check Handover Interruption time fail events using Macro

— Check the specific event time for HO interruption time fail


p



65/135

13. RACH Success Rate


66/135

Description

• Ratio of Contention-based RACH Success

— RACH Procedure (Contention based): from Random Access Request message(MSG1) to Contention

Resolution message(MSG4)

Target Value

• >99%

Used Call Model

• VoS

Formula

• The number of RACH Success / The number of RACH Attempts(CT Based)

— RACH Success : LTE MAC RACH Attempt Success (Success of Message 4)

— RACH Attempts(CT Based) :‐ LTE MAC RACH trigger followed by ‘rrcConnectionRequest’

‐ LTE MAC RACH trigger followed by ‘rrcConnectionReestablishmentRequest’




67/135

Message from XCAP

• Initial RRC Connection Setup

— From LTE MAC RACH trigger by Connection request to LTE MAC RACH attempt Success

• RRC Connection Re-establishment

— From LTE MAC RACH trigger by Radio Link failure to LTE MAC RACH attempt Success


Complete

Attempts

Complete

Attempts



68/135



• Check Contention-based RACH fail events using Macro and XCAP

— Check the event times for RACH fails

※ RACH triggered by RLF should be also counted

• Check the detailed messages in XCAP for the fail event




69/135

Fail Cases

• Failure at MSG2

— UE didn’t receive Random Access Response (RAR) from eNB due to poor RF environments

— RACH will be aborted when T300 timer is expired▶ Check RF environment (RSRP/SINR)




70/135

Fail Cases

• Failure at MSG4 due to CT Timer expired

— UE didn’t receive MSG4 from eNB within the pre-defined CT timer(56sf)



14. Attach Success Rate


71/135

Description

• Ratio of LTE Attach Success

— Attach Procedure : from RRC Connection Setup Complete with Attach Request to IMS Registration

Target Value

• >99%

Used Call Model

•

VoS

Formula

• The number of Attach Success / The number of Attach Attempts

— Attach Success : Receiving SIP 200 OK for IMS Registration after ‘rrcConnectionSetupComplete’

with Attach Request

— Attach Attempts : Sending ‘rrcConnectionSetupComplete’ with Attach Request




72/135

Attach Procedure




73/135

Message from XCAP

• From ‘rrcConnectionSetupComplete’ message (with Attach Request) to SIP 200 OK message

for IMS REGISTER


Complete

Attempts



74/135



• Check Attach fail events in ActixOne

— ActixOne does not provide the items which can check Attach fails yet




75/135


• Check Attach fail events using Macro

— Check the event times for attach setup fails

— Check the event times for IMS registration fails


• Define the issues whether the issue is comes from core network(IMS) or not

— Refer IMS issue code in Appendix



76/135



77/135

Fail Cases

• No Receiving TCP ACK for Sync

— UE didn’t receive TCP Acknowledgement (SYNC+ACK) for IMS Register after Default Bearer Setup

▶ Check Configuration of N7K router▶ Check the stability of Backhaul and S1 interface


No receiving ACK message for TCP Sync.



78/135

Fail Cases

• 403 Forbidden

— UE didn’t receive SIP 200 OK message for IMS registration due to IMS server error

※ Another core side issue codes could be analyzed with the similar way▶ Check Core network (including IMS server)


No 200 OK for Registration



79/135

Fail Cases

• Initial Attach Fail

— After RRC Connection Setup Complete, UE was released due to ‘PDN CONNECTIVITY REJECT’

message from MME

※ Reject Cause : #30 (Request rejected by Serving GW or PDN GW)

▶ Check Core network (including IMS server)


Attach Reject : PDN Connectivity Reject

15. VoLTE Setup Success Rate


80/135

Description

• Ratio of VoLTE Call Setup Success

— VoLTE Setup Procedure : from INVITE to 200 OK for INVITE

Target Value

• > 99%

Used Call Model

•

VoS

Formula

• The number of VoLTE Setup Successes / The number of VoLTE Setup Attempts

— VoLTE Setup Success : Receiving SIP 200 OK message for INVITE

(After a dedicate bearer with QCI 1 is activated)

— VoLTE Setup Attempt : Sending SIP INVIVTE message (with originating SDP)




81/135

VoLTE Setup Procedure




82/135

Message from XCAP

• From SIP INVITE message to SIP 200 OK message for INVITE


P-RACK

Complete

Attempts

200 OK for INVITE



83/135



• Check VoLTE Setup fail events in ActixOne

※ ActixOne does not implement analyzing function for VoLTE Call in detail


Network Optimization

Call Event Explorer



84/135


• Check VoLTE Setup fail events using Macro

— Check the event time for VoLTE setup fails

‐ Long Setup time more than 10 seconds‐ No Setup time

※ The number of fails found from Macro file could be different with ActixOne (Refer to Appendix)


• Define the issues whether the issue is comes from IMS or not





85/135


• Check the fail events found in ActixOne in XCAP also


• Define the issues whether the issue is comes from core network or not— Refer IMS issue code in Appendix




86/135

Fail Cases

• 487 Request Terminated

— UE didn’t receive SIP 200 OK message for invite due to IMS server error

※ Another core side issue code error could be analyzed with the similar way▶ Check Core network (including IMS Server)


No 200 OK for INVITE



87/135

Fail Cases

• 500 Server Internal Error

— UE didn’t receive SIP 183 Session Progress message from IMS due to server error



No 200 OK for INVITE



88/135

Fail Cases

• No 200 OK for PRACK

— UE didn’t receive SIP 200 OK message for P-RACK when RF environment was good

※SIP 200 Status Code : The 200 status code is the ACK message which represents the request wasreceived and processed successfully



No 200 OK for PRACK



89/135

Fail Cases

• No 100 Trying

— UE didn’t receive 100 Trying message after invite even through RF Environment is good





90/135

Fail Cases

• No 100 Trying

— UE didn’t receive 100 Trying message which comes from IMS Server



No 100 Trying between INVITE and PRACK

16. Attach Setup Time < 1.5s


91/135

Description

• Ratio of Attach Setup Success which meets the criteria (1.5s) among the Attach Setup

Success Procedure

— Attach Setup Procedure : from RRC Connection Setup Complete with Attach Request to AttachComplete

Target Value

• >95%

Used Call Model• VoS

Formula

• The number of Attach Setup Success within 1.5s / The number of Attach Setup Success

— Attach Setup Success within 1.5s : Sending ‘Attach complete’ message (UL EPS MM) after

‘rrcConnectionSetupComplete’ with Attach Request

within 1.5s

— Attach Setup Success : Sending ‘Attach complete’ message (UL EPS MM) after

‘rrcConnectionSetupComplete’ with Attach Request




92/135

Attach Setup Procedure




93/135

Message from XCAP

• From ‘rrcConnectionSetupComplete’ (with Attach Request) to Attach Complete (UL EPS MM)


Complete

Attempts



94/135



• Check Attach setup time fail events in ActixOne

— ActixOne does not provide the items which can check Attach setup time fail event yet


l d



95/135


• Check Attach setup time fail events using Macro

— Check the event time for attach setup time fails

: Filter out the cases having more than 1.5s


• Define the issues whether the issue is comes from core network or not




F il C


96/135

Fail Cases

• Security protected NAS message Delay

— UE received Security protected NAS message from MME too late

▶ Check the stability of S1 Interface


16:59:59.819

16:00:02.088

D i i

17. IMS Registration Time < 1.5s (Report Only)


97/135

Description

• Ratio of IMS Registration Success within the criteria (1.5s) among IMS Registration Success

Procedure

— IMS Registration Procedure : from PDN Connectivity Request to 200 OK for IMS Registration

Target Value

• >95% (Report Only)

Used Call Model

• VoS

Formula

• The number of IMS Registration within 1.5s / The number of IMS Registration Success

— Attach Setup Success within 1.5s : Receiving SIP 200 OK for IMS Registration after ‘PDN Connectivity

Request’ (UL EPS SM) within 1.5s

— IMS Registration Success : Receiving SIP 200 OK for IMS Registration after ‘PDN Connectivity Request’

(UL EPS SM)



98/135


M f XCAP


99/135

Message from XCAP

• From ‘PDN connectivity request’ (UL EPS SM) to SIP 200 OK for IMS registration


Complete

Attempts

Anal sis sing Acti One and XCAP



100/135



• Check IMS registration setup time fail events in ActixOne

— ActixOne does not provide the items which can check IMS registration setup time fails yet



Fail Cases


101/135

Fail Cases

• 504 Server Time-out

— UE received 200 OK for IMS registration too late due to server error

※ Another IMS error code could be analyzed with the similar way▶ Check Core network (including IMS Server)


Description

18. VoLTE Setup Time < 1.5s


102/135

Description

• Ratio of VoLTE Setup Success which meets the criteria (1.5s) among the VoLTE Setup Success

Procedure

— VoLTE Setup Procedure : from INVITE to 200 OK for IVITE

Target Value

• >95%

Used Call Model

• VoS

Formula

• The number of VoLTE Setup Success within 1.5s / The number of VoLTE Setup Success

— VoLTE Setup Success within 1.5s : Receiving SIP 200 OK for INVITE after SIP INVITE within 1.5s

— VoLTE Setup Success : Receiving SIP 200 OK for INVITE after SIP INVITE





103/135



18. VoLTE Setup Time < 1.5s Message from XCAP


104/135

Message from XCAP

• From SIP INVITE message to SIP 200 OK message for INVITE


P-RACK

Complete

Attempts

200 OK for INVITE




105/135



• Check VoLTE setup time fail events in ActixOne

— ActixOne does not provide the items which can check VoLTE setup time fails yet





106/135


• Check VoLTE setup time fail events using Macro

— Check the event time for VoLTE setup time fails

: Filter out the cases having more than 1.5s


• Define the issues whether the issue is comes from core network or not



Fail Cases



107/135

Fail Cases

• QCI1 Activation Delay by IMS

— QCI1 Bearer was activated too late



Time: 16:55.25.687

Time: 16:55.31.093

Fail Cases



108/135

Fail Cases

• 183 Session Progress Delay by IMS

— UE received 183 Session Progress message with delay more than 20000ms



Time: 17:25.13.799

Time: 17:25.18.623

19. VoLTE Jitter < 40ms Description


109/135

Description

• Ratio of VoLTE Jitters which meets the criteria (40ms)

— VoLTE Jitters : Statistical variation of the RTP data packet inter-arrival time (RFC 3550)

Target Value

• >95%

Used Call Model

• VoL

Formula

• The number of VoLTE Jitters within 40ms / The number of VoLTE Jitters

— Lost VoLTE RTP Packets will be calculated based on the received RTP Packet Sequence Number



19. VoLTE Jitter < 40ms


110/135



• Check VoLTE Jitter fail events in ActixOne

— ActixOne does not provide the items which can check VoLTE Jitter yet


19. VoLTE Jitter < 40ms Analysis using XCAP


111/135

Analysis using XCAP

• Check Jitter value in XCAP



Description

20. VoLTE RTP Packet Loss Rate


112/135

Description

• Ratio of Lost VoLTE RTP Packet

— Causes for packet loss : network congestion, late arrivals etc.

Target Value

•


113/135

y g


• Check RTP Packet Loss in ActixOne





114/135

y g

• Check RTP Packet Loss events using XCAP

— Check the event time for RTP Packet Loss

‐ Packet count/Packet Loss Count


Fail Cases



115/135


— RTP Packets were lost due to poor RF environments(RSRP/SINR)

▶RF Optimization


Packet Loss Point


Fail Case



116/135

• Handover Fail

— RTP Packets were lost due to HO Fail



H/O Fail

Packet Loss

Description

21. Average BLER DL (Info)


117/135

p

• Average NACK rate in DL for the initial transmission within exclusion zone

Target Value

• -

Used Call Model


Formula

• Average of DL BLER for the binned samples within exclusion zone



21. Average BLER DL (Info)


118/135


• Check BLER DL in XCAP

— Check Graph or Table> Qualcomm LTE/LteAdv> PDSCH BLER Info> Total Info> Total BLER


Description

22. Average BLER UL (Info)


119/135

• Average NACK rate in UL for the initial transmission within exclusion zone

Target Value

• -

Used Call Model


Formula

• Average of UL BLER for the binned samples within exclusion zone



22. Average BLER UL (Info)


120/135


• Check BLER DL in XCAP

— Check Graph or Table> Qualcomm LTE/LteAdv> PDSCH BLER Info> Total Info> Total BLER


22. Average BLER UL (Info) Fail Cases


121/135

• High UL BLER due to Interference

— High UL BLER was detected withtin specific area due to interference

※ UL would be affected more severely than DL due to high sensitivity and antenna gain of eNB

▶ Check IoT level in LSM

▶ Detect Interference source using spectrum analyzer at the field

※ Interference source should be removed before CLOT


High UL BLER area


122/135

Appendix

CLOT Report Download from ActixOne

CLOT Test Scenarios

Issue Code – IMS & EMM

Automation Process

CLOT Report Download from ActixOne Drive log cycle and Cluster name are required to download CLOT report


123/135

Log On to ActixOne(http://49.40.2.69/ActixOne/) with provided login credentials

• ID/PW : samsung1/samsung1


CLOT Report Download from ActixOne Follow the below steps sequentially

http://49.40.2.69/ActixOne/http://49.40.2.69/ActixOne/


124/135

1. Click on Network

optimization module

2. Click on Task

Management tab

3. Click on filter

tab

4. Select & click

create filter




125/135

5. Put the filter

Name you want

6. Keep one row for filter ,

select other row and delete

with clicking “delete row” b

utton




126/135

7. From “Field “ column sel

ect “description” and from

“operator” column select ‘

contains’ and put Cluster n

ame of required report whi

ch you want to download

in “Value” column and the

click on “Save “




127/135

8. Right click on Task

management window

9. click on Reports




128/135

10. Double click on “R

JIL CLOT Acceptance R

eports “ template




129/135

Cluster NameTask ID

11. Fill the Cluster Name

, Task ID detail and click

on “Run”




130/135

12. Save the CLOT report


CLOT Test Scenario Drive Test

f f


131/135


• It will be used 5 handsets to perform five tests using Accuver DM tool installed in mobile

handset

• Each handset will perform one of the test given below

1. FTP DL 1

2. FTP DL 2

3. FTP UL

4. VoLTE Long

5. VoLTE Short

• All test scenarios are already predefined in Accuver DM tool

— Drive test engineer will select different test scenarios per handset

IMS Issue Code for VoLTE

Issue Codes


132/135


NO Codes Description1 403 Forbidden

2 487 Request Terminated

3 500 Internal Server Error

4 501 Not Implemented

5 502 Bad Gateway

6 503 Service Unavailable

7 504 Service Time-out

8 505 SIP Version Not Supported

9 513 Message Too Large

PDN Connectivity Reject

Issue Codes


133/135

• If the PDN CONNECTIVITY REQUEST cannot be accepted by the network, then MME shall

send a PDN CONNECTIVITY REJECT message to the UE.

— This message shall contain an ESM cause value which indicates the reason for rejecting the UE

requested PDN connectivity.

— The ESM cause IE typically indicates one of the following ESM cause values;

‐ Refer to section 6.5.1.4 in the 3GPP TS 24.301 for detailed information


# EMM Cause # EMM Cause

#8 operator determined barring #35: PTI already in use

#26 insufficient resources #38 network failure

#27 missing or unknown APN #50 PDN type IPv4 only allowed

#28 unknown PDN type #51 PDN type IPv6 only allowed

#29 user authentication failed #53: ESM information not received

#30 request rejected by Serving GW or PDN GW #54 PDN connection does not exist

#31 request rejected, unspecified #55 multiple PDN connections for a

given APN not allowed

#32 service option not supported #95 111: protocol errors

#33 requested service option not subscribed #112 APN restriction value incompatible

with active EPS bearer context.

#34 service option temporarily out of order

Automation Process Automation Process Improvement

XCAL M bil → FMS → A ti O


134/135

• XCAL-Mobile → FMS → ActixOne※ XCAL-Mobile : UE Diagnostic Monitor App. in S4mini, FMS : Fleet Management System, ActixOne : Automatic field test log analysis and report

generation tool

XCAL-

MobileFMS

(Storage 1 / Processing 1)

ActixOne(Storage 1 / Processing 28)

Backup

Server

DRX DRM

DRXContent Collect Log file

File Type DRX

Content • Management Call Scenario, IMEI & Log file

File Type • Convert DRX → DRM

Storage • 600GB → Upgrade 6TB (10DEC ~ 15DEC2014)

Processing • Processing 1 = max 2,000 Devices

Save file • The type of Log file is DRX saved 3 month (User option)

Content • Generation SCFT & CLOTreport

• Provide Plot & Map

File Type • DRM

Storage • 7TB → upgraded 60TB(15JAN2015)

Processingsever

• 8 → 28 upgraded• 1 = 16 files processing

Content • Save Log files

File Type • DRX

Storage • 10TB

Save file • The type of Log file is DRX saved(Not delete)

Point Improvement Plan Target Owner

A To add file check and feedback process in FMS when engineer upload log files. End of Feb. Accuver

B-1 To enhance link bandwidth between FMS and ActixOne to raise file transfer speed up. ASAP RJIL

B-2 Need to add status managing system between FMS and ActixOne. TBD Accuver, RJIL

BA



135/135

Copyright and Confidentiality

Documents

CLOT KPI Analysis Guideline(RJIL)_v1.0