SummaryThis spreadsheet presents the results form the BCR3.0 KPI
AT&T subsetThe spreadsheet contains multiple worksheets which
will increase when test cases and results/KPI's are completed and
availableThere are 3 main worksheets:1. Summary - this worksheet
provides an overview of progress2. BCR3.0 KPI - this worksheet
contains detailed development progress and execution results for
KPI's and links to test reports in html and iTest formats3.
KPI-3.0-xyz - these spreadsheets provide additional information for
individual KPI test cases such as test statistics generated in
iTestNot planned test cases are KPI's that have been excluded from
the BCR2.4 KPI subset as these are not required for AT&T
Summary0611018000
BCR 3.0 KPI Test Case Development
BCR3.0 KPI2870000
BCR 3.0 KPI Execution Results
DevelopmentRun 1Run 2Run 3Run 4Run 5Marked
OveralDevelopmentExecutionAT&TFeature NumberTest
NameRequirement 118617 NumberTest DetailsTest AreaTest case
Developed in iTestValidation
detailsPass/FailResultPass/FailDateResults NotesDetailed
ResultsTest report location (HTML)Test report location (iTest)Femto
DetailsCluster DetailsPass/FailResultDateResults NotesDetailed
ResultsTest report location (HTML)Test report location (iTest)Femto
DetailsCluster DetailsPass/FailResultDateResults NotesDetailed
ResultsTest report location (HTML)Test report location (iTest)Femto
DetailsCluster DetailsPass/FailResultDateResults NotesDetailed
ResultsTest report location (HTML)Test report location (iTest)Femto
DetailsCluster DetailsPass/FailResultDateResults NotesDetailed
ResultsTest report location (HTML)Test report location (iTest)Femto
DetailsCluster DetailsResultDeveloped UntestedDeveloped 10
iteration testedDeveloped Partially TestedIn progressBlockedNot
PossibleNot PlannedPass/Fail List OptionsFailPassNot
PlannedRemaininginProgessNotPossibleyFTS-118617From BSR Femto to 3G
Macro Network Intra Frequency in handover region for PS and CS+PS,
with MAHOR2Test Ref: KPI-3.0-001Requirement: FTS-77715, FTS-83234,
FTS-106897, FTS-120107, FTS-118617 R2Femto to 3G Macro Network
Intra Frequency in handover region for CS, with MAHO
Handoverfailure rate.This KPI provides the percentage of failed
Intra Frequency Mobile Assisted CS Handoversbetween Femto and 3G
network..The percentage of failed handovers should be <
1%.DefinitionFemto to 3G Macro Network Intra Frequency Mobile
Assisted CS Handover Failure Rate =
100*(VS.AttIntraFreqMAHOFemto3GMacroCS
-VS.SuccIntraFreqMAHOFemto3GMacroCS)
/VS.AttIntraFreqMAHOFemto3GMacroCS Femto to 3G Macro Network Intra
Frequency in handover region for CS+PS, with MAHOHandover failure
rate.This KPI provides the percentage of failed Intra Frequency
Mobile Assisted CS+PS handoversbetween Femto and 3G network.The
percentage of failed handovers should be < 1%.Definition Femto
to 3G Macro Network Intra Frequency Mobile Assisted CSPS Handover
Failure Rate = 100* (VS.AttIntraFreqMAHOFemto3GMacroCSPS
-VS.SuccIntraFreqMAHOFemto3GMacroCSPS)/
VS.AttIntraFreqMAHOFemto3GMacroCSPSMobility - Cell
ReselectionDeveloped 10 iteration testedTest case has been run
successfully with 10 iterationsPass0% 98%.DefinitionMacro 3G to
Femto CS+PS Handover Success Rate =100 *
(VS.HHO.Succ3GMacroToBsrCSPSHO /
VS.HHO.Att3GMacroToBSRCSPSHO).Mobility - Femto To 3G -
InterDeveloped Partially TestedU-Dragon
issuePass100%>98%11/29/11Serial:Software version:
BSR-03.00.27_pHardware version: v2Power: 250mW Femto
709Pass0010000Not planned010000yFTS-118617From BSR Femto to 2G
Macro NetworkInterRAT (Cell Change Order) in handoverregionR8Test
Ref: KPI-3.0-004Requirement: FTS-77715, FTS-83234, FTS-106897,
FTS-120107, FTS-118617 R8Femto to 2G Macro PS Blind Handover
Success Rate within handover regionThis KPI defines the percentage
of successful Femto to 2G Macro Blind PS Handover
operationsperformed (these are triggered by the Inter-RAT Cell
Change Order).The percentage of successful handovers of this type
should be > 98%.DefinitionFemto to 2G Macro PS Blind Handover
Success Rate
=100*(VS.SuccInterFreqBlindFemto2GMacroHOPS/VS.AttInterFreqBlindFemto2GMacroHOPS)
UE Assisted Femto to 2G Macro PS Handover Success Rate within
handover regionThis KPI defines the percentage of successful UE
assisted Femto to 2G Macro PS Handoveroperations performed (these
are triggered by the Inter-RAT Cell Change Order).The percentage of
successful handovers of this type should be >
98%.DefinitionFemto to 2G Macro PS Blind Handover Success Rate
=100*(VS.SuccMAHOFemto2GMacroPS/VS.AttMAHOFemto2GMacroPS)Mobility -
Inter RATDeveloped Partially TestedU-Dragon
issuePass100%>98%12/2/11Serial:Software version:
BSR-03.00.29_pHardware version: v2Power: 250mW Femto
709Pass0010000NotPossible010000yFTS-118617Macro to Femto mean
interruption time duringthe handover phase for PS and CS+PSR9Test
Ref: KPI-3.0-005Requirement: FTS-77715, FTS-83234, FTS-106897,
FTS-120107, FTS-118617 R9Macro to Femto Mean Interruption time
during the handover phase for PSThis KPI measures the typical
interruption time during PS handover from Macro network toFemto.The
mean interruption time should be < 200 msDefinitionThis KPI is
not measured via counters, since the measurement is actually at the
UE. The handovercommences when the UE receives an RRC RADIO BEARER
RECONFIGURATION message.The PS data handover interruption period
will start when the UE stops receiving RLC PDUs andwill finish at
the resumption of the PDUs. Macro to Femto Mean Interruption time
during the handover phase for CS+PSThis KPI measures the typical
interruption time during CS+PS handover from Macro network
toFemto.The mean interruption time should be < 200
msDefinitionThis KPI is not measured via counters, since the
measurement is actually at the UE. The handovercommences when the
UE receives an RRC RADIO BEARER RECONFIGURATION message.The CS+PS
data handover interruption period will start when the UE stops
receiving RLC PDUsand will finish at the resumption of the
PDUs.Mobility - Femto To 3G - IntraDeveloped Partially TestedResult
to be checked manuallyPass114ms 120ms3.212/7/11Serial:Software
version: BSR-03.00.29Hardware version: v2Power: 250mW Femto
709Pass0100000010000yFTS-118618CS Data Call Set up Time (UMTS
signaling only)
Test Requirement:FTS-77715 (BCR 2.2) R12FTS-106897 (BCR 2.4)
R46R15Test Ref:KPI-3.0-009Requirement: FTS-77715 R12, FTS-106897
R15Test Description: R15 CS Data Call Set up Time (UMTS signaling
only)The call setup delay is defined as the time required to
establish a CS data call from a UMTSmobile to another UE in the
same UMTS network. The delay assumes both mobiles areattached to
the same MSC.Starting event: Originating UE sends CCCH:RACH:RRC
Connection Request on firstPreambleFinishing event: Originating UE
receives DCCH:DCH:Direct Transfer Alerting responseThe total UE to
UE call setup delay shall be less than 6s (mean) and 10 s(95%ile
)Notes: This requirement supercedes FTS-77715-581 (related to R2.2
R11).Call-PDeveloped 10 iteration testedTest case has been run
successfully with 10
iterationsPass5.3878s6..0s9/1/11Serial:Software version:
BSR-03.00.8_pHardware version: v1.2Power: 20mW Femto 306IOT
Cluster2 (SGSN 3, MSC3)Release 3.0Pass0100000010000yFTS-118619The
mean interruption time during PS or CS+PS handover phaseR16Test
Ref: KPI-3.0-010Requirement: FTS-83234 R3, FTS-106897 R16Test
Description: R16 Femto to Femto mean interruption time during PS or
CS+PS handover phaseThe mean interruption time during PS or CS+PS
Femto to Femto handover is < 200ms (downlink)and < 300 ms
(uplink) measured on the voice and packet user planes.The handover
can be identified as having commenced when the UE receives
anrrc_radioBearerReconfiguration message with a new_U_RNTI.The PS
data handover interruption period will start when the UE stops
receiving RLC PDUs andwill finish at the resumption of the
PDUs.NoteQUALCOMM UE LOG_CODE 0x4134 (WCDMA RLC DL AM PDU) provides
information onreceived number of PDUs .This requirement supercedes
FTS-106897-1575 (related to R2.4 R4)..Mobility - Femto To
FemtoDeveloped Partially TestedResult to be checked manuallyPass173
DL 190 UL55kbps9/2/11Serial:Software version: BSR-03.00.8_pHardware
version: v2Power: 250mW Femto 709Pass0100000010000FTS-118623Two
Simultaneously HSPDA User ThroughputR20Test Ref:
KPI-3.0-014Requirement: FTS-77715 R5, FTS-106897 R20Test
Description: R20 Two Simultaneously HSPDA User ThroughputAverage
user downlink application layer throughput for 2 simultaneous HSPDA
users shall belarger than 2 Mbps for each user. FTP downloading
file size shall be 50 Mbytes.Note: The following conditions apply
to this KPICQI>16 for Category 8 UE with 10 codes.CQI>19 for
Category 6 UE with 5 codes.Reference [VF_GRS] CAP_KPI_003.PS Data
HSDPADeveloped 10 iteration testedTest case has been run
successfully with 10 iterationsPassUE1: 3.2 Mbps UE2: 3.74 Mbps QC
chipset U900 and QC 6695>2Mbps9/13/11Serial:Software version:
BSR-03.00.13_pHardware version: v2Power: 250mW Femto
709Pass0100000010000FTS-118624Four Simultaneous Voice callsR21Test
Ref: KPI-3.0-015Requirement: FTS-77715 R5, FTS-106897 R21Test
Description: R21 Four Simultaneous Voice callsThroughput indicated
by FTS-77715-523 shall be met with 4 simultaneous voice calls.Voice
quality (MOS) indicated by FTS-77715-582 shall be met with 4
simultaneous voice calls.Voice QualityDeveloped 10 iteration
testedTest case has been run successfully with 10
iterationsPass3.9990>3.2 MTM >3.5 MTC >3.5
MOC9/26/11Serial:Software version: BSR-03.00.16_pHardware version:
v2Power: 250mW Femto 709Pass0100000010000BCR2.5yFTS-120107SSCP
setup success rate (without support of new PM countersR5Test Ref:
KPI-3.0-016Requirement: FTS-77715, FTS-83234, FTS-106897,
FTS-120107 R5 Test Description: Incoming SCCP Setup Success Rate
This KPI provides the percentage of Iu connections which were
successfully set up at the BSG(where Iu connections are terminated)
in response to requests coming from the SGSN. The KPIdoes not
distinguish between Iu-IP or Iu-ATM connections.This KPI measures
the success of setting up Iu connections within the Femto
Gateway.This success rate should exceed 99%.DefinitionIncoming SCCP
Setup Success Rate = 100 * VS.SuccIncomingSCCPReq /
VS.AttIncomingSCCPReq Outgoing SCCP Setup Success RateThis KPI
provides the percentage of Iu Connections which were successfully
set up within theSGSN (and IPC if present) in response to requests
originating from a BSG. The KPI does notdistinguish between Iu-IP
or Iu-ATM connections. This KPI measures the success of setting up
Iu connections associated with a particular BSGwithin the SGSN.This
success rate should exceed 99%.DefinitionOutgoing SCCP Setup
Success Rate = 100 * VS.SuccOutgoingSCCPReq / VS.AttOutgoingSCCPReq
Overall SCCP Setup Success RateThis KPI provides the percentage of
Iu Connections which were successfully set up through aparticular
BSG. This KPI does not distinguish between Iu-IP or Iu-ATM
connections.This KPI measures the overall success rate of Iu
connections associated with a particular BSGirrespective of whether
the connections originate at the BSG or the SGSN.This success rate
should exceed 99%.DefinitionOverall SCCP Setup Success Rate = 100 *
(VS.SuccOutgoingSCCPReq + VS.SuccIncomingSCCPReq) /
(VS.AttOutgoingSCCPReq +
VS.AttIncomingSCCPReq)AvailabilityDeveloped Partially TestedResult
to be checked manuallyPass100%>99%10/5/11Serial:Software
version: BSR-03.00_17pHardware version: v1.2Power: 20mW Femto
306Pass0001000010000yFTS-120107HSUPA RB setup success rateR6Test
Ref: KPI-3.0-017Requirement: FTS-77715, FTS-83234, FTS-106897,
FTS-120107 R6HSUPA RB setup success rateThis KPI provides the
percentage of RAB attempts mapped on E-DCH tranport channel in
ULwhich succeed based on RAB assignment procedure. RABs mapped on
E-DCH transport channelare considered if either initially attempted
on E_DCH or configured as E-DCH during the RABassignment procedure.
Note: Normal call release during the procedure will impact the
success ratio.This success rate should exceed
99%.AvailabilityDeveloped 10 iteration testedTest case has been run
successfully with 10 iterationsPass100%>99%9/4/11Serial:Software
version: BSR-03.00.8_pHardware version: v2Power: 250mW Femto
709Pass0001000010000yFTS-120107Macro 2G to Femto network handover
success rate for CSR7Test Ref: KPI-3.0-018Requirement: FTS-77715,
FTS-83234, FTS-106897, FTS-120107 R7Macro 2G to Femto network
handover success rate for CSThis KPI provides to percentage of
successful handovers from 2G Macro to Femto.This success rate
should exceed 99%.Mobility - Inter RATDeveloped Partially
TestedU-Dragon
issuePass100%>99%10/2/11Pass0001000010000yFTS-120107RRC
redirection to macro success rateR8Test Ref:
KPI-3.0-019Requirement: FTS-77715, FTS-83234, FTS-106897,
FTS-120107 R8RRC Redirection to macro success rateThis KPI provides
the percentage of successful RRC redirection attempts from a Femto
BSR to themacro network.This success rate should exceed
99%.Definition:RRC Redirection to Macro Success Ratio = 100 *
((VS.SuccActiveCallReDirectUMTS +VS.NumSuccActCallBRM +
VS.NumSuccCallRedirectBRM +VS.SuccActiveCallRedirect2G) /
(VS.NumAttCallRedirectBRM +VS.NumAttActCallBRM +
VS.AttActiveCallReDirectUMTS +VS.AttActiveCallReDirect2G))Mobility
- Cell ReselectionDeveloped 10 iteration testedTest case has been
run successfully with 10
iterationsPass100%>99%9/28/11Serial:Software version:
BSR-03.00.15_pHardware version: v2Power: 250mW Femto
708Pass0100000010000yFTS-120107Service based handover to macro
success rateR9Test Ref: KPI-3.0-020Requirement: FTS-77715,
FTS-83234, FTS-106897, FTS-120107 R9Service based handover to macro
success rateThis KPI provides the percentage of successful service
based handover attempts from Femto BSRto Macro network.This success
rate should exceed 99%.Definition:Service Based Handover to Macro
Success Ratio = 100 *((VS.HHO.SuccCSServiceBased2GHO +
.HHO.SuccCSServiceBased3GHO) /(VS.HHO.AttCSServiceBased2GHO +
.HHO.AttCSServiceBased3GHO))Mobility - Femto To 3G - InterDeveloped
10 iteration testedTest case has been run successfully with 10
iterationsPass100%>99%9/29/11Serial:Software version:
BSR-03.00.17_pHardware version: v2Power: 250mW Femto
7097/23/10Current results are MTM 100 iterations
onlyhttp://svn.malab.uk.alcatel-lucent.com/svn/repos/FEMTO-Automation/reports/nlt/KPI/saved_test_reports/KPI-2.4-005_2010-07-23_17-56-48.htmlSerial:2150254412Software
version: BSR-02.04.40.28Hardware version: v1.2Power: 20mWIOT
Cluster2 (SGSN 3, MSC3)Release 2.37/30/10Current results are MTM 13
iterations
onlyhttp://svn.malab.uk.alcatel-lucent.com/svn/repos/FEMTO-Automation/reports/nlt/KPI/saved_test_reports/KPI-2.4-005_2010-07-30_16-08-38.htmlSerial:2150254412Software
version: BSR-02.04.40.34Hardware version: v1.2Power: 20mWIOT
Cluster1 (SGSN 3, MSC3)Release 2.4Pass0100000010000yFTS-120107Radio
Bearer Reconfiguration Success RateR10Test Ref:
KPI-3.0-021Requirement: FTS-77715, FTS-83234, FTS-106897,
FTS-120107 R10Radio Bearer Reconfiguration Success RateThis KPI
provides the percentage of successful performed RB reconfigurations
from UE point-ofviewfrom the attempts.The number of successful
Radio Bearer reconfigurations is defined as the difference between
thenumber of attempted RB reconfigurations and the number of failed
RB reconfigurations where afailure is defined as any
reconfiguration operation which does not result in receit of an
RBReconfiguration Complete message.This success rate should exceed
99%.Mobility - Cell ReselectionDeveloped 10 iteration testedTest
case has been run successfully with 10
iterationsPass100%>99%9/27/11Serial:Software version:
BSR-03.00.16_pHardware version: v2Power: 250mW Femto
709Pass0100000010000yFTS-120107Call Release Time- Mobile to
PSTNR13Test Ref: KPI-3.0-022Requirement: FTS-77715, FTS-83234,
FTS-106897, FTS-120107 R13Call Release time - mobile to PSTNThis
KPI represents the time between the NAS request to release a call
and the acknowledgementthat the radio resources have been released
for that call for a call between a UE connected to aFemto BSR and a
UE connected to the PSTN.The call release time (mobile to PSTN)
should be less than 6 seconds.The start time for the KPI is defined
as the point where the NAS: Disconnect message is receivedby the
small cell.The end time for the KPI is defined as the point where
the Iu Release Complete Message is sentfrom the small cell to the
gateway.CS VoiceDeveloped 10 iteration testedTest case has been run
successfully with 10 iterationsPass1.2059 sec6.0
seconds8/22/11Serial:Software version: BSR-03.00.8_pHardware
version: v2Power: 250mW Femto 709Pass0100000010000yFTS-120107Call
Release Time- Mobile to MobileR14Test Ref: KPI-3.0-023Requirement:
FTS-77715, FTS-83234, FTS-106897, FTS-120107 R14Call Release time -
mobile to mobileThis KPI represents the time between the NAS
request to release a call and the acknowledgementthat the radio
resources have been released for that call for a call between a UE
connected to aFemto BSR and a UE connected to the macro network.The
call release time (mobile to mobile) should be less than 8
seconds.The start time for the KPI is defined as the point where
the NAS: Disconnect message is receivedby the small cell.The end
time for the KPI is defined as the point where the Iu Release
Complete Message is sentfrom the small cell to the gateway.CS
VoiceDeveloped 10 iteration testedTest case has been run
successfully with 10 iterationsPass0.3534s8.0
seconds8/31/11Serial:Software version: BSR-03.00.8_pHardware
version: v1.2Power: 20mW Femto
306Pass0100000010000yFTS-120107Recovery Time - Link LossR15Test
Ref: KPI-3.0-024Requirement: FTS-77715, FTS-83234, FTS-106897,
FTS-120107 R15Recovery Time - Link LossThis KPI measures the time
that the Femto BSR takes to reestablish communication with the
Femto Gateway in the case of a link loss.The link loss recovery
time for a Femto BSR should be no more than 3 minutes.The time
should be measured from the point where a link loss has been
detected, and the link isreconnected to the point where the Femto
is fully functioning again.AvailabilityIn progressManual
TestPass2.10s3.0 mins9/1/11Serial:Software version:
BSR-03.00.8_pHardware version: v1.2Power: 20mW Femto
212Pass0001000010000yFTS-120107Recovery Time - Power LossR16Test
Ref: KPI-3.0-025Requirement: FTS-77715, FTS-83234, FTS-106897,
FTS-120107 R16Recovery Time - Power LossThis KPI measures the time
that the Femto BSR takes to reestablish communication with theFemto
Gateway following a power loss.The power loss recovery time for a
Femto BSR should be no more than 12 minutes.The time should be
measured from the point where a link loss has been detected by the
gateway,and power is re-provided to the Femto to the point where
the Femto is fully functioning again.Note: This KPI is only
applicable if the Femto has previously been successfully in service
(i.e. theoscillator has previously been fully disciplined) and the
Femto's NTP parameters are recent. ThisKPI does not apply if the
Femto has been switched off for an extended period of time and has
toperform a cold start to re-discipline the oscillator
behaviour.AvailabilityIn progressManual TestPass4.31min12.0
mins9/1/11Serial:Software version: BSR-03.00.8_pHardware version:
v1.2Power: 20mW Femto 212Pass0001000010000yFTS-120107Provisioning
Time: Boot to Assign CGI (may delete, Dave to check with
Akhilesh)R19Test Ref: KPI-3.0-026Requirement: FTS-77715, FTS-83234,
FTS-106897, FTS-120107 R19Provisioning Time: Boot to Assign CGIThis
KPI measures the time that a Femto network takes to assign a CGI or
unique identifier to aFemto BSR that has not been previously
connected to the Femto network, or a Femto BSR that hasbeen
previously connected to the network but for some reason needs a new
CGI assigned.The duration of this KPI should not exceed 32
minutes.Open Issue: This KPI is still under debate with AT&T.
Since CGI cannot be assigned until theoscillator has been
disciplined, it is not clear that this KPI can be met.The time
should be measured from the point where a Femto BSR is booted for
the first time to thepoint where the gateway completes the assign
CGI interaction.AvailabilityIn progressManual TestPass12minDL
audio. PESQ is an algorithmic non subjective (objective) audio
quality score from which audio scoring tools derive their unique
Estimated Mean Opinion Score (EMOS). It is preferable that PESQ
scores are used for KPI as these are comparable with other audio
application PESQ scores where as EMOS/MOS scores cannot be
compared. Next all calls are terminated and the test is repeated.
The pass criteria is that the MOS should be larger than 3.5
(mean).
Note: MTM voice quality target in all other test cases and KPI
is 3.2. However this test case requires 3.5 target which is more
challenging combined with the configuration of 16 active UE calls
which is also more challenging for the Femto. This has been
highlighted to the KPI document owners.
"The target is intended to be appropriate for the enterprise
market where we expect voice quality on a fully loaded system to be
more of a sensitive issue for operators than a fully loaded
residential. It is a deliberate target that we must try hard to
achieve." Kevin Pendleton, Bill Dickson 28/01/2010Voice
QualityDeveloped 10 iteration testedTest case has been run
successfully with 10
iterationsPass3.6670>3..29/6/11Serial:Software version:
BSR-03.00.8_pHardware version: v2Power: 250mW Femto 709IOT Cluster3
(SGSN 4, MSC3)Release 2.0Pass0100000010000nFTS-106897Delay between
first power-on with factory default settings and femtocell service
availability (including NTP synch)R17aTest Ref: KPI-3.0-043 old
KPI-2.4-046Requirement: FTS-106897 R17aTest Description: To
automate the test it must be possible to reset the Femto to factory
default settings via a CLI. Next the power is controlled via e.g. a
Blackbox power control system and the Femto is switched off and
then back to create first power on cycle. A mechanism has to be
established to measure the time from power on until the Femto is in
service and transmitting including NTP synch. The pass criteria for
this test is that the total time from power on until providing
service should be less than 90 minutes.AvailabilityIn
progressManual TestPass20 mins3..58/14/11Serial:Software version:
BSR-03.00.4_pHardware version: v2Power: 250mW Femto 709IOT Cluster3
(SGSN 3, MSC3)Release 2.2Pass0100000010000nFTS-83234Voice Speech
Quality AMR 12.2 kbps Femto Mobile to Femto MobileFour Simultaneous
Voice callsR8Test Ref: KPI-3.0-048 old KPI-2.4-048Requirement:
FTS-83234 R8, FTS-106897 R22Test Description: This test case is
based on measuring the DL quality of a MTM call. Whilst there are
another 2 UE's are in call. There will therefore be in total 4
active calls. Firstly a MTM UE to UE call is setup then a second
automated MTM call is established. There are now 4 active CS voice
calls in progress. On one pair of MTM a pre-recorded audio waveform
is played backed and received and recorded at the UE. An audio
analysis and scoring tool (TBD) is then used to compare the source
and received signals to establish a Perceptual Evaluation of Speech
Quality (PESQ) score of the UL->DL audio. PESQ is an algorithmic
non subjective (objective) audio quality score from which audio
scoring tools derive their unique Estimated Mean Opinion Score
(EMOS). It is preferable that PESQ scores are used for KPI as these
are comparable with other audio application PESQ scores where as
EMOS/MOS scores cannot be compared. Next all calls are terminated
and the test is repeated. The pass criteria is that the MOS should
be larger than 3.2 (mean).Voice QualityDeveloped 10 iteration
testedTest case has been run successfully with 10
iterationsPass4.0300>3..29/8/11Serial:Software version:
BSR-03.00.8_pHardware version: v2Power: 250mW Femto 709IOT Cluster3
(SGSN 3, MSC3)Release 2.0Pass0100000010000nFTS-83234Voice Speech
Quality on Sample Basis when One Simultaneous R99 PS User with 384
kbps Bearer In Very Close Proximity To The femto CPE For Three
Simultaneous Voice CallsDL Voice Speech Quality AMR 12.2 kbps
PSTN/ISDN to Femto MobileR9Test Ref: KPI-3.0-049 old
KPI-2.4-058Requirement: FTS-83234 R9Test Description: his test case
is based on measuring the DL quality of a MOC to PSTN/ISDN call.
Whilst there are another 4 UE's are in call one of which is a
multi-RAB/simbearer R99 call. There will therefore be in total 4
active calls. Firstly a MTM UE1 to UE2 call is setup then a second
automated pair of automated calls are setup as MOC to PSTN/ISDN
from UE3 and UE4. There are now 4 active CS voice calls in progress
2 are MTM and 2 are MOC to PSTN/ISDN. Next a R99 PS RAB is
established on UE1 and a data transfer started. UE1 should be in
close proximity to the Femto. A pre-recorded audio waveform is
played backed from one of the PSTN/ISDN handsets and received and
recorded at UE3 or UE4. An audio analysis and scoring tool (TBD) is
then used to compare the source and received signals to establish a
Perceptual Evaluation of Speech Quality (PESQ) score of the
UL->DL audio. PESQ is an algorithmic non subjective (objective)
audio quality score from which audio scoring tools derive their
unique Estimated Mean Opinion Score (EMOS). It is preferable that
PESQ scores are used for KPI as these are comparable with other
audio application PESQ scores where as EMOS/MOS scores cannot be
compared. Next all calls are terminated and the test is repeated.
The pass criteria is that the MOS should be larger than 3.5
(mean).Voice QualityDeveloped 10 iteration testedTest case has been
run successfully with 10
iterationsPass3.5420>3..58/14/11Serial:Software version:
BSR-03.00.4_pHardware version: v2Power: 250mW Femto 709IOT Cluster3
(SGSN 4, MSC3)Release 2.1Pass0100000010000nFTS-83234Voice Speech
Quality on Sample Basis when One Simultaneous R99 PS User with 384
kbps Bearer In Very Close Proximity To The Femto CPE For Three
Simultaneous Voice CallsDL Voice Speech Quality AMR 12.2 kbps Femto
Mobile to Femto MobileR10Test Ref: KPI-3.0-050 old
KPI-2.4-057Requirement: FTS-83234 R10Test Description: This test
case is based on measuring the DL quality of a MTM call. Whilst
there are another 4 UE's are in call one of which is a
multi-RAB/simbearer R99 call. There will therefore be in total 4
active calls. Firstly a MTM UE1 to UE2 call is setup then a second
automated MTM call is established between UE3 and UE4. There are
now 4 active CS voice calls in progress. Next a R99 PS RAB is
established on UE1 and a data transfer started. UE1 should be in
close proximity to the Femto. On one pair of MTM calls a
pre-recorded audio waveform is played backed and received and
recorded at the end UE. An audio analysis and scoring tool (TBD) is
then used to compare the source and received signals to establish a
Perceptual Evaluation of Speech Quality (PESQ) score of the
UL->DL audio. PESQ is an algorithmic non subjective (objective)
audio quality score from which audio scoring tools derive their
unique Estimated Mean Opinion Score (EMOS). It is preferable that
PESQ scores are used for KPI as these are comparable with other
audio application PESQ scores where as EMOS/MOS scores cannot be
compared. Next all calls are terminated and the test is repeated.
The pass criteria is that the MOS should be larger than 3.2
(mean).Voice QualityDeveloped 10 iteration testedTest case has been
run successfully with 10
iterationsPass4.4800>3..28/8/11Serial:Software version:
BSR-03.00.4_pHardware version: v2Power: 250mW Femto 709IOT Cluster3
(SGSN 4, MSC3)Release 2.0Pass0100000010000nFTS-83234UL Voice Speech
Quality AMR 12.2 kbps Femto Mobile to PSTN/ISDNEight Simultaneous
Voice callsR11Test Ref: KPI-3.0-051 old KPI-2.4-052Requirement:
FTS-83234 R11, FTS-106897 R25Test Description: This test case is
based on measuring the UL quality of a UE to PSTN/ISDN call. Whilst
there are another 7 UE's are in call. In order to make 7 UE calls,
6 (3 pairs) can be setup as MTM with the 7th either as loopback
call or call to another Femto UE or 2nd PSTN/ISDN end point.
Automated MTM calls are established next either a MOC to PSTN/ISDN,
MOC to UE on another Femto, or MOC loopback is established. There
are now 7 active CS voice calls in progress. Next an 8th automated
call, MOC AMR 12.2 kbps CS voice call to PSTN/ISDN is established.
A pre-recorded audio waveform is played backed from the UE and
received and recorded at PSTN/ISDN side. An audio analysis and
scoring tool (TBD) is then used to compare the source and received
signals to establish a Perceptual Evaluation of Speech Quality
(PESQ) score of the UL audio. PESQ is an algorithmic non subjective
(objective) audio quality score from which audio scoring tools
derive their unique Estimated Mean Opinion Score (EMOS). It is
preferable that PESQ scores are used for KPI as these are
comparable with other audio application PESQ scores where as
EMOS/MOS scores cannot be compared. Next all calls are terminated
and the test is repeated. The pass criteria is that the MOS should
be larger than 3.5 (mean).Voice QualityDeveloped 10 iteration
testedTest case has been run successfully with 10
iterationsPass4.0400>3..59/9/11Serial:Software version:
BSR-03.00.8_pHardware version: v2Power: 250mW Femto 709IOT Cluster3
(SGSN 3, MSC3)Release 2.1Pass0100000010000nFTS-83234DL Voice Speech
Quality AMR 12.2 kbps PSTN/ISDN to Femto MobileEight Simultaneous
Voice callsR12Test Ref: KPI-3.0-052 old KPI-2.4-053Requirement:
FTS-83234 R12, FTS-106897 R26Test Description: This test case is
based on measuring the DL quality of a UE to PSTN/ISDN call. Whilst
there are another 7 UE's are in call. In order to make 7 UE calls,
6 (3 pairs) can be setup as MTM with the 7th either as loopback
call or call to another Femto UE or 2nd PSTN/ISDN end point.
Automated MTM calls are established next either a MOC to PSTN/ISDN,
MOC to UE on another Femto, or MOC loopback is established. There
are now 7 active CS voice calls in progress. Next an 8th automated
call, MOC AMR 12.2 kbps CS voice call to PSTN/ISDN is established.
A pre-recorded audio waveform is played backed from the PSTN/ISDN
side and received and recorded at the UE side. An audio analysis
and scoring tool (TBD) is then used to compare the source and
received signals to establish a Perceptual Evaluation of Speech
Quality (PESQ) score of the DL audio. PESQ is an algorithmic non
subjective (objective) audio quality score from which audio scoring
tools derive their unique Estimated Mean Opinion Score (EMOS). It
is preferable that PESQ scores are used for KPI as these are
comparable with other audio application PESQ scores where as
EMOS/MOS scores cannot be compared. Next all calls are terminated
and the test is repeated. The pass criteria is that the MOS should
be larger than 3.5 (mean).Voice QualityDeveloped 10 iteration
testedTest case has been run successfully with 10
iterationsPass4.0190>3..59/11/11Serial:Software version:
BSR-03.00.13_pHardware version: v2Power: 250mW Femto 709IOT
Cluster3 (SGSN 3, MSC3)Release 2.2Pass0100000010000nFTS-83234Voice
Speech Quality AMR 12.2 kbps Femto Mobile to Femto MobileEight
Simultaneous Voice callsR13Test Ref: KPI-3.0-053 old
KPI-2.4-051Requirement: FTS-83234 R13, FTS-106897 R27Test
Description: This test case is based on measuring the DL quality of
a MTM call. Whilst there are another 7 UE's are in call. There will
therefore be in total 8 active calls. First 6 UE's (3 pairs) as
automated to perform MTM calls to one another. Next the 4th call
setup is again between a 4th pair UE's 7 and 8. UE's 7 and 8 are
the UE's that voice quality will be perofmed on. There are now 8
active CS voice calls in progress. A pre-recorded audio waveform is
played backed on UE7 and received and recorded at UE8. An audio
analysis and scoring tool (TBD) is then used to compare the source
and received signals to establish a Perceptual Evaluation of Speech
Quality (PESQ) score of the UL->DL audio. PESQ is an algorithmic
non subjective (objective) audio quality score from which audio
scoring tools derive their unique Estimated Mean Opinion Score
(EMOS). It is preferable that PESQ scores are used for KPI as these
are comparable with other audio application PESQ scores where as
EMOS/MOS scores cannot be compared. Next all calls are terminated
and the test is repeated. The pass criteria is that the MOS should
be larger than 3.2 (mean).Voice QualityDeveloped 10 iteration
testedTest case has been run successfully with 10
iterationsPass4.4500>3..28/11/11Serial:Software version:
BSR-03.00.4_pHardware version: v2Power: 250mW Femto 709IOT Cluster3
(SGSN 3, MSC3)Release 2.1Pass0100000010000nFTS-83234Voice Speech
Quality on Sample Basis when One Simultaneous R99 PS User with 384
kbps Bearer In Very Close Proximity To The femto CPE For Seven
Simultaneous Voice CallsDL Voice Speech Quality AMR 12.2 kbps
PSTN/ISDN to Femto MobileR14Test Ref: KPI-3.0-054 old
KPI-2.4-060Requirement: FTS-83234 R14, FTS-106897 R28Test
Description: This test case is based on measuring the DL quality of
a of a MOC to PSTN/ISDN call. Whilst there are another 8 UE's are
in call one of which is a multi-RAB/simbearer R99 call. There will
therefore be in total 8 active calls. First 3 pairs of MTM calls
are established e.g. UE1 to UE2, UE3 to UE4 and UE5 to UE6. There
are now 6 active MTM CS voice calls in progress. Then two further
calls are established as UE to PSTN/ISDN on UE7 and UE8. Next a R99
PS RAB is established on UE1 and a data transfer started. UE1
should be in close proximity to the Femto. On one pair of MTM calls
e.g. UE 7 to UE8, a pre-recorded audio waveform is played backed
and received and recorded at the end UE. An audio analysis and
scoring tool (TBD) is then used to compare the source and received
signals to establish a Perceptual Evaluation of Speech Quality
(PESQ) score of the UL->DL audio. PESQ is an algorithmic non
subjective (objective) audio quality score from which audio scoring
tools derive their unique Estimated Mean Opinion Score (EMOS). It
is preferable that PESQ scores are used for KPI as these are
comparable with other audio application PESQ scores where as
EMOS/MOS scores cannot be compared. Next all calls are terminated
and the test is repeated. The pass criteria is that the MOS should
be larger than 3.5 (mean).Voice QualityDeveloped 10 iteration
testedTest case has been run successfully with 10
iterationsPass4.0170>3..59/11/11Serial:Software version:
BSR-03.00.13_pHardware version: v2Power: 250mW Femto 709IOT
Cluster3 (SGSN 4, MSC3)Release 2.3Pass0100000010000nFTS-83234Voice
Speech Quality on Sample Basis when One Simultaneous R99 PS User
with 384 kbps Bearer In Very Close Proximity To The femto CPE For
Seven Simultaneous Voice CallsDL Voice Speech Quality AMR 12.2 kbps
Femto Mobile to Femto MobileR15Test Ref: KPI-3.0-055 old
KPI-2.4-059Requirement: FTS-83234 R15, FTS-106897 R29Test
Description: This test case is based on measuring the DL quality of
a MTM call. Whilst there are another 8 UE's are in call one of
which is a multi-RAB/simbearer R99 call. There will therefore be in
total 8 active calls. First 4 pairs of MTM calls are established
e.g. UE1 to UE2, UE3 to UE4 ...UE7 to UE8. There are now 8 active
CS voice calls in progress. Next a R99 PS RAB is established on UE1
and a data transfer started. UE1 should be in close proximity to
the Femto. On one pair of MTM calls e.g. UE 7 to UE8, a
pre-recorded audio waveform is played backed and received and
recorded at the end UE. An audio analysis and scoring tool (TBD) is
then used to compare the source and received signals to establish a
Perceptual Evaluation of Speech Quality (PESQ) score of the
UL->DL audio. PESQ is an algorithmic non subjective (objective)
audio quality score from which audio scoring tools derive their
unique Estimated Mean Opinion Score (EMOS). It is preferable that
PESQ scores are used for KPI as these are comparable with other
audio application PESQ scores where as EMOS/MOS scores cannot be
compared. Next all calls are terminated and the test is repeated.
The pass criteria is that the MOS should be larger than 3.2
(mean).Voice QualityDeveloped 10 iteration testedTest case has been
run successfully with 10
iterationsPass4.0200>3..29/8/11Serial:Software version:
BSR-03.00.8_pHardware version: v2Power: 250mW Femto 709IOT Cluster3
(SGSN 4, MSC3)Release 2.2Pass0100000010000yFTS-83234FTP R99
Throughput. % of max bit rate. (measured at FTP application)Three
Simultaneous Voice Calls Are Already Established
Test Requirement:FTS-83234 (BCR 2.3) R16FTS-106897 (BCR 2.4)
R30R16Test Ref: KPI-3.0-056 old KPI-2.4-020Requirement: FTS-83234
R16, FTS-106897 R30Test Description: 3 automated CS voice calls are
started on UE1, UE2 and UE3. Next an automated R99 PS call is
performed on UE4. Once the R99 bearer is established then an
automated FTP session is started and connected to the data server
(FTP server) within the test environment. After connection is
established files are repeatedly downloaded from the data server
(FTP server) and times/data rate recorded. Next files are
repeatedly uploaded to the data server (FTP server) and times/data
rates recorded. All UL and DL throughputs are to be measured at FTP
application layer (includes TCP, RLC, RF overheads) over entire
file transfer period. The intent is to measure the time necessary
to perform a FTP transfer of an ASCII file, from the FTP Server to
the UE in DL and from UE to FTP Server in UL. The duration can be
measured as the time elapsed between; First FTP data packet
received on the mobile/Server and Last TCP ACK FIN received by
mobile acknowledging all TCP packets. Both messages can be captured
with WireShark on test PC that issued the FTP. Also the reported
times/data rates as reported by the FTP client application may also
be used for results.
The following file sizes should be used throughput testing and
bearer configurations:1MB file for UL and DL R99 RAB 64 kbps and
128 kbps10 MB file for UL and DL R99 RAB 384 kbps
Automated test will be split evenly between the various R99
bearer configuration options for R99 UL and DL. Such that UL 64
(16.6%), UL 128 (16.6%), UL 384 (16.6%), DL 64 (16.6%), DL128
(16.6%), DL384 (16.6%). Test pass criteria is throughput than 94 %
of corresponding radio bearer physical layer bit rate.PS Data -
R99Developed 10 iteration testedTest case has been run successfully
with 10 iterationsPass58k UL 119kUL 367kUL-u-dragon ftp issue 60kDL
120kDL 360kDL60k UL 120kUL 360kUL 60kDL 120kDL
360kDL8/18/11Serial:Software version: BSR-03.00.8_pHardware
version: v2Power: 250mW Femto 709IOT Cluster3 (SGSN 3, MSC3)Release
2.13Pass0100000010000nFTS-83234HSxPA avg thoughput % of peak on air
interface (measured at FTP application)Three Simultaneous Voice
Calls Are Already EstablishedR17Test Ref: KPI-3.0-057 old
KPI-2.4-022Requirement: FTS-83234 R17, FTS-106897 R31Test
Description: As per KPI-2.4-018 with the exception that prior to
the automated R5 PS call starting, 3 UE's located on the same Femto
are in a CS call. This KPI should also be automated to run with
HSUPA as per KPI-2.3-014 and with 3 UE's located on the same Femto
in a CS call. For HSUPA test Category 3 UE should be used. For
HSDPA Category 8 UE should be used.PS Data HSDPADeveloped 10
iteration testedTest case has been run successfully with 10
iterationsPass3 voice calls + 1 packetAverage DL data rate is
5250Kb/s - 7.5MbpsAverage UL data rate is 1187Kb/s - 1.2
mbps1176kb/s11/18/11Serial:Software version: BSR-03.00.22_pHardware
version: v2Power: 250mW Femto 709Serial:1109450002Software version:
BSR-02.04.50.42Hardware version: DNI v2.5 EE Power: 250mW Femto
705Pass0100000010000yFTS-83234PS User Data R99 Throughput In Uplink
Per femto CPE when One Simultaneous Voice Call Already Established
In Very Close Proximity To The femto CPE. % of max bit rate.
(measured at FTP application)R18Test Ref: KPI-3.0-058 old
KPI-2.4-019Requirement: FTS-83234 R18, FTS-106897 R32Test
Description: An automated CS voice call is started on UE1. Next an
automated R99 PS call is performed on UE2. Once the R99 bearer is
established then an automated FTP session is started and connected
to the data server (FTP server) within the test environment. After
connection is established files are repeatedly downloaded from the
data server (FTP server) and times/data rate recorded. Next files
are repeatedly uploaded to the data server (FTP server) and
times/data rates recorded. All UL and DL throughputs are to be
measured at FTP application layer (includes TCP, RLC, RF overheads)
over entire file transfer period. The intent is to measure the time
necessary to perform a FTP transfer of an ASCII file, from the FTP
Server to the UE in DL and from UE to FTP Server in UL. The
duration can be measured as the time elapsed between; First FTP
data packet received on the mobile/Server and Last TCP ACK FIN
received by mobile acknowledging all TCP packets. Both messages can
be captured with WireShark on test PC that issued the FTP. Also the
reported times/data rates as reported by the FTP client application
may also be used for results.
The following file sizes should be used throughput testing and
bearer configurations:1MB file for UL and DL R99 RAB 64 kbps and
128 kbps10 MB file for UL and DL R99 RAB 384 kbps
Automated test will be split evenly between the various R99
bearer configuration options for R99 UL and DL. Such that UL 64
(16.6%), UL 128 (16.6%), UL 384 (16.6%), DL 64 (16.6%), DL128
(16.6%), DL384 (16.6%). Test pass criteria is throughput than 94 %
of corresponding radio bearer physical layer bit rate.PS Data -
R99Developed 10 iteration testedTest case has been run successfully
with 10 iterationsPass60k UL 118kUL 361kUL-udragon ftp issue 60kDL
120kDL 360kDL60k UL 120kUL 360kUL 60kDL 120kDL
360kDL8/18/11Serial:Software version: BSR-03.00.8_pHardware
version: v2Power: 250mW Femto 709IOT Cluster3 (SGSN 3, MSC3)Release
2.12Pass0100000010000nFTS-83234FTP R99 Throughput % of max bit
rate. (measured at FTP application)Three/One/Seven Simultaneous
Voice Calls Are Already Established
Test Requirement:FTS-83234 (BCR 2.3) R16,18,19FTS-106897 (BCR
2.4) R33R19Test Ref: KPI-3.0-059 old KPI-2.4-021Requirement:
FTS-83234 R16, FTS-106897 R33Test Description: As per KPI-2.4-015
with the exception that prior to the automated R99 PS call
starting, 3 UE's located on the same Femto are in a CS call.
Requirement: FTS-83234 R18, FTS-106897 R33Test Description: As
per KPI-2.4-015 with the exception that prior to the automated R99
PS call starting, 1 UE's located on the same Femto are in a CS
call.
Requirement: FTS-83234 R19, FTS-106897 R33Test Description: As
per KPI-2.4-015 with the exception that prior to the automated R99
PS call starting, 7 UE's located on the same Femto are in a CS
call.PS Data - R99Developed 10 iteration testedTest case has been
run successfully with 10 iterationsPass60k UL 118kUL 364kUL-udragon
ftp issue 60kDL 120kDL 360kDL60k UL 120kUL 360kUL 60kDL 120kDL
360kDL8/18/11Serial:Software version: BSR-03.00.8_pHardware
version: v2Power: 250mW Femto 709IOT Cluster3 (SGSN 3, MSC3)Release
2.14Pass0100000010000nFTS-83234HSxPA avg thoughput % of peak on air
interface (measured at FTP application)Three/One/Seven Simultaneous
Voice Calls Are Already EstablishedR20Test Ref: KPI-3.0-060 old
KPI-2.4-023Requirement: FTS-83234 R17, FTS-106897 R34Test
Description: As per KPI-2.4-018 with the exception that prior to
the automated R5 PS call starting, 3 UE's located on the same Femto
are in a CS call. This KPI should also be automated to run with
HSUPA as per KPI-2.3-014 and with 3 UE's located on the same Femto
in a CS call. For HSUPA test Category 3 UE should be used. For
HSDPA Category 8 UE should be used.
Requirement: FTS-106897 R34Test Description: As per KPI-2.4-018
with the exception that prior to the automated R5 PS call starting,
1 UE's located on the same Femto are in a CS call. This KPI should
also be automated to run with HSUPA as per KPI-2.3-014 and with 3
UE's located on the same Femto in a CS call. For HSUPA test
Category 3 UE should be used. For HSDPA Category 8 UE should be
used.
Requirement: FTS-83234 R20, FTS-106897 R34Test Description: As
per KPI-2.4-018 with the exception that prior to the automated R5
PS call starting, 7 UE's located on the same Femto are in a CS
call. This KPI should also be automated to run with HSUPA as per
KPI-2.3-014 and with 3 UE's located on the same Femto in a CS call.
For HSUPA test Category 3 UE should be used. For HSDPA Category 8
UE should be used.PS Data HSDPADeveloped 10 iteration testedTest
case has been run successfully with 10 iterationsPass1 voice call +
1 packet ( for all 7.5Mbps DL and 1.2Mbps UL)Average DL data rate
is 5109 Average UL data rate is 1186 3 voice calls + 1
packetAverage DL data rate is 5414 Average UL data rate is 1187 7
voice calls + 1 packetAverage DL data rate is 5182 Average UL data
rate is 11771176kb/s 5040kb/s11/18/11Serial:Software version:
BSR-03.00.22_pHardware version: v2Power: 250mW Femto 709IOT
Cluster3 (SGSN 3, MSC3)Release 2.311/23/10changed ul datarate due
to cpu overload sae mr2053 > 50% of ul rate
~700Kb/sPass0100000010000BCR2.2yFTS-77715Success rate of CS calls
for MOC, MTC and MTMR2Test Ref: KPI-3.0-061 old
KPI-2.4-001Requirement: FTS-77715 R2, FTS-106897 R37Test
Description: Automated CS calls performing a mixture of MOC, MTC
and MTM within 1 Femto. The mixture of calls will be evenly split
33.3% MOC, 33.3% MTC and 33,3% MTM. The success rate will be
calculated and recorded. Success rate and pass criteria are defined
as:CS Call Setup Success Rate = (CS Successful RRC Connection
Establishment Rate) * (1-(Standalone CS SRB Drop Rate/100)) * CS
RAB Establishment Success Rate )/100. CS Voice Call Setup Success
Rate for MOC, MTC and MTM shall be greater than 99%CS
VoiceDeveloped 10 iteration testedTest case has been run
successfully with 10 iterationsPass100%99%8/30/11Serial:Software
version: BSR-03.00.8_pHardware version: v2Power: 250mW Femto 709IOT
Cluster2 (SGSN 3, MSC3)Release
3.0Pass0100000010000yFTS-77715Success rate of PS callsR3Test Ref:
KPI-3.0-062 old KPI-2.4-008Requirement: FTS-77715 R3, FTS-106897
R38Test Description: Automated repeated PS calls within 1 Femto. PS
calls will be evenly split between R99 33.3%, R5 (HSDPA) 33.3% and
R6 (HSUPA) 33.3%. The success rate will be calculated and recorded.
Success rate and pass criteria are defined as:PS Call Setup Success
Rate = ( PS Successful RRC Connection Establishment Rate) * (
1-(Standalone PS SRB Drop Rate/100)) * PS RAB Establishment Success
Rate /100PS Call Setup Success Rate shall be greater than 99%.
Results for R99, R5 and R6 will be stored separately. Where
possible within the automation environment the same UE device will
be used for R99, R5 and R6.PS Data HSDPA & R99Developed 10
iteration testedTest case has been run successfully with 10
iterationsPass100%99%8/24/11Serial:Software version:
BSR-03.00.8_pHardware version: v2Power: 250mW Femto 709IOT Cluster3
(SGSN 3, MSC3)Release 2.11Pass0100000010000yFTS-77715Success rate
of CSD callsR4Test Ref: KPI-3.0-063 old KPI-2.4-004Requirement:
FTS-77715 R4, FTS-106897 (BCR 2.4) R39Test Description: Automated
CSD MTM call within 1 Femto. The success rate will be calculated
and recorded. Success rate and pass criteria are defined as: CS
Call Setup Success Rate = (CS Successful RRC Connection
Establishment Rate) * (1-(Standalone CS SRB Drop Rate/100)) * CS
RAB Establishment Success Rate )/100. CS Data Call Setup Success
Rate for MTM shall be greater than 99%CS DataDeveloped 10 iteration
testedTest case has been run successfully with 10
iterationsPass100%99%7/28/11Serial:Software version:
BSR-03.00.10.38Hardware version: v1.5Power: 100mW Femto 322IOT
Cluster2 (SGSN 3, MSC3)Release 3.0Pass0100000010000yFTS-77715CS
Voice Drop Call RateR6Test Ref:KPI-3.0-064 old
KPI-2.4-002Requirement: FTS-77715 R6, FTS-106897 R41Test
Description: Automated CS calls are performed using 1 Femto. Calls
are to be held for a minimum of 2 minutes. Then call should be
released. Any dropped calls within the 2 minute period should be
recorded. CS drop call rate is defined as the total number of
abnormally released CS RAB over the total number of Successful RAB
Establishments for CS. Pass criteria is CS voice drop call rate
shall be less than 0.5%CS VoiceDeveloped 10 iteration testedTest
case has been run successfully with 10
iterationsPass0%0.5%8/30/11Serial:Software version:
BSR-03.00.8_pHardware version: v2Power: 250mW Femto 709IOT Cluster2
(SGSN 3, MSC3)Release 3.0Pass0100000010000yFTS-77715CS Data Drop
Call Rate (call duration 2 mins)R7Test Ref:KPI-3.0-065 old
KPI-2.4-005Requirement: FTS-77715 R7, FTS-106897 R42Test
Description: Automated CSD MTM call within 1 Femto. Calls are to be
held for a minimum of 2 minutes. Then call should be released. Any
dropped calls within the 2 minute period should be recorded. CSD
drop call rate is defined as the total number of abnormally
released CSD RAB over the total number of Successful RAB
Establishments for CSD. Pass criteria is that CSD drop call rate
shall be less than 0.6%.CS DataDeveloped 10 iteration testedTest
case has been run successfully with 10
iterationsPass0%0.6%7/28/11Serial:Software version:
BSR-03.00.10.38Hardware version: v1.5Power: 100mW Femto 322IOT
Cluster2 (SGSN 3, MSC3)Release 3.07/23/10Current results are MTM
100 iterations
onlyhttp://svn.malab.uk.alcatel-lucent.com/svn/repos/FEMTO-Automation/reports/nlt/KPI/saved_test_reports/KPI-2.4-005_2010-07-23_17-56-48.htmlSerial:2150254412Software
version: BSR-02.04.40.28Hardware version: v1.2Power: 20mWIOT
Cluster2 (SGSN 3, MSC3)Release 2.37/30/10Current results are MTM 13
iterations
onlyhttp://svn.malab.uk.alcatel-lucent.com/svn/repos/FEMTO-Automation/reports/nlt/KPI/saved_test_reports/KPI-2.4-005_2010-07-30_16-08-38.htmlSerial:2150254412Software
version: BSR-02.04.40.34Hardware version: v1.2Power: 20mWIOT
Cluster1 (SGSN 3, MSC3)Release 2.4Pass0100000010000nFTS-77715Femto
-> Macro Handover Failure Rate From BSR Femto To 3G Macro
Network Intra Frequency in handover regionR8Test Ref: KPI-3.0-066
old KPI-2.4-035Requirement: FTS-77715 R8, FTS-106897 R43Test
Description: This KPI should have the following pre-requites in
place. The Femto is appropriately configured such that the 3G Macro
is present in the intra-frequency neighbour list. An automated
controllable attenuation system is in place and can control Femto
and 3G Macro. The attenuation control is set such that the UE can
only see the Femto and marginally see the 3G Macro, or not see the
3G Macro at all. Next the automated UE is placed into a CS voice
call once the call has been successfully held for approximately 10
seconds the attenuators are cross faded such that the neighbouring
3G Macro is a better candidate, quality (Ec/No), and signal
strength (RSCP) than the serving Femto. The automated UE should
maintain the CS call and correctly handover to the 3G Macro and
continue to maintain the call for at least 30 seconds once the
handover is completed. We can verify that the UE has performed the
handover onto the correct 3G Macro by checking the cell id. Next
the automated UE call is ended and the attenuators reconfigured
such that UE can only see the Femto and marginally see the 3G
Macro, or not see the 3G Macro at all. As the automated UE is in
idle mode it should perform a cell reselection back to the Femto
cell and this can be verified by checking the cell id to verify the
UE is back on the original test Femto. The test is now ready to be
repeated.
Femto to Macro Cell Handover Failure Rate is defined as the
number of unsuccessful handovers from BSR Femto to 3G Macro Network
intra-frequency, in the handover region, relative to the total
number of handover attempts. The number of unsuccessful handovers =
(number of handover attempts) - (number of Successful handovers).
The Femto -> Macro Handover Failure Rate shall be less than
1%.Mobility - Femto To 3G - IntraDeveloped 10 iteration testedTest
case has been run successfully with 10 iterationsPass0% Macro
HandoverFailure Rate From BSR Femto To 3G Macro Network Inter
Frequency in handover regionR9Test Ref: KPI-3.0-067 old
KPI-2.4-036Requirement: FTS-77715 R9, FTS-106897 R44Test
Description: This KPI should have the following pre-requites in
place. The Femto is appropriately configured such that the 3G Macro
is present in the inter-frequency neighbour list. An automated
controllable attenuation system is in place and can control Femto,
3G Macro and if required a noise source. The attenuation control is
set such that the UE can only see the Femto and marginally see the
3G Macro, or not see the 3G Macro at all. Next the automated UE is
placed into a CS voice call once the call has been successfully
held for approximately 10 seconds the attenuators are cross faded,
and noise source potentially increased, such that the neighbouring
3G Macro is a better candidate, quality (Ec/No), and signal
strength (RSCP) than the serving Femto. The automated UE should
maintain the CS call and correctly handover to the 3G Macro and
continue to maintain the call for at least 30 seconds once the
handover is completed. We can verify that the UE has performed the
handover onto the correct 3G Macro by checking the cell id. Next
the automated UE call is ended and the attenuators reconfigured
such that UE can only see the Femto and marginally see the 3G
Macro, or not see the 3G Macro at all. As the automated UE is in
idle mode it should perform a cell reselection back to the Femto
cell and this can be verified by checking the cell id to verify the
UE is back on the original test Femto. The test is now ready to be
repeated.
Femto to Macro Cell Handover Failure Rate is defined as the
number of unsuccessful handovers from BSR Femto to 3G Macro Network
inter-frequency, in the handover region, relative to the total
number of handover attempts. The number of unsuccessful handovers =
(number of handover attempts) - (number of Successful handovers).
The Femto -> Macro Handover Failure Rate shall be less than
1%.Mobility - Femto To 3G - InterDeveloped 10 iteration testedTest
case has been run successfully with 10
iterationsPass100%99%9/1/11Serial:Software version:
BSR-03.00.8_pHardware version: v2Power: 250mW Femto 709IOT Cluster3
(SGSN 3, MSC3)Release 2.15Pass0100000010000yFTS-77715Femto ->
Macro 2G Handover Failure Rate in handover regionR10Test Ref:
KPI-3.0-068 old KPI-2.4-038Requirement: FTS-77715 R10, FTS-106897
R45Test Description: This KPI should have the following
pre-requites in place. The Femto is appropriately configured such
that the 2G Macro is present in the inter-RAT neighbour list. An
automated controllable attenuation system is in place and can
control Femto and 2G Macro. The attenuation control is set such
that the UE can only see the Femto and marginally see the 2G Macro,
or not see the 2G Macro at all. Next the automated UE is placed
into a CS voice call once the call has been successfully held for
approximately 10 seconds the attenuators are cross faded such that
the neighbouring 2G Macro is a better candidate, quality (Ec/No),
and signal strength (RSCP) than the serving Femto. The automated UE
should maintain the CS call and correctly handover to the 2G Macro
and continue to maintain the call for at least 30 seconds once the
handover is completed. We can verify that the UE has performed the
handover onto the correct 2G Macro by checking the cell id. Next
the automated UE call is ended and the attenuators reconfigured
such that UE can only see the Femto and marginally see the 2G
Macro, or not see the 2G Macro at all. As the automated UE is in
idle mode it should perform a cell reselection back to the Femto
cell and this can be verified by checking the cell id to verify the
UE is back on the original test Femto. The test is now ready to be
repeated.
Femto to Macro Cell Handover Failure Rate is defined as the
number of unsuccessful handovers from BSR Femto to 2G GSM Macro, in
the handover region, relative to the total number of handover
attempts. number of unsuccessful handovers = (number of handover
attempts) - (number of Successful handovers) Femto -> Macro
Handover Failure Rate should be less than 1%.Mobility - Inter
RATDeveloped Partially TestedU-Dragon
issuePass100%99%9/28/11Serial:Software version:
BSR-03.00.16_pHardware version: v1.2IOT Cluster1 (SGSN 3,
MSC3)Release 2.0Pass0010000010000yFTS-77715FTP - R99 Throughput on
UL 64, UL 128, UL 384, DL 64, DL128, DL384. % of max bit
rate.(measured at FTP application)R15Test Ref: KPI-3.0-069 old
KPI-2.4-015Requirement: FTS-77715 R15, FTS-106897 R47Test
Description: Automated R99 PS call is performed on 1 Femto cell.
Once the R99 bearer is established then an automated FTP session is
started and connected to the data server (FTP server) within the
test environment. After connection is established files are
repeatedly downloaded from the data server (FTP server) and
times/data rate recorded. Next files are repeatedly uploaded to the
data server (FTP server) and times/data rates recorded. All UL and
DL throughputs are to be measured at FTP application layer
(includes TCP, RLC, RF overheads) over entire file transfer period.
The intent is to measure the time necessary to perform a FTP
transfer of an ASCII file, from the FTP Server to the UE in DL and
from UE to FTP Server in UL. The duration can be measured as the
time elapsed between; First FTP data packet received on the
mobile/Server and Last TCP ACK FIN received by mobile acknowledging
all TCP packets. Both messages can be captured with WireShark on
test PC that issued the FTP. Also the reported times/data rates as
reported by the FTP client application may also be used for
results.
The following file sizes should be used throughput testing and
bearer configurations:1MB file for UL and DL R99 RAB 64 kbps and
128 kbps10 MB file for UL and DL R99 RAB 384 kbps
Automated test will be split evenly between the various R99
bearer configuration options for R99 UL and DL. Such that UL 64
(16.6%), UL 128 (16.6%), UL 384 (16.6%), DL 64 (16.6%), DL128
(16.6%), DL384 (16.6%). Test pass criteria is throughput than 94 %
of corresponding radio bearer physical layer bit rate.PS Data -
R99Developed 10 iteration testedTest case has been run successfully
with 10 iterationsPass60k UL 118kUL 369 kUL-udragon ftp issue 60kDL
120kDL 360kDL60k UL 120kUL 360kUL 60kDL 120kDL
360kDL8/18/11Serial:Software version: BSR-03.00.8_pHardware
version: v2Power: 250mW Femto 709IOT Cluster1 (SGSN 3, MSC3)Release
2.43/30/10Test run with 102 iterations. 17 iteratins per
combinationUL64, UL128, UL384, DL64, DL128,
DL384KPI-2.3-015svn://svn.malab.uk.alcatel-lucent.com/FEMTO-Automation/trunk/KPI/saved_test_reports/KPI-2.3-015_2010-03-30_16-04-14.htmlsvn://svn.malab.uk.alcatel-lucent.com/FEMTO-Automation/trunk/KPI/saved_test_reports/KPI-2.3-015_2010-03-30_16-04-14.fftzSerial:2150254412Software
version: BSR-02.03.10.64Hardware version: v1.2Power: 20mW Femto
306IOT Cluster2 (SGSN 3, MSC3)Release 2.34/16/10Iterations was 100
per bearer combination per UL/DL. Total iteration
600KPI-2.3-015http://svn.malab.uk.alcatel-lucent.com/svn/repos/FEMTO-Automation/trunk/KPI/saved_test_reports/KPI-2.3-015_2010-04-09_15-56-02.htmlhttp://svn.malab.uk.alcatel-lucent.com/svn/repos/FEMTO-Automation/trunk/KPI/saved_test_reports/KPI-2.3-015_2010-04-09_15-56-02.fftzSerial:2150254412Software
version: BSR-02.03.10.64Hardware version: v1.2Power: 20mWIOT
Cluster2 (SGSN 3, MSC3)Release 2.34/12/10Iterations was 17 per
bearer combination per UL/DL. Total iteration
102KPI-2.3-015http://svn.malab.uk.alcatel-lucent.com/svn/repos/FEMTO-Automation/trunk/KPI/saved_test_reports/KPI-2.3-015_2010-04-12_16-04-20.htmlhttp://svn.malab.uk.alcatel-lucent.com/svn/repos/FEMTO-Automation/trunk/KPI/saved_test_reports/KPI-2.3-015_2010-04-12_16-04-20.fftzSerial:2150254412Software
version: BSR-02.03.10.66Hardware version: v1.2Power: 20mWIOT
Cluster2 (SGSN 3, MSC3)Release 2.3Pass0100000010000yFTS-77715HSxPA
avg thoughput % of peak on air interface (measured at FTP
application)R16Test Ref: KPI-3.0-070 old KPI-2.4-018Requirement:
FTS-106897 R48Test Description: Automated HSxPA PS call is
performed on 1 Femto cell. Once the HxDPA bearers (HSDPA and HSUPA)
are established then an automated FTP session is started and
connected to the data server (FTP server) within the test
environment. After connection is established files are repeatedly
downloaded and also uploaded from the data server (FTP server) and
times/data rate recorded. All seperate DL and UL throughputs are to
be measured at FTP application layer (includes TCP, RLC, RF
overheads) over entire file transfer period. The intent is to
measure the time necessary to perform a FTP transfer of an ASCII
file, from the FTP Server to the UE in DL and from UE to FTP Server
in UL. The duration can be measured as the time elapsed between;
First FTP data packet received on the mobile/Server and Last TCP
ACK FIN received by mobile acknowledging all TCP packets. Both
messages can be captured with WireShark on test PC that issued the
FTP. Also the reported times/data rates as reported by the FTP
client application may also be used for results. For HSxPA 10MB
file size should be used. Pass criteria that single UE HSDPA
average user throughput shall be greater than 84.00% of peak
physical layer bit rate for the corresponding UE category with
CQI> 24. All HSxPA tests should be conducted with a UE which is
capable of the maximum data rate supported by the Femto e.g.
current Femto BSR can support HSDPA Category 8, bearer of 7.2Mbps
and HSUPA Category 3, bearer 1.46Mbps.PS Data HSDPADeveloped 10
iteration testedTest case has been run successfully with 10
iterationsPass1347kb/s 4268kb/s1176kb/s 3364kb/s or V2
5040kb/s9/13/11Serial:Software version: BSR-03.00.13_pHardware
version: v2Power: 250mW Femto 709IOT Cluster3 (SGSN 3, MSC3)Release
2.1110/29/10Test run manual 10 iterationsSerial:2150254412Software
version: BSR-02.04.40.58Hardware version: v1.2Power: 20mW Femto
306IOT Cluster1 (SGSN 3, MSC3)Release
2.4Pass0100000010000yFTS-77715UL Voice Speech Quality - AMR 12.2
kbps Femto Mobile to PSTN/ISDNR17Test Ref: KPI-3.0-071 old
KPI-2.4-046Requirement: FTS-77715 R17, FTS-106897 R49Test
Description: An automated MOC AMR 12.2 kbps CS voice call to
PSTN/ISDN is established. A pre-recorded audio waveform is played
backed from the UE and received and recorded at PSTN/ISDN side. An
audio analysis and scoring tool (TBD) is then used to compare the
source and received signals to establish a Perceptual Evaluation of
Speech Quality (PESQ) score of the UL audio. PESQ is an algorithmic
non subjective (objective) audio quality score from which audio
scoring tools derive their unique Estimated Mean Opinion Score
(EMOS). It is preferable that PESQ scores are used for KPI as these
are comparable with other audio application PESQ scores where as
EMOS/MOS scores cannot be compared. Next the automated call is
ended and the test is repeated. The pass criteria is that the MOS
should be larger than 3.5 (mean), 3.2 (95%tile).Voice
QualityDeveloped 10 iteration testedTest case has been run
successfully with 10
iterationsPass4.0400>3..58/12/11Serial:Software version:
BSR-03.00.4_pHardware version: v2Power: 250mW Femto 709IOT Cluster3
(SGSN 3, MSC3)Release 2.20Pass0100000010000yFTS-77715DL Voice
Speech Quality - AMR 12.2 kbps PSTN/ISDN to Femto MobileR18Test
Ref: KPI-3.0-072 old KPI-2.4-047Requirement: FTS-77715 R18,
FTS-106897 R50Test Description: An automated MOC AMR 12.2 kbps CS
voice call to PSTN/ISDN is established. A pre-recorded audio
waveform is played backed from the PSTN/ISDN side and received and
recorded at the UE side. An audio analysis and scoring tool (TBD)
is then used to compare the source and received signals to
establish a Perceptual Evaluation of Speech Quality (PESQ) score of
the DL audio. PESQ is an algorithmic non subjective (objective)
audio quality score from which audio scoring tools derive their
unique Estimated Mean Opinion Score (EMOS). It is preferable that
PESQ scores are used for KPI as these are comparable with other
audio application PESQ scores where as EMOS/MOS scores cannot be
compared. Next the automated call is ended and the test is
repeated. The pass criteria is that the MOS should be larger than
3.5 (mean), 3.2 (95%tile).Voice QualityDeveloped 10 iteration
testedTest case has been run successfully with 10
iterationsPass3.5080>3..58/14/11Serial:Software version:
BSR-03.00.4_pHardware version: v2Power: 250mW Femto 709IOT Cluster3
(SGSN 3, MSC3)Release 2.213/14/1116 User
FemtoSerial:1109450002Software version: BSR-02.04.50.42Hardware
version: DNI v2.5 EE Power: 250mW Femto 702IOT Cluster3 (SGSN 4,
MSC3)Release 2.0Pass0100000010000yFTS-77715Voice Speech Quality -
AMR 12.2 kbps Femto Mobile to Femto MobileR19Test Ref: KPI-3.0-073
old KPI-2.4-045Requirement: FTS-77715 R19, FTS-106897 R51Test
Description: An automated MTM CS AMR 12.2 kbps voice call is
established. A pre-recorded audio waveform is played backed from
UE1 and received and recorded at UE2. An audio analysis and scoring
tool (TBD) is then used to compare the source and received signals
to establish a Perceptual Evaluation of Speech Quality (PESQ)
score. PESQ is an algorithmic non subjective (objective) audio
quality score from which audio scoring tools derive their unique
Estimated Mean Opinion Score (EMOS). It is preferable that PESQ
scores are used for KPI as these are comparable with other audio
application PESQ scores where as EMOS/MOS scores cannot be
compared. Next the automated call is ended and the test is
repeated. The pass criteria is that the MOS should be larger than
3.2 (mean), 3.0 (95%tile).Voice QualityDeveloped 10 iteration
testedTest case has been run successfully with 10
iterationsPass4.4720>3..28/8/11Serial:Software version:
BSR-03.00.4_pHardware version: v2Power: 250mW Femto 709IOT Cluster3
(SGSN 3, MSC3)Release 2.19Pass0100000010000nFTS-77715Voice Latency
Femto Mobile to Femto Mobile call excluding DSL contribution0Test
Ref: KPI-3.0-074 old KPI-2.4-063Requirement: FTS-77715 R20,
FTS-106897 R52Test Description: An automated MTM CS AMR 12.2 kbps
voice call is established. A pre-recorded audio waveform is played
backed from UE1 and received and recorded at UE2. An audio analysis
tool will be required to measure the voice call latency. Femto
Mobile to Mobile Voice call latency, call bearer delay, is one way
end-to-end delay for the speech channel when the call is in
progress. The end reference points are the Mouth Reference Point
(MRP) of one mobile and the Ear Reference Point (ERP) in the other
mobile. The pass criteria is such that the Femto Mobile to Femto
Mobile Voice call latency should be less than 200 msec mean and
(95% - 250 msec), excluding any delay intruduced by DSL.The latency
shall be end to end. DSL delay contribution is not easy to be
meaured.Voice QualityDeveloped 10 iteration testedManual
TestPass185 ms200ms9/28/11Serial:Software version:
BSR-03.00.15_pHardware version: v2Power: 250mW Femto 708IOT
Cluster3 (SGSN 3, MSC3)Release 2.15Pass0100000010000nFTS-77715Delay
between following power-on and femtocell service
availabilityR24Test Ref: KPI-3.0-075 old KPI-2.4-024Requirement:
FTS-77715 R24, FTS-106897 R56Test Description: Femto power-on to be
automated with the use of reboot command. Femto will be controlled
via automation system and rebooted, during reboot Femto to be
monitored and logged. Service availability will be measured from
the start of the reboot until Femto is correctly started and
transmitting and in service. Pass criteria is that Femto should be
in service in less than 3 minutes.AvailabilityDeveloped Partially
TestedResult to be check manuallyPass
