1

What is Field Testing & WhyWhat is Field Testing & Why• The main reason for Field Testing is to verify the compatibility of the

mobile terminal against different networks

• Field testing is carried out in 'live' commercial networks because

• Systematic lab tests cannot simulate all CDMA network configurations.

• Network functionality varies when comparing networks of different vendors.

• Although the network vendor may be the same, operators have possibilities to adjust network parameters in many ways.

• To provide a quality product by ensuring

• its complete interoperability with different infrastructures

• its robustness in good and adverse field conditions

2

• Field Testing should fulfill the following goals

• To discover such problems that can't be found out in systematic laboratory testing

• To confirm that all features of the phone will work without problems in all major cellular networks

• To ensure that the phone works according to end user expectations, mainly based on user manual

• To ensure that the quality of the product will satisfy typical end-user and the quality will be better than in the competitors phones.

Field Testing GoalsField Testing Goals

3

• Field test cases are aimed to cover all phone features

• Test cases are usually divided into following groups

• CDMA specific idle activity• Voice call handling and quality• Short Message Service (SMS)• Supplementary Services (SS)• Data• Internet / WAP

• Tests are also divided according to test environment

• Stationary tests (good signaling conditions, verifies signaling interoperability)• Tests when moving (Handovers HO, Location Updates LU, cell re-selection...)• Tests in weak field (searching for service, call crashing, call success rates...)

Test CategoriesTest Categories

4

• Level 1 : Lab Testing

• Level 2 : DEBUG and Feature verification Testing

• Level 3 : Local Field Test Network Testing

• Level 4 : Field Testing with Different Infrastructures / Network

Field Test LevelsField Test Levels

5

• Level 1 : Lab Testing• Objective : To evaluate the protocol status of the SW and the minimum

product performance needed to begin testing the product on a live network

• Type of Testing :

1. CDMA Protocol Test Rack 2. HP 8924C/E8285A Digital Parametric and Digital Call Processing Test 3. HP 8924C/E8285A Analog Parametric and Analog Call Processing Test 4. HP 8924C/E8285A Analog Parametric and Analog Call Processing Test 5. Mobile Originated Testing on HP8924C/E8285A & TOM 6. Mobile Terminated Testing on HP8924C/E8285A & TOM ……n. Final UI and Flash and Align

• Minimum Criteria to Pass :

• Product must pass Test n, no exceptions. • Product must pass all critical tests cases in Test 1-4 • Product must have minimum of 75% success rate on Test 5 & 6

Field Test LevelsField Test Levels

6

• Level 2 : DEBUG and Feature Verification Testing

• Objective : To verify fixed errors in the field, new feature implementation and perform minimal call statistics

• Type of Testing :

• Feature verification – will travel to different networks if feature is not supported in local network • Error/ Bug fix verification in local field test network • Call Origination and Termination in local field test network (no more than 100 calls per phone)• Call Maintenance in local field test network (no more than 5 hrs per phone)

• Minimum Criteria to Pass :

• Product must still pass all minimum criteria for Field Test Level 1• Product should have performance at least equal to “current products” being sold

Field Test LevelsField Test Levels

7

• Level 3 : Local Field Test Network Testing

• Objective : To evaluate the stability and performance of a product before taking it to other networks/infrastructures

• Type of Testing :

• All test cases outlined in the Project’s Field Test Specification • Call Origination and Termination in local field test network (min 100 calls per phone)• Call Origination and Termination in local field test network (no more than 100 calls per phone)• Call Maintenance in local field test network (min 10 hrs per phone)

• Minimum Criteria to Pass :

• Product must pass all Tests in Field Test Level 1 • Product must have no power downs • Product should have performance at least equal to “current products” being sold• Product must pass critical test cases in Test 1 & 2

Field Test LevelsField Test Levels

8

• Level 4 : Field Testing with Different Infrastructures /Networks

• Objective : To evaluate product performance and stability versus different infrastructures, various network providers, and various markets

• Type of Testing :

• All test cases outlined in the Project’s Field Test Specification • Call Origination and Termination in local field test network (min 100 calls per phone)• Call Maintenance in local field test network (min 10 hrs per phone) • All Features testing that is specific to a particular network • Error/bug fix verification that is specific to a particular network

• Minimum Criteria to Pass :

• Product must pass all minimum criteria in Field Test Level 3 • Product must pass all test cases in Test 1 & 2 for all infrastructures • Product must pass all Features requirement for a particular Carrier/Customer • Product must perform better than “current products” being sold

Field Test LevelsField Test Levels

9

The Field Testing ProcessThe Field Testing Process

Preliminary Check Preliminary Field Testing Field Testing

Level 1

Lab Testing

Level 2

PreliminaryField

Testing

Level 3

MainCountries

Level 3

All MajorCountries

Level 4

Correctionverification

ProductDevelopment

Final Mobile

No

Yes

Errors

Corrections

Error Correction/Internal Testing

Prepared forPreliminary

FT

Prepared for FT

Integration Testing

Regression Testing

Corrections Corrections

Errors

Errors

Correct-ions

L 3 L 4L 2L 2

10

Field Test ResponsibleField Test Responsible• Field test responsible is a person who organises the FT

activities• Field test responsible will do

• Planning for:• Resources• Timing (protos, accessories and SW)• Estimated cost calculation• Country selection (according network infrastructure, subscribers and services)• Tests to be executed

• Arrangements for:• Field test training• Field test resources• Contact persons in target countries and local SIM-cards• Information about network status, vaccination, visa, ata carnet, ..• Co-operation with other projects• Dedicated plan for each field test trip

11

Field Test ResponsibleField Test Responsible• Field test responsible will do (cont)

• Reporting for: • test manager (status reports) • project (errors /usability issues / maturity estimations)

• Follow up work for:

• found errors (error reproducing at office / fix verification)• error check lists• FT reports• FT plan• HW/SW Updates• local & global field test database• network news