Introduction to CTS July 7, 2010 Jingtao. What is CTS Compatibility Test System open source...

introduction to CTSJuly 7, 2010Jingtao

What is CTS

Compatibility Test System

open source

commercial-grade(20000+ cases)

Automation compatibility test tool

type of case

• Unit tests

• Functional tests

• Reference application tests

• Robustness tests

• Performance tests

what does it test

• Signature

• Platform API

• Dalvik VM

• Platform Data Model

• Intents

• Permissions

• Resources

basic topic

CTS history

CTS work flow

download, install and run CTS

check CTS report

CTS History

• Android1.0 - 16784 cases

• Cupcake - 21383 cases

• Donut - 22756 cases

• CTS2.1-early-look-r2(Eclair) - 22847 cases

• CTS2.1-r1 - 23107 cases

• CTS2.1-r2 - 23123 cases

• CTS2.1-r3 - 23138 cases

• Next ? Froyo

CTS work flow

download CTSbinary: http://source.android.com/compatibility/downloads.html

source: git ls-remote --heads git://android.git.kernel.org/platform/manifest.git

repo init -u git://android.git.kernel.org/platform/manifest.git -b android-cts-2.1_r3

repo sync

install CTS

install Android SDK

unzip CTS package

vi android-cts/tools/startcts, set SDK_ROOT=<absolute fold name of SDK>

run CTSconnect phone to PC

android-cts/tools/startcts to enter CTS shell

3 mode to start test:

plan mode

package mode

case mode

Sample of CTS commands

• start --plan CTS

• start --plan Android -p android.app.cts.ActivityManagerTest

• start --plan Android -t android.app.cts.ActivityManagerTest#testGetRunningServices

run CTS under Windows1. unzip CTS package and android SDK package to your windows PC.

2. Modified the SDK_ROOT in cts\android-cts\tools\startcts :

SDK_ROOT=D:/CTS/sdk/android-sdk_r05-windows/android-sdk-windows/tools

3. use below command to enter CTS shell:

java -Xmx512M -cp D:\CTS\cts\android-cts\tools\cts.jar; D:\CTS\cts\android-cts\tools\

hosttestlib.jar;D:\\lib\ddCTS\cts\android-cts\tools\junit.jar;D:\CTS\sdk\android-sdk_r05-windows\

android-sdk-windows\tools\lib\ddmlib.jar com.android.cts.TestHost D:\CTS\cts\android-cts\

repository\host_config.xml

Please change red marked to you current file path.

known issue:

The file (cts_result.css, cts_result.xsl, logo.gif, newrule-green.png ) under CTS result can not

automatically copy in new session folder.

Attention:

If you need to clear the data in addon emulator, Please use: emulator @addon -wipe-data

config phone before run CTS empty SD card

never Screen Timeout

English locale

install and set input method to LatinIME

open WIFI and GPRS

correct date/time

/opl/etc/mountd.conf, system.prop

CtsDelegatingAccessibilityService.apk

test report firefox repository/results/20xx.../testResult.xml

issue of CTS itself

Test Package Test Cases CTS Error

android.permission.cts.NoNetworkStatePermissionTest testStopUsingNetworkFeature will be fixed in future release

android.provider.cts.ContactsTest testGroupMembershipTable bug 2258907, needs investigation

android.util.cts.TimeUtilsTest testWorld bug 2323433, needs investigation

advance topic

design CTS plan

Modify testResult.xml

fix syntax error

run fail cases

Tool in CTS

design new cases

build CTS

Automation test on Android

CTS folder structure• |-- repository

|   |-- ...• |   |-- plans

|   |   |-- Android.xml|   |   |-- ...|   |   `-- CTS.xml|   |-- results|   |   |-- 2009.12.29_14.48.20|   |   |   |-- ...|   |   |   `-- testResult.xml|   |   |-- 2009.12.29_14.48.20.zip|   `-- testcases|       |-- CtsAppTestCases.apk|       |-- CtsAppTestCases.xml|       |-- ...|       |-- android.core.vm-tests.jar|       `-- android.core.vm-tests.xml`-- tools    |-- cts.jar    `-- startcts

CTS cases structurePlan CTS

apidemos

Android Java Performance RefApp VMSignature

vm-testspackages…contentapp widget

Test cases …

You plan…

Design your own CTS test plan 20000+ cases, 24+ hours

most cases never fail before, plan “Java”

plan “Android”, 3000+ cases, 3+ hours

specific test package for plan.

specific test case for package

content of plan “sanity”<?xml version="1.0" encoding="UTF-8" ?>

<TestPlan version="1.0">

<Entry uri="android.tests.sigtest"/>

<Entry uri="android.app"/>

<Entry uri="android.content"/>

<Entry uri="android.database"/>

<Entry uri="android.graphics"/>

<Entry uri="android.hardware"/>

<Entry uri="android.location"/>

<Entry uri="android.net"/>

<Entry uri="android.os"/>

<Entry uri="android.permission"/>

<Entry uri="android.provider"/>

<Entry uri="android.text"/>

<Entry uri="android.util"/>

<Entry uri="android.view"/>

<Entry uri="android.widget"/>

</TestPlan>

fail to open test report by firefox?

wait for test finished

vi testResult.xml, delete the error line

save and open it with firefox again

repeat to delete error line until can open by firefox

how to run fail case again?

open testResult.xml by gedit

replace all the result=”fail” to result=”notExecuted”

reset “Summary failed=” and “notExecuted=”

run tools/startcts, start --plan CTS, select Choose a session, usually select 1

run CTS more faster?• In respository/host_config.xml of CTS

• <!-- Number of tests executed between reboots. A value <= 0 disables reboots. -->

• <IntValue name="maxTestCount" value="200" />

• <!-- Max size [tests] for a package to be run in batch mode. -->

• <IntValue name="maxTestsInBatchMode" value="5000" />

• <!-- Max time [ms] between test status updates. -->

• <IntValue name="testStatusTimeoutMs" value="300000" />

• <!-- Max time [ms] from start of package in batch mode and the first test status update. -->

• <IntValue name="batchStartTimeoutMs" value="1800000" />

• <!-- Max time [ms] from start of test in individual mode to the first test status update. -->

• <IntValue name="individualStartTimeoutMs" value="300000" />

• ...

Tools in CTS

• Annotation helper

• Device info collector

• Process controller

• Test plan generator

• Signature generator

• Host configer

• Script starter

• …

develop new cases

• build/core/tasks/cts.mkCTS_CASE_LIST

• tests/tests/<yourpackage>/Android.mk(LOCAL_MODULE_TAGS, LOCAL_CLASSPATH, LOCAL_PACKAGE_NAME)

AndroidManifest.xml(package, android:label)

generate test description

buildCts.py <testRoot> <ctsOutputDir>

<tempDir> <androidRootDir> <docletPath>

e.g.

cts/tools/utils/buildCts.py cts/tests/tests

out/host/linux-x86/cts/android-cts /tmp .

out/host/linux-x86/framework/descGen.jar

build CTS

• . build/envsetup.sh

• tapas

• make cts

• out/host/linux-x86/cts/android-cts.zip

CTS in-depth

• java com.android.cts.TestHost in startcts

• cts/tools/host

• cts/tools/host/src/com/android/cts/TestHost.java

32