Welcome! [] · •Code coverage •Headless test execution Show & Tell •Reporting •ALM...

Welcome!

If you haven‘t already set up, please come and get a USB stick!

You will need:

1. An installation of Jubula 7.2

2. The example projects

3. To perform the pre-tutorial checks in the PDF

11.03.2014 EclipseCon 2014 1

Jubula TutorialAlex Schladebeck

11.03.2014 2EclipseCon 2014

Agenda

Quick introduction to Jubula

Exercises: • Hands-on test specification and execution

• Automated testing in the project lifecycle

• Code coverage

• Headless test execution

Show & Tell• Reporting

• ALM integration

Further aspects

11.03.2014 3EclipseCon 2014

Philosophy 101:

Why do we test?• To gain information with the intention of making a decision

• Information can be:

• Acceptance – “this works“

• Regression – “this doesn‘t work (anymore)“

• Further questions – “is this a problem?“

Why do we automate?• To gain specific information quickly, frequently, reliably

• To reduce gaps in knowledge

11.03.2014 EclipseCon 2014 4

Constant feedback

Automated testing• Info during development

• Fewer nasty surprises

• Safety net

• Lower error costs

UI testing• User perspective at all times

11.03.2014 EclipseCon 2014 5

What is Jubula?

11.03.2014 EclipseCon 2014 6

What is Jubula?

Automated functional (black box) testing• Swing, SWT/RCP/GEF, HTML, .NET, iOS, JavaFX

Tests through the GUI• No / minimal difference to manual test

• All layers tested

• Complete workflows

• Intelligent tests

• Information about acceptance and regression

Jubula == GUIdancer

11.03.2014 7

http://www.eclipse.org/jubula

EclipseCon 2014

How is Jubula different?

11.03.2014 EclipseCon 2014 8

Quick demo: writing tests

Specify via drag and drop

Add data

Do object mapping

Execute and analyse

11.03.2014 EclipseCon 2014 9

Let‘s go!

Start the ITE

Import the projects

Open the EclipseCon project

Connect to the embedded AUT Agent

Check for any strange red squares

11.03.2014 EclipseCon 2014 10

Set up your perspective

11.03.2014 EclipseCon 2014 11

The AUT (Application Under Test)

DVD Manager (extraordinaire)

In 3 different versions• “Project“ lifecycle

11.03.2014 EclipseCon 2014 12

Version 1

Execute an existing test

Write a simple test

Write a less simple test

Finish off a test and run it

11.03.2014 EclipseCon 2014 13

0: Run an existing test

Project is imported and opened

Connected to AUT Agent

Start the AUT for your system in Version 1• E.g. DVDTool (Win_V1): Win_V1@localhost

Wait for AUT to start

Start Test Suite FIRSTTEST• Yes to relevant

• Yes to change perspective

• Hands off the keyboard!

11.03.2014 EclipseCon 2014 14

1: Create a simple test

Consists of actions on application and menu• Which means no object mapping

Use Case:• Open Waiting Dialog from Edit menu

• Unbound modules > Actions (basic) > Select > Menu Bar

• Wait for dialog

• Unbound modules > Actions (basic) > Wait > Application

• Wait for dialog to close

• Unbound modules > Actions (basic) > Wait > Application

11.03.2014 EclipseCon 2014 15

2: Create slightly less simple test

This time with other components (object mapping)

Use Case:• Open Help/Info

• Wait for Info Dialog

• Modules > Generic > Dialogs

• Check version number

• ubm > Actions (basic) > Check > Component with Text

• Click OK

• ubm > Actions (basic) > Click

11.03.2014 EclipseCon 2014 16

Sounds like weneed a module!

3: Finish off test for name uniqueness

Category names should be unique

Open Test Case: Check uniqueness of category names

Look at what we‘ve got already

Add final test actions:• Check enablement of OK button

• Ubm > Actions (basic) > Check

• Close dialog with cancel

• Modules > Generic > Dialogs

Do mapping and run FULLTEST

11.03.2014 EclipseCon 2014 17

Version 2

Check that previous bug is gone

Make any necessary changes

Write a new test

11.03.2014 EclipseCon 2014 18

4:Rerun uniqueness test on new version

Stop Version 1 AUT

Start Version 2 AUT

Rerun FULLTEST (to check that the bug is fixed)

Do any changes need to be made?

11.03.2014 EclipseCon 2014 19

5: Comment in deletion test and run

In the FULLTEST Test Suite• comment in the Delete Single Category Test Case

Run the test

Is everything ok?

11.03.2014 EclipseCon 2014 20

What do we do about errors?

Can we fix or deal with this error?

Yes:• RETRY Event Handler (later)

No / Don‘t Know• RETURN Event Handler

• Use restart / reset AUT to put AUT into correct state

11.03.2014 EclipseCon 2014 21

About event handling

Event Types• Action error

• Component not found

• Check failed

• Configuration error

Reentry types (test continuation)• Continue

• Exit

• Return

• Retry

• Pause

• Break

11.03.2014 EclipseCon 2014 22

Why do we need Event Handlers?

What if an error occurs?

11.03.2014 EclipseCon 2014 23

Why do we need Event Handlers?

11.03.2014 EclipseCon 2014 24

Using Event Handlers

11.03.2014 EclipseCon 2014 25

6: Adding a global Event Handler

Create normal Test Case : Restart• Add a restart action

• ubm > Actions (basic) > Restart Application > Restart

Add this Test Case as an Event Handler• To the Delete Category Test Case

• Error Type Action Error

• Reentry Type Return

Run the test

11.03.2014 EclipseCon 2014 26

..if an action error occurs in this Test Case, then restart the AUT, and continue the test at the nextTest Case in the hierarchy –it will be ready for you!

Version 3

Check that previous bug is gone

Check that there are no side effects

11.03.2014 EclipseCon 2014 27

7: Run test on new version

Stop version 2

Start version 3• (Mac Java 7 users: switch context menu action for menu action)

Re-run FULLTEST on latest version

11.03.2014 EclipseCon 2014 28

Return to Event Handling

What if the category to delete contains DVDs?

Check for non-existence of confirmation dialog• If it is there, close it and check again

11.03.2014 EclipseCon 2014 29

Retry EH

8: Code Coverage

Stop the AUT

Open the Project Properties• Select AUTs from the left

• Select DVDTool and select „Edit“

• Find your system_v3_CC configuration and select edit

• Open the Expert properties

• In the Jacoco Configuration

• Browse to the examples/AUTs/DVDTool directory on your system for:• The installation directory

• Click all the OKs

Start the AUT configuration for code coverage

11.03.2014 EclipseCon 2014 30

8: Code Coverage

Run the FULLTEST

Open the Reporting Perspective

See the code coverage values

Open the report

To discuss over a beer:• What should we use code coverage for?

• http://testing.bredex.de/code-coverage.html

http://www.eclemma.org/jacoco/

11.03.2014 EclipseCon 2014 31

What do we do with test results?

11.03.2014 EclipseCon 2014 32

Gathering and analysing test results

Daily test result analysis• Check e.g. Hudson jobs

• Ensure all tests ran (collectResults)

• Gather overview of status (Test Execution Overview)

• Analyse any tests that failed in the ITE / Dashboard / HTML Reports

• Add comments to Test Results Summary View

• Create issues in external systems from ITE

• Automatically report test results to external systems (v8)

• Print out comments report for e.g. standup discuss and react!

Long-term test trend overview• E.g. History report with code coverage

11.03.2014 EclipseCon 2014 33

http://www.eclipse.org/birt/phoenix/

Continuous Integration

11.03.2014 EclipseCon 2014 34

Develop

Commit

Check out

Build anddeploy

Setup environment

Run tests

Gather andanalyse results

Discussion andfeedback

testexec

ITE ResultHTML ResultDashboard ResultAutomated comment to ALM

(AUT Agent)

9: Headless execution (now/later)Locate the config.xml file

Make changes:• Change the autconfig parameter if you‘re not using Windows• Mac users – change ~ to actual path• Enter a writeable path for the datadir and resultdir parameters

Mac users – use an absolute path in the AUT configuration

Stop the AUT

Disconnect from the AUT Agent*

Minimize all windows

Via the command line, navigate to:• <jubulaInstallation>/jubula

Enter testexec(.exe) -c <pathToConfigFile>/config.xml

11.03.2014 EclipseCon 2014 35

* we‘re cheating by using the running embedded Agent. For a real CI test, use a separately started agent or, in V8, theembedded agent support in testexec

Closing some gaps with Mylyn

View, edit, create tasks in the ITE

Automatically report test results to external ALM systems

11.03.2014 EclipseCon 2014 36

www.eclipse.org/mylyn

Summary

Complete workflows tested based on acceptance criteria• Encourages early and constant discussion about quality

• Even before the application is available

• Automated regression testing

Manual intelligence incorporated• Test almost indistinguishable from user working with system

• Synchronization

• Reacting to clues in AUT

• Error handling

Reuse instead of copy• Tests made up of similar units – reuse to improve maintainability

11.03.2014 EclipseCon 2014 37

Current status

Version 8.0 will be released before Easter• JavaFX support

• Comment to ALM system after headless run

• Support for HP ALM connection

• Support for embedded Agent in testexec

JavaFX support will also be released with Luna

11.03.2014 EclipseCon 2014 38

Don‘t forget…

Please evaluate the talk!

For more testing fun, join me on Wednesday:• Grand Peninsula C - 11:15 to 11:50

Exploratory testing and session-based test management

11.03.2014 EclipseCon 2014 39

Homework: Continue writing tests

Startup

Import DVD library

Select category (Science Fiction)

Select DVD from table

Select technics tab

Select region from combo box

Select multiple languages

Select content tab

Enter content information

11.03.2014 EclipseCon 2014 40

Thanks!

Community help• Forum

• Mailing lists

• Webinar

• Examples & Articles

Professional services• Training

• Proof of Concept

• Reviews, Consulting

• Custom extensions

• Sponsored development

@alex_schl

alexandra.schladebeck@bredex.de

http://testing.bredex.de

www.bredexsw.com

www.eclipse.org/jubula

@EclipseJubula

11.03.2014 EclipseCon 2014 41

Sneak preview JavaFX

Simple Adder

Brick Breaker

Planned components for first release• Buttons, Radio Boxes, Check Boxes, Toggle Buttons

• Labels, Text

• Text input controls

• Combo boxes

• Content controls: Lists, Trees, Tables

• Accordeons

11.03.2014 EclipseCon 2014 42