Property-based Testing – ProTestFP7 Strep

John Derrick

University of Sheffield,

and members of the ProTest team.

Consortium

University of Sheffield - USFD - UK

University of Kent - Kent - UK

Universidad Politécnica de Madrid - UPM - Spain

University of Göteborg - ITU1 - Sweden

Chalmers University of Technology - ITU2 - Sweden

Ericsson - Sweden

Erlang Training and Consulting Ltd - ETC - UK

Quviq AB - Sweden

LambdaStream - Lambda - Spain

MotivationCommunication networks, based on telephony,

wireless and Internet are converging.

Increasing numbers of services are being added to these merging networks, and services are becoming more complex.

Telecoms industry provides 5-nines reliability, ie 99.999% reliability, of core systems.

How to maintain 5-nines reliability in future service-oriented networks and systems.

Ensuring reliability

The software for new services and network devices is rapidly growing in complexity.

Key ingredients for ensuring reliability of complex systems: suitable architecture and technologies, improving the software process, and thorough and efficient in testing.

Aim to support the European software industry in its testing methodology for software in network and service infrastructures.

TestingTesting of complex systems is difficult and time-consuming. Radical approaches are needed, and will be welcomed by

industry — if they work.

In ProTest we build upon on the idea of using properties as objects for testing software. Eg random testing driven by LTL description of properties.

In order to deliver dynamic services and interoperable network applications with guaranteed properties, we focus testing around these properties.

Property-based testing

Property-based testing aims to deliver more effective tests, more efficiently, and so deliver economic benefits to the European software industry.

Testing with properties as objects improves the competitiveness of software developers, since they can deliver higher quality software for a lower price.

It also allows collaborating companies to improve the definition of their software interfaces and so improve the compatibility between their services.

We will deliver methods and tools to support property-based development of systems.

Project componentsIn order to deliver full benefits we need tools to integrate

property-based testing into the development life cycle:

Property discovery. Current testing is based on sets of test cases embedded in test suites;

we are building techniques and tools to aid the software developers to extract properties from this test data.

Current specifications and models are often informal: we are developing specialised property languages to ease the formalisation of existing specifications.

Test and property evolution. All software systems are subject to change and evolution; we are building refactoring tools to support the evolution of tests and properties in line with the evolution of the system itself.

Property monitoring. Not all properties can be tested in advance of systems being executed; not all faults will be found during testing.

We will provide tools to support the post hoc examination of trace details for conformance to particular constraints.

Analysing concurrent systems. At the heart of service oriented systems is concurrency. We are designing model-checkers so that concurrent systems can be analysed for fundamental properties.

Erlang/OTPProperty-driven development can be used in a variety of

programming languages and systems.

Chosen platform for initial implementation is Erlang/OTP, but dissemination into model-driven development arena (UML) and other languages (C/C++, Java, etc).

Erlang/OTP was developed by Ericsson during the early nineties as a platform for rapid development of network applications.

Erlang is a concurrent functional language with specific support for the development of distributed, fault-tolerant systems with soft real-time requirements.

Erlang/OTP - robustness and reliability within the telecoms sector, eg, in the implementation of the AXD301 ATM telecoms switch by Ericsson.

Applications of Erlang include computer telephony, banking, TCP/IP programming (HTTP, SSL, Email, Instant messaging, etc) and 3D-modelling.

There is a wide user base which over the years has grown beyond the telecoms sector, and companies such as Amazon are interested in the suitability of Erlang for programming massively concurrent distributed systems.

Erlang is particularly well placed to exploit multi-core processor chips: the concurrency model of Erlang allows separate processes to run on the different cores with no change to the source code.

ConsortiumMix of Universities, SMEs, and a larger industrial company.

The academics bring experience of testing, formal verification, language development, and refactoring.

Quviq is a spin-off, founded to commercialise an innovative property-based testing tool (QuickCheck).

System builders (Ericsson, LambdaStream), consultants, and trainers (ETC), provide insights into what is required of practical tools, what properties will need to be checked, and ways of fitting the results from the project into practical software development methods.

Overview - 5 technical WPs

WP1 - Property discovery

Property discoveryUsing QuickCheck - a tool combines random test case

generation from properties written in Erlang, with a flexible language for specifying generators. Automatic simplification of test cases.

Develop Domain Specific Languages for dealing with several fragments of UML. Likely to include behavioural aspects such as sequence diagrams.

Developing a method and prototype tool to obtain properties from combining a number of test cases.

Developed a technique to reverse-engineer testable models from program execution traces.

Being integrated into an automated iterative process, where hypothesised models are refined by testing them against the implementation.

WP2 - System evolution and assurance

System Evolution and Assurance

Refactoring means changing the design or structure of a program … without changing its behaviour.

Develop a method to refactor properties and test cases. Implemented in the Wrangler refactoring tool.

Refactoring of test cases and properties by connecting it to refactoring of the actual subject under test.

-module (test).-export([f/1]). add_one (N, [H|T]) -> [H+N | add_one(N,T)];

add_one (N,[]) -> [].

f(X) -> add_one(1, X).

-module (test).-export([f/1]). add_int (N, [H|T]) -> [H+N | add_int(N,T)];

add_int (N,[]) -> [].

f(X) -> add_int(1, X).

Generalisation

-module (test).-export([f/1]). add_one ([H|T]) -> [H+1 | add_one(T)];

add_one ([]) -> [].

f(X) -> add_one(X).

Wrangler

Embedded in Emacs … and Eclipse.

Structural, data type and module refactorings.

Tree-based analysis and transformation.

Work with multiple modules.

Support undo of refactorings

Preserve layout and comments.

Respects aspects of the macro system.

Issues in refactoring and testing

• Keeping track of names, including module names.

• Some refactorings carry over easily …

•… others require respecting of conventions.

• The role of makefiles and project management in system evolution …

• … and the question of integration: Erlide.

WP3 - Property monitoring

Property monitoring

The Erlang run-time system has a built-in trace recording functionality.

How to check properties for a trace.

Develop methods for presenting trace information to the user in a comprehensible way, and also present minimal counter-examples in case a property does not hold for a trace.

By studying the trace we can see which test cases that are efficient and what aspects of the system is not exercised by a test case.

WP4 - Analysis of concurrent systems

Model checking is a complementary technique to testing for verifying concurrent systems.

Model checking provides the possibility to fully verify a system.

Develop two model-checkers for Erlang: McErlang, and etomcrl - based on an abstraction of Erlang into the process algebra, mCRL.

We are integrating the QuickCheck based testing framework with the model checker. This involves harmonizing the property specification language of QuickCheck and McErlang, and we plan to investigate the link between model based specification and testing using QuickCheck.

WP5 - Property driven development

Undertake a larger case study that will be used to evaluate and assess the results in an industrial setting.

LambdaStream will use the property-based development methodology supported by the QuickCheck tool and refactoring techniques, and evaluate the property monitoring methodology and the model-checking tools and techniques.

More information: www.protest-project.eu

Papers at Refactoring Workshop

Papers at ACM SIGPLAN Erlang Workshop

Papers at:• IEEE/ACM conf on Automated Soft. Eng. (ASE’08)• International Workshop on Model Checking and AI• FORTE 2008, etc, etc.

A number of video’s, tutorials and lectures available. See webpages.

Questions?