Wix Automation - The False Positive Paradox

1

Wix AutomationThe False Positive Paradox

Efrat Attas

2

What?!What is “False Positive”?

3

What?!Release cycle

• Daily / Weekly deployments of multiple products• QA verifies each version using

manual and automation tests (accompanied by automation engineers)

4

What?!

Test failure not necessarily a bugFalse positives – What is it?

5

What?! Understand!False positives – What are the consequences?• Time consuming (in case of many

failures)• Bottleneck

6

Why?!Why is it happening?

7

Why?

• Unstable test – “flaky”• Outdated test

False positives – Why is it happening?

8

Why? Invent!Reduce false positives to a minimum• Identify • Isolate• Analyze• Fix the flakiness

9

How?!How to reduce outdated tests?

10

How?!

Bonus! Can also be used for “real” issues in product – true positives

Solution• Ignore On Bug annotation

Outdated test• Ignore test on bug with a link to Jira

issue

Motivation• Outdated test must be adapted to

new behavior without causing unnecessary failures during the refactor process

11

Solution

@IgnoreOnBug(issue = ”SE-5541”)@Test public void verifyButtonText() { … }

@IgnoreOnBug

12

Solution@IgnoreOnBug

@Retention(RetentionPolicy.RUNTIME) @Target(ElementType.METHOD) public @interface IgnoreOnBug {

String issue();

}

13

Solution@IgnoreOnBug

public interface TestObserver {

default void starting(Description description) { } default void before(Description description) { } default void ending() { } default void skipped(AssumptionViolatedException e, Description description) { }default void succeeded(Description description) { } default void failed(Throwable e, Description description) { } default void finished(Description description) { }

}

14

Solution@IgnoreOnBug

@Override public void before(Description description) { if(hasAnnotation(description, IgnoreOnBug.class)) {

String issue = getJiraIssueFromMethodAnnotation(description); if(!isJiraIssueSolved(issue)) {

ignoreTest(); } } }

15

How?!How to reduce unstable tests?

16

How?!Unstable test

* What is stable enough?

• New tests are executed independently, without affecting existing test results

Motivation• New test has to prove it is stable

enough* before entering the entire test collection

Solution• Beta test mechanism

17

Solution@BetaTest

• Beta tests will run in a specific configuration, X times every night• Each test will report its run result to a dedicated server• Beta tests which passed more than Y times will be

“approved”

18

Solution@BetaTest

@BetaTest(owner = "someone") public void verifyButtonText() { … }

19

Solution@BetaTest

@Retention(RetentionPolicy.RUNTIME) @Target({ElementType.METHOD, ElementType.TYPE}) public @interface BetaTest {

String owner();

}

20

Solution@BetaTest

Collection<Class<? extends Annotation>> testMethodAnnotations = Arrays.asList(Test.class, BetaTest.class);

@Override protected List<FrameworkMethod> computeTestMethods() {

return getAnnotatedMethods(getTestClass(), testMethodAnnotations); }

21

Solution@BetaTest

public interface TestObserver {default void starting(Description description) { } default void before(Description description) { } default void ending() { } default void skipped(AssumptionViolatedException e, Description description) { }default void succeeded(Description description) { } default void failed(Throwable e, Description description) { } default void finished(Description description) { }

}

22

Solution@BetaTest

@Override public void before(Description description) {

if (shouldIgnoreBetaTest(description)) {

ignoreTest("Test in beta mode - ignored"); }}

23

Solution@BetaTest

@Override public void succeeded(Description description) {

sendStatus(description, TestStatus.TEST_PASSED); }

@Override public void failed(Throwable e, Description description) {

sendStatus(description, TestStatus.TEST_FAILED); }

24

Solution@BetaTest

25

Thank you!Questions?