|

Writing Bug Reports & uTest Etiquette Presented by Nicola Sedgwick June 20, 2012

| 2

1. What makes a good bug report?

2. Bug Titles

3. Reproduction steps

4. Bug attachments

5. uTest etiquette

6. A bug’s life after uTest

7. Wrap Up & Questions

Agenda

The Challenge

| | 3

What makes a good bug report?

| 4

What makes a good bug report?

Before we talk about how to log a report, we should look at why we log bugs.

In general: To assist in the

development of quality products.

Additionally at uTest:

To increase our rating.

To earn money. To learn.

To compete with our peers.

Always:

Attempting to avoid a situation where a launch or update results in a situation where

users lose data, customers move away to competitors or internal teams have to

perform any form of rollback procedure

Why do we log bugs?

| 5

What makes a good bug report?

Is there an industry standard?

Apple https://developer.apple.com/bugreporter/bugbestpractices.html

uTest Mozilla https://developer.mozilla.org/en/Bug_writing_guidelines

Bug Header Information

Product

Classification

Reproducibility

Version/Build Number

Bug Title & Description

Title

Description

Additional Information Requirements (General)

Configuration Information (System Profiler

Report)

Crashing Issues

Kernel Panics

Hanging/Performance Issues

Screen shots

Contact Information

Title

Bug Type

Frequency

Severity

Environment

Additional Environment Info

Action Performed

Expected Result

Actual Result

Error Message

Attachment

Preliminaries

Writing Precise Repro Steps

Writing a clear summary

Finding the correct product and

component (FireFox/Toolkit/Core)

Specific types of bugs (including crash

logs, stack trace, output of

about::memory, web page involved

depending on type of bug)

| 6

What makes a good bug report?

Common elements in most bug reports

Apple https://developer.apple.com/bugreporter/bugbestpractices.html

uTest Mozilla https://developer.mozilla.org/en/Bug_writing_guidelines

Title / Bug Title & Description

Description

Screen shots

Title

Action Performed / Expected

Result / Actual Result

Attachment

Writing Precise Repro Steps

Writing a clear summary

Specific types of bugs

The Challenge

| | 7

Bug Titles

– What’s the big deal?

|

Bug Titles

8

What’s the big deal?

• Summary

– A concise summary of the issue allowing you to:

- Compare bugs whilst looking at lists of titles

- Easily spot existing issues

• Understandability

– Can you perform initial bug triage without opening the bug details?

– What is the audience for the bug?

• Format assists list ordering

– Bug titles can be used to group bugs:

- Area of site – Problem within that area

- OS – Browser – Summary of problem

|

Bug Titles

9

The Good, The Bad & The Ugly

• “Checkout – MasterCard only - Error code 01

seen when submitting payment form”

– A good example of a bug confirming:

- The area of the site affected

– Any constraining factors

- The error that occurred and when it was seen

• “Crash”

– What crashed?

– After performing which action?

• “App never loads” but details say this is only

when there is no data connection

– Titles should accurately reflect the bug

The Challenge

| | 10

Reproduction steps

– telling the “story” of the bug

|

Reproduction Steps

11

Telling the “story” of the bug

• Minimum steps

– Bug reports should detail the quickest route to an issue

• Bullet points

Consider which of the the following are easiest to read when reproducing a reported issue:

– Load the eCommerce site at http://siteurl and follow the link to the Homeware section

and then to Kitchen. Find the item labelled “Saucepan set” and click on the “Add to

Basket” button.

– Or,

1. Load http://siteurl

2. Click on “Homeware”

3. Click on “Kitchen”

4. Click “Add to Basket” for

the “Saucepan set” item

|

Reproduction Steps

12

Telling the “story” of the bug – part 2

• Videos do not replace written reproduction steps

– Videos show exact situational detail

– Written steps provide “at a glance” information

• Historical significance

– Products evolve

– Staffing levels fluctuate

– Technologies change

• Eliminate Assumptions

– When working direct with a project team you may all have

the same implicit assumptions

- Address of test server, PC builds, Corporate virus checker, etc.

– When working externally to a project team you must state your assumptions

|

Reproduction Steps

13

Telling the “story” of the bug – part 3

• Bug Advocacy

– First step is finding a bug

– Next step, is convincing others that the bug is real and reproducible

– Then, the bug can be triaged, assessed and prioritised for fixing

The Challenge

| | 14

Bug attachments

- Necessity or nice added extra?

|

Bug attachments

15

Necessity or nice added extra?

•Proof that the bug occurred

– If others cannot reproduce a bug when following your

steps they still need to see the outcome

•Be careful of formats

– Are you using a generic file format or a one that others

will have to install particular software to view

•Screenshot or Video or Both?

– Snapshot of the page on which a link was clicked or an

error occurred

– Full visibility of where the user clicked and how long

screens took to load

•NOT a replacement for providing full repro steps

– Not searchable

The Challenge

| | 16

uTest Etiquette

|

uTest Etiquette

17

•Respect the creators of the

product you are testing

– Products are created by

real people

•Read the Scope & Instructions

•Clear and concise reproduction steps

– Avoid repro steps akin to flowery poetry

•Video attachments are not a replacement

for repro steps

– Videos confirm your reproduction steps

|

uTest Etiquette

18

•Follow bug title formats

– Helps TTLs & PMs when triaging bugs

– Helps other testers avoid logging duplicates

– May be requested by customers themselves

•Bug templates

– As with bug titles formats, a TTL, PM or Customer may request certain information

be included under “Additional environment info”. Templates should be followed

even if duplicating information (e.g. “Environment”)

•Avoid logging permutations of the same bug

– Repeating issues should be logged as a single bug unless otherwise requested

•Log quality bug reports to help customers produce quality products

– uTest ratings will change in the future to be take quality of bug reports into account

|

uTest Etiquette

19

•Use the discussion thread

– Problems loading/installing product

– Queries on scope

•Avoid emotive phrases in bugs

– Sometimes testing can be frustrating, especially with an early version of a product.

The bug report should detail the problem in a professional manner.

•Be professional at all times

The Challenge

| | 20

A bug’s life after uTest

– where do the bugs go?

|

A bug’s life after uTest

21

Where do bugs go after the cycles lock/close in uTest?

•Exported for, or by, the customer:

– To internal bug tracking systems

– As CSV files for internal analysis

•Passed to uTest bug verification cycles

– Will another tester understand your bug?

– Will you understand your bug if the verification

is requested months after logging?

•Used as a known bugs list for a follow-up test cycle

– Screenshots are unlikely to be accessible

– New uTesters may come on board who do not have access to earlier cycles

Pixar.com

| 22

Wrap Up

•Be respectful and professional

•Include all relevant information & stay within the scope

•Remember that everyone has a different perspective

The Challenge

| | 23

Thank You

Questions? Comics from:

Trenchescomic.com

Urbanjunglecomic.com

Cartoontester.blogspot.co.uk