1 A Day in the Life of a Tester Irinel Crivat Irinel Crivat SDET Technical Lead ASP.NET

1

A Day in the Life of a TesterA Day in the Life of a Tester

Irinel CrivatIrinel Crivat

SDET Technical LeadSDET Technical Lead

ASP.NETASP.NET

http://www.asp.nethttp://www.asp.net

2

OverviewOverview

The purpose of testing a program is to The purpose of testing a program is to find problems in itfind problems in it

What a tester does?What a tester does? Design test case (automated & manual)Design test case (automated & manual) Find problems is the core of a tester workFind problems is the core of a tester work Write problem reportsWrite problem reports

3

What are we testing?What are we testing?Writing dynamic, high-performance Web applications has never been easier

ASP.NET helps you deliver real world Web

applications in record time.

ASP.NET combines developer productivity with performance, reliability, and deployment.

4

What are we testing?What are we testing?

Developer Productivity Easy Programming Model Flexible Language Options Flexible Language Options Great Tool SupportGreat Tool Support

Rich Class FrameworkRich Class Framework

Enhanced Reliability Memory Leak, Memory Leak,

DeadLock and Crash DeadLock and Crash Protection Protection

Easy Deployment

"No touch" application "No touch" application deploymentdeployment

Dynamic update of running Dynamic update of running applicationapplication

Easy Migration PathEasy Migration Path

New Application Models XML Web ServicesXML Web Services Mobile Web Device SupportMobile Web Device Support

Improved Performance and Scalability

Compiled execution Rich output cachingRich output caching Web-Farm Session StateWeb-Farm Session State

5

Feature CrewsFeature Crews

Feature CrewFeature Crew

““A small A small self-organizingself-organizing, , cross-disciplinecross-discipline group that group that works together to works together to completelycompletely deliver a discrete deliver a discrete Feature”.Feature”.

Crew has flexibility to decide how to work, how to meet Crew has flexibility to decide how to work, how to meet

quality barquality bar Feature: Independently testable functionality taking 1-6 Feature: Independently testable functionality taking 1-6

weeks of elapsed time to completeweeks of elapsed time to complete Crew: 1PM, 1-5 Dev, 1-5 Test, other disciplines as Crew: 1PM, 1-5 Dev, 1-5 Test, other disciplines as

appropriate appropriate No remaining work is left to do on a feature when it gets No remaining work is left to do on a feature when it gets

checked inchecked in interim builds of the product are higher qualityinterim builds of the product are higher quality Tasks are done earlier in the development cycleTasks are done earlier in the development cycle

6

Feature Crew TenetsFeature Crew Tenets

Small, agileSmall, agile work units – the smaller the work units – the smaller the betterbetter

Dedicated Dedicated resources to reduce context resources to reduce context switchingswitching

AutonomyAutonomy and decision making pushed to and decision making pushed to lowest levels – you decide how to work, who lowest levels – you decide how to work, who does whatdoes what

All members “All members “in it togetherin it together” – begin and end ” – begin and end togethertogether

Feature Crew continues until all work is Feature Crew continues until all work is “done” - feature complete, no bugs, tests in “done” - feature complete, no bugs, tests in place, etc.place, etc.

7

Key ConceptsKey Concepts

Quality GatesA set of quality metrics and deliverables that must accompany A set of quality metrics and deliverables that must accompany a feature when it is first submitted to the producta feature when it is first submitted to the product. .

FeatureFeatureA discrete piece of independently testable functionality that A discrete piece of independently testable functionality that integrates at the same time into the PU Branchintegrates at the same time into the PU Branch..

Feature BranchFeature BranchA light-weight source branch that is dedicated to the A light-weight source branch that is dedicated to the development of a single feature. Goal is to allow Feature Crew development of a single feature. Goal is to allow Feature Crew to fully stabilize in isolation before integratingto fully stabilize in isolation before integrating..

SCRUMA project management methodology that involves regular A project management methodology that involves regular check-in meetings to map progresscheck-in meetings to map progress..

8

What do Feature Crews What do Feature Crews offer?offer? Increased EfficiencyIncreased Efficiency

More parallel involvement of all disciplines (better Dev and Test More parallel involvement of all disciplines (better Dev and Test integration)integration)

Less context switching in the core phase of building a productLess context switching in the core phase of building a product Design issues and feature bugs are found earlier when they are Design issues and feature bugs are found earlier when they are

cheaper to fixcheaper to fix More effective and regular communication with cross discipline More effective and regular communication with cross discipline

counterpartscounterparts Fewer intra-version breaking changesFewer intra-version breaking changes

Enhanced Agility and PredictabilityEnhanced Agility and Predictability More honest schedulesMore honest schedules More consistent definition of “done” with quality gatesMore consistent definition of “done” with quality gates More confidence in release datesMore confidence in release dates Spend more time building and less time stabilizingSpend more time building and less time stabilizing

Improved QualityImproved Quality Higher quality new featuresHigher quality new features More stable builds to allow for better customer feedbackMore stable builds to allow for better customer feedback

9

Feature Crew Life CycleFeature Crew Life Cycle

10

De

sig

n P

ha

se

Te

sting

Co

din

g

Checkpoint 2Status Review

Feature complete,

ready to integrate

Dev Test PM UE

Discipline ActivitiesDiscipline ActivitiesIm

ple

me

nta

tion

Ph

as

eIn

teg

ratio

n P

ha

se

Feature design, dev doc,

prototyping

Coding,bug fixing

Unit tests, bug fixing

Finalizing quality gates,

testing, bug fixing

Feature design, test plans,

Test libraries,automation

Testing


testing

Feature design, functional spec

Threat model, bug triage,

Feature DCRs,Bug triage,

writing samples, quickstarts


testing

Feature design, content plans

Writing content

Finalizing quality gates

RI

Checkpoint 1Design Review

11

Responsibility AreasResponsibility Areas

PMPM Lead the feature crew Status tracking/reporting FC schedule

Functional specs Bug Triage

Write samples, quickstarts App BuildingReview UE docs Test features (scenario testing)

DevDev Design docs Code features, F1 Code unit tests

Fix bugs Test features Review UE docs

TestTest Test plan Automation tests

Test features Review UE docs

UEUE Content plan Write documentation Write samples, quickstarts

Review FC plans, designs Review UI strings Test features (scenario testing)

Architects Architects Review FC plans Review dev designs

Review code

Loc, UX, Loc, UX, Fundamentals, Fundamentals, PartnersPartners

Review FC plans, specsReview FC plans, specs Test features as appropriateTest features as appropriate

Leads, MgmtLeads, Mgmt Review FC plans, specs, statusReview FC plans, specs, status Manage change, peopleManage change, people

Fea

ture

Cre

w M

emb

ers

12

SCRUMSCRUM

Agile programming methodologyAgile programming methodology Small, dedicated groupsSmall, dedicated groups

Daily sync up meetings among crew membersDaily sync up meetings among crew members Anyone can attend, but only crew members can talkAnyone can attend, but only crew members can talk Led by a Scrum Master who tracks progress and drives Led by a Scrum Master who tracks progress and drives

problem solvingproblem solving

Ideally – short ~10 minute meeting where crew Ideally – short ~10 minute meeting where crew members answer only three questions:members answer only three questions: What did you do since we last met?What did you do since we last met? What are you doing next?What are you doing next? Are you blocked? Yes/NoAre you blocked? Yes/No

Problem solving and discussion happens after the Problem solving and discussion happens after the SCRUM meetingSCRUM meeting

13

Phase-by-Phase BreakdownPhase-by-Phase Breakdown

FC KickoffFC Kickoff Design PhaseDesign Phase Checkpoint #1 – Design ReviewCheckpoint #1 – Design Review Implementation PhaseImplementation Phase Checkpoint #2 – Progress ReviewCheckpoint #2 – Progress Review Integration PhaseIntegration Phase Feature Complete/Integration VerificationFeature Complete/Integration Verification

14

FC Design PhaseFC Design Phase

ActivitiesActivities – agreeing what to do, how to do it – agreeing what to do, how to do it Create functional specs, dev design plans, test plans, UE plansCreate functional specs, dev design plans, test plans, UE plans

All disciplines document plans from their perspective in parallelAll disciplines document plans from their perspective in parallel Enumerate Quality Gates, Dependencies, FundamentalsEnumerate Quality Gates, Dependencies, Fundamentals

Build into FC plans, decide when/who/howBuild into FC plans, decide when/who/how Identify risks, issues with anyIdentify risks, issues with any Schedule Quality Gate requirementsSchedule Quality Gate requirements

Updated costing of features, updates to schedule if neededUpdated costing of features, updates to schedule if needed Define Checkpoint process for FC (depends on feature complexity)Define Checkpoint process for FC (depends on feature complexity)

DeliverablesDeliverables functional spec, dev design, test plans, UE planfunctional spec, dev design, test plans, UE plan FC Schedule solidified, Work items identified, Feature list updated with appropriate FC Schedule solidified, Work items identified, Feature list updated with appropriate

infoinfo Feature Branch created, builds setupFeature Branch created, builds setup Tools in placeTools in place

How it worksHow it works FC works together to define designs, all participate in all aspects of design, FC works together to define designs, all participate in all aspects of design,

iterative brainstorming meetingsiterative brainstorming meetings Interactive, iterative discussions between PM, Dev, Test, UE Interactive, iterative discussions between PM, Dev, Test, UE Design phase completion triggers Checkpoint #1Design phase completion triggers Checkpoint #1

FC decides when/how to schedule checkpoint with mgmt teamFC decides when/how to schedule checkpoint with mgmt team

Design specs

DesignPhase

Functional specs

15

Testing - Design PhaseTesting - Design Phase

Characteristics of a good testCharacteristics of a good test It has a reasonable probability of catching It has a reasonable probability of catching

an erroran error It is not redundantIt is not redundant It’s the best of its breedIt’s the best of its breed It’s neither too simple nor too complexIt’s neither too simple nor too complex It makes program failures obviousIt makes program failures obvious

16

Testing - Design PhaseTesting - Design Phase

Techniques to come up with powerful Techniques to come up with powerful test casestest cases Equivalence class analysisEquivalence class analysis Boundary analysisBoundary analysis Testing state transitionsTesting state transitions Testing race conditions and other time Testing race conditions and other time

dependenciesdependencies Doing error guessingDoing error guessing

17

Test case design Test case design Equivalence class analysisEquivalence class analysis

If you expect the same result from two If you expect the same result from two tests, you consider them equivalenttests, you consider them equivalent

A group of tests forms an equivalence A group of tests forms an equivalence class whenclass when They involve the same input variablesThey involve the same input variables They result in similar operations in the They result in similar operations in the

programprogram They affect the same output variablesThey affect the same output variables None force the program to do error None force the program to do error

handling or all of them dohandling or all of them do

18

Test case design Test case design Boundary analysisBoundary analysis

The boundary values are the biggest, The boundary values are the biggest, smallest, soonest, shortest, loudest, smallest, soonest, shortest, loudest, fastest, ugliest members of the class, fastest, ugliest members of the class, i.e. the most extreme valuesi.e. the most extreme values

Many testers include a mid-range value Many testers include a mid-range value in their boundary tests. This is a good in their boundary tests. This is a good practice.practice.

19

Test case design Test case design Testing state transitions Testing state transitions

Every interactive program moves from one visible Every interactive program moves from one visible state to another. If you do something that changes state to another. If you do something that changes the range of available choices or makes the the range of available choices or makes the program display something different on the screen, program display something different on the screen, you’ve changed the program stateyou’ve changed the program state

Advice:Advice: Test all paths that you think people are particularly likely Test all paths that you think people are particularly likely

to followto follow If you suspect that choices at one menu level or data entry If you suspect that choices at one menu level or data entry

screen can affect the presentation of choices elsewhere, screen can affect the presentation of choices elsewhere, tests the effects of those choices or entriestests the effects of those choices or entries

Try a few random paths through the program. Try a few random paths through the program.

20

FC Implementation FC Implementation PhasePhase ActivitiesActivities – getting it done – getting it done

Coding, creating unit tests,Coding, creating unit tests, Test automationTest automation Testing and bug fixingTesting and bug fixing Quality work: API Reviews, Threat models, etc.Quality work: API Reviews, Threat models, etc. Fundamentals (perf, stress, etc.)Fundamentals (perf, stress, etc.)

DeliverablesDeliverables Daily SCRUM meetings (minimally quick hallway/email sync-ups among FC)Daily SCRUM meetings (minimally quick hallway/email sync-ups among FC) Feature code and unit tests complete Feature code and unit tests complete Automation tests in placeAutomation tests in place Documentation, including tech reviews of content, samples, app buildingDocumentation, including tech reviews of content, samples, app building All FC bugs addressedAll FC bugs addressed

How it worksHow it works Dev and Test work in tandem - iterative, parallel progress made throughoutDev and Test work in tandem - iterative, parallel progress made throughout PM facilitates, tracks/communicates status, drives bug triagePM facilitates, tracks/communicates status, drives bug triage All FC members help out as appropriate for load balancing – no discipline All FC members help out as appropriate for load balancing – no discipline

boundariesboundaries Checkpoint #2 scheduled by FC during Implementation phase when main Checkpoint #2 scheduled by FC during Implementation phase when main

coding is completecoding is complete

Coding

Testing

Bug fixing

ImplementationPhase

21

Testing – Testing – Implementation PhaseImplementation Phase

ActivitiesActivities – getting it done – getting it done Find bugsFind bugs File problem reportFile problem report,, Test automationTest automation Run automated tests and perform analysis on the Run automated tests and perform analysis on the

failuresfailures Quality work: API Reviews, Threat models, etc.Quality work: API Reviews, Threat models, etc. Fundamentals (perf, stress, etc.)Fundamentals (perf, stress, etc.)

DeliverablesDeliverables 70% automation tests in place70% automation tests in place 100% passing rate on the automated suite100% passing rate on the automated suite All problem report (bugs) are verified/closedAll problem report (bugs) are verified/closed

Coding

Testing

Bug fixing

ImplementationPhase

22

Fundamentals DetailsFundamentals Details

Perf and ScalabilityPerf and Scalability Stress and CapacityStress and Capacity Globalization/LocalizationGlobalization/Localization Security Security CompatibilityCompatibility Side-by-SideSide-by-Side AccessibilityAccessibility User ExperienceUser Experience HostingHosting 64-bit64-bit ServicingServicing

23

Problem reportProblem report

A good report isA good report is

You need to track the problem so you You need to track the problem so you must describe it in writing. Otherwise, must describe it in writing. Otherwise, some details (or the whole problem) will some details (or the whole problem) will be forgotten. be forgotten.

You also need a report for testing the fix You also need a report for testing the fix laterlater

24


A good report is A good report is

Track Problem Reports numerically.Track Problem Reports numerically.Assign a unique number to each report.Assign a unique number to each report. If you use a computerized database, the If you use a computerized database, the

report number will serve as a report number will serve as a key fieldkey field. . This is the one piece of information that This is the one piece of information that

always distinguishes one report from all always distinguishes one report from all the rest.the rest.

25



simple = not compoundsimple = not compound

Describe only one problem on one reportDescribe only one problem on one report

Why multiple bugs on a single report are a Why multiple bugs on a single report are a problem?problem?

The programer can fix only some of them and The programer can fix only some of them and pass the report back as fixed, even though pass the report back as fixed, even though some bugs have not been fixed. This wastes some bugs have not been fixed. This wastes time and leads to bad feelings. Remaining time and leads to bad feelings. Remaining problems often stay unfixed long time…problems often stay unfixed long time…

26



You must describe the program’s You must describe the program’s problematic behavior clearly. problematic behavior clearly.

Keep all unnecessary steps out of the Keep all unnecessary steps out of the steps required to reproduce the steps required to reproduce the problem.problem.

27



Think of the person reading it. Unless you Think of the person reading it. Unless you are reporting a disaster, the programmer are reporting a disaster, the programmer will toss an illegible report onto his/her will toss an illegible report onto his/her pile of things to look at next yearpile of things to look at next year

28



Nobody likes being told that what they did Nobody likes being told that what they did was wrong. As a tester, that’s what you was wrong. As a tester, that’s what you tell people every day. tell people every day.

Do not express personal judgments in Do not express personal judgments in your reportsyour reports

The programmer and the tester are a team The programmer and the tester are a team that makes the product betterthat makes the product better

29



How can I make the bug reproducible?How can I make the bug reproducible? You can describe how to get the program into a You can describe how to get the program into a

known state, and from there describe an exact series known state, and from there describe an exact series of stepsof steps

Find the simplest, shortest and most general Find the simplest, shortest and most general conditions that will trigger the bugconditions that will trigger the bug

Find related problemsFind related problems Find alternate paths to the same problemFind alternate paths to the same problem Find the most serious consequencesFind the most serious consequences Look for the critical stepLook for the critical step

30

Test AutomationTest Automation

Automation framework (in house)Automation framework (in house) Building Maintainable Testsuites Building Maintainable Testsuites Automation Results ReportingAutomation Results Reporting Automation auto-analysis toolsAutomation auto-analysis tools Code coverage Code coverage

Keeping test suites reliable Keeping test suites reliable Gauntlet and Code ReviewsGauntlet and Code Reviews

31

FC Integration PhaseFC Integration Phase

Activities Activities – making sure it works– making sure it works integrate from PU branchintegrate from PU branch Run all tests, verify feature works in integrated worldRun all tests, verify feature works in integrated world

Several functional runsSeveral functional runs StressStress PerformancePerformance Code Coverage RunCode Coverage Run

Fix and verify bugs (includes Product bugs)Fix and verify bugs (includes Product bugs) Quality Gates final verificationQuality Gates final verification integrate into PU branch, final verification in PU branchintegrate into PU branch, final verification in PU branch

DeliverablesDeliverables Feature List data updatedFeature List data updated Quality Gates complete, verifiedQuality Gates complete, verified Final integration testing completeFinal integration testing complete

How it worksHow it works FC tests feature with latest integrated code from PU BranchFC tests feature with latest integrated code from PU Branch All FC members verify feature meets requirements and works as expectedAll FC members verify feature meets requirements and works as expected Team members move onto next Feature CrewTeam members move onto next Feature Crew

32

Resources:Resources:

BooksBooks How to Break Software: A Practical Guide to How to Break Software: A Practical Guide to

Testing, Testing, by Whittaker, James A. (Author)by Whittaker, James A. (Author) Lessons Learned in Software Testing, Lessons Learned in Software Testing, by Cem by Cem

Kaner (Author) Kaner (Author) Effective Software Testing: 50 Specific Ways to Effective Software Testing: 50 Specific Ways to

Improve Your TestingImprove Your Testingby Elfriede Dustin (Author)(Author)

33

Comments, Q & AComments, Q & A

Documents

1 A Day in the Life of a Tester Irinel Crivat Irinel Crivat SDET Technical Lead ASP.NET