Impact of context switching and work interruptions on software development processes

International Conference on Software and System Processes

July 5-7 2017, Pierre and Marie Curie University, Paris, France

Alexey Tregubov*, Barry Boehm, Jo Ann Lane, Natalia Rodchenko

University of Southern California

* presenting author

Motivation Cross-project multitasking overhead is often observed in

• Matrix organizations - resources are shared between several projects for

better resource utilization.

• Multiple releases of a product – if a product is released more than one

time, resources are shared between maintenance of previous releases

and a new version (typical for small mid-size teams).

• System of System (SoS) environments – in SoSs, if a constituent system

is developed for several customers (e.g. different software

distributions/releases for each customer), resources are shared between

different work contexts.

2

Motivation Cross-project multitasking overhead is often not accounted when

projects are estimated. This may cause underestimation in project

planning.

3

Multitasking in work environment

4

Work productivity

Corporateculture

OfficeEnvironment

Personality

Personal process

Cross-project multitasking

Management Psychology

Work scheduling, technical processes

imp

acts

Work quality

Work interruptions

Work interruptions

5

Cross-project multitasking in software development

When developers work on several projects at the same time

(over a week or even a day), they switch between them

spending some time on context switching.

Not all context switching is bad, but we only focused on

excessive switching between different projects, which is

interrupting work and causing productivity decline.

Context switching between projects cost/time consists of

• physical time to switch– switch between repositories,

DBs, servers, etc. takes time

• cognitive context switching –getting into ‘flow mode’

takes time

6

Research questions and hypothesis

RQ1. What is the quantitative effect of cross-project multitasking

overhead on development effort and quality?

• H1.a The number of cross-project interruptions is (not) linearly proportional

to the number of projects.

• H1.b The G. Weinberg’s heuristic is (not) applicable for cross-project

multitasking overhead estimation in software development teams.

• H1.c The number of cross-project interruptions is (not) linearly proportional

to the number of reopened tasks (rework).

RQ2. How can COCOMO II model be improved to account for cross-

project multitasking overhead?

• H2. Using the multitasking effort multiplier (MEM) in the locally calibrated

COCOMO II model can improve prediction accuracies of the COCOMO II.

7

Gerald M. Weinberg’s heuristic

8

0

10

20

30

40

50

60

70

80

1 2 3 4 5

% o

f ef

fort

sp

en

t o

n m

ult

itas

kin

g

Number of projects at the same time

Weinberg's heuristic

Research methodology

9

COCOMO IIcalibration

Compute MEM for each

data point

Test hypothesisH2a,b

Source data: work logs of different project groups

R1R2R3R3

Project 1 schedule

R1R2R3R3

Project 1 schedule

R1R2R3R3

Project 1 schedule

R1R2R3R3

Project 1 schedule

R1R2R3R3

Project 1 schedule

R1R2R3R3

Project 1 schedule

R1R2R3R3

Project 1 schedule

R1R2R3R3

Project 1 schedule

R1R2R3R3

Project 1 schedule

R1R2R3R3

Project 1 schedule

Collected data (work logs and schedules)

Ind

ust

ry p

roje

cts

Stu

den

ts’ p

roje

cts

R1R2R3R3

Project 1 schedule

R1R2R3R3

Project 1 schedule

R1R2R3R3

Project 1 schedule

Extended data set from simulation

Sim

ula

tio

n

Test hypothesis H1a,b,c

Count cross-project interruptions for each data point

Compute MEM for each

data point

Count cross-project interruptions for each data point

Compute quality metric

Evaluate model prediction accuracies

RQ1: impact of multitasking on effort and quality

RQ2: COCOMO II calibrationlink between data source and data

transformation/manipulation

Data collection

Industry projects:

• Two groups of projects, one year each

• Work logs in JIRA – hours reported daily for each task in each project

• Schedules were updated every 2-3 days

• SLOC added and changed for each project

577 students projects

• 10 teams/projects, 5-11weeks development phase

• Work logs in JIRA – hours reported weekly for each task in each project

• Schedules were updated every 2 weeks

• SLOC added and changed for each project

10

Cross-project work interruptions (CSCI577 class projects)

11

R2 0.61110

S 2.418

p 0.0003

n 29

Cross-project work interruptions (industry)

12

R2 0.4563

S 2.198

p <0.005

n 154

Cross-project work interruptions (CSCI577 class projects)

13

R2 0.785

S 0.617

p 0.0001

n 21

Cross-project work interruptions (industry)

14

R2 0.48

S 379

p <0.005

n 142

Cross-project work interruptions (CSCI577 class projects)

15

Cross-project work interruptions (industry)

16

Multitasking effort multiplier (MEM)

𝑬𝒇𝒇𝒐𝒓𝒕 = 𝑨 × [𝑺𝒊𝒛𝒆]𝑩+ 𝑺𝑭𝒋𝟓𝒋=𝟏 × 𝑬𝑴𝒋𝝎

𝟏𝟕

𝒋=𝟏

×𝑴𝑬𝑴

𝑀𝐸𝑀 = 𝑓 𝐼, 𝑅, 𝐸 =𝐸

𝐸 − 𝐼𝑅=1

1 − 𝑅𝐼𝐸

=1

1 − 𝑅𝐼/𝑡𝐸/𝑡

Cross-project multitasking effort multiplier (CSCI577 class projects)

18

R2 0.11

S 0.2

p 0.008

n 29

Cross-project multitasking effort multiplier (industry)

19

R2 0.45

S 0.01

p <0.005

n 154

Comparison with Weinberg’s heuristic (CSCI577 class projects)

20

Comparison with Weinberg’s heuristic (industry)

21

Impact on quality (CSCI577 class projects)

22

R2 0.44

S 2.25

p 0.051

n 9

Impact on quality (industry)

23

R2 0.71

S 1.01

p 0.004

n 9

Results: RQ1 H1.a. The number of cross-project interruptions is (not) linearly proportional to number

of projects.

• Tested negative

• For a given group of industry projects and CSCI577 projects the number of cross-project interruptions is not linearly proportional to number of projects. The linear correlation is relatively weak (R^2 is between 0.46 and 0.61 in all tests).

H1.b. The G. Weinberg’s heuristic is (not) applicable for cross-project multitasking overhead estimation in software development teams.

• Tested negative

• For a given group of industry projects and CSCI577 projects cross-project multitasking overhead cannot be evaluated with the G.Weinberg’s heuristic because it overestimates effort for this type of work interruptions.

H1.c. The number of cross-project interruptions is (not) linearly proportional to the number of reopened tasks.

• Tested positive

• For a given group of industry projects cross-project interruptions are linearly correlated with the number of reopened JIRA tickets.

H2. The multitasking effort multiplier (MEM) can be automatically evaluated based on work log observations.

• Tested positive

• Work log analysis algorithm was implemented as part of the DES simulation framework and used to compute results presented above.

24

RQ2: Calibrated COCOMO II prediction accuracies

25

COCOMO II Local

calibration of

COCOMO II

Local calibration of

COCOMO II with MEM

PRED(.20) 32% 56% 63%

PRED(.25) 37% 58% 75%

PRED(.30) 38% 61% 81%

Results: RQ1

26

Amount of work Interruptions

Number of projects

Multitasking Overhead (Metric: MEM)

Quality(metric: reopened tasks

or grade deduction)

Effort

Reimmersion time

Eval

uat

ion

Evaluation

H1.a weakcorrelation

H1.c positive correlation

H1.b no correlation

Causal Graphical Model – results summary

Implied causality or functional relationship

Hypothetical association or causality

Observation

Observation & Evaluation

Observation Observation

ObservationEvaluation

Individual level Individual and team level

Individual and team level

Team level

Individual level

Individual level

Number of projects

Metric or measurment

H1.a – tested by hypothesis

Challenges and threats to validity

Reimmersion time does not necessarily capture the time needed to restore

work context.

• Used the average minimum estimation of the reimmersion time. In the worst case, the values

found in this study serve as an lower bound for multitasking overhead.

Self-tracking of the reimmersion time and hours devoted to projects may not be

accurate

Students may inaccurately report effort in Jira

• To address this threat, students effort reports in Jira were graded every week. Additionally,

weekly reminders were sent out about the reports.

Academic environment, where graduate students of CSCI577 class worked, is

different from developers in industry.

• To limit the impact of deadlines in other classes, students in the selected sample were

enrolled in the same classes (e.g. students who took 3 classes were all enrolled in software

engineering class, algorithms class and AI class).

27

Future work

The approach for counting work interruptions and evaluating their

impact on effort can be applied to other types multitasking. To do so

we need to develop methods for measuring the number of work

interruptions caused by certain type of multitasking.

The further accuracy evaluation of models with MEM (any

COCOMO family model with MEM) should be done on a bigger

sample of projects.

The work log analysis tools, we used in the research, can be

integrated with project tracking systems such as Atlassian JIRA to

provide real-time information about work interruptions and their

impact on productivity (e.g. calculate MEM).

Further research should be done on how to schedule and

organize work to reduce negative effects of work interruption

on productivity

28

Q&A, discussion

29

References

1. Meyer, D. E. & Kieras, D. E., 1997. A computational theory of executive cognitive processes and multiple-task performance: Part 2. Accounts of psychological refractory-period phenomena. Psychological Review, 104, 749-791.

2. Weinberg, G.M., 1993. Quality software management. New York.

3. DeMarco, T. and Lister, T., 2013. Peopleware: productive projects and teams. Addison-Wesley.

4. American Psychological Association, 2006. Multitasking: switching costs. URL: http://www. apa. org/research/action/multitask. aspx.

5. Dzubak, C.M., 2008. Multitasking: The good, the bad, and the unknown. The Journal of the Association for the Tutoring Profession, 1(2), pp.1-12.

6. Delbridge, K.A., 2000. Individual differences in multi-tasking ability: Exploring a nomological network. Unpublished Doctoral Dissertation, University of Michigan.

7. Tregubov, A. and Lane, J.A., 2015. Simulation of Kanban-based Scheduling for Systems of Systems: Initial Results. Procedia Computer Science, 44, pp.224-233.

8. Leonard-Barton, D. and Swap, W.C., 1999. When sparks fly: Igniting creativity in groups. Harvard Business Press.

9. Jett, Q.R. and George, J.M., 2003. Work interrupted: A closer look at the role of interruptions in organizational life. Academy of Management Review, 28(3), pp.494-507.

10. Tregubov, A. and Lane, J.A., 2015. Simulation of Kanban-based Scheduling for Systems of Systems: Initial Results. Procedia Computer Science, 44, pp.224-233.

30

References

11. Dingsøyr, T., Nerur, S., Balijepally, V. and Moe, N.B., 2012. A decade of agile methodologies: Towards explaining agile software development. Journal of Systems and Software, 85(6), pp.1213-1221.

12. Boehm, B. and Turner, R., 2003. Balancing Agility and Discipline: A Guide for the Perplexed, Portable Documents. Addison-Wesley Professional.

13. Larman, C. and Vodde, B., 2008. Scaling lean & agile development: thinking and organizational tools for large-scale Scrum. Pearson Education.

14. Poppendieck, M. and Poppendieck, T., 2007. Implementing lean software development: from concept to cash. Pearson Education.

15. NDIA-National Defense Industrial Association, 2010. Top Systems Engineering Issues In US Defense Industry. Systems Engineering Division Task Group Report, http://www. ndia. org/Divisions/Divisions/SystemsEngineering/Documents/Studies/Top% 20SE% 20Issues, 202010.

16. Turner, R., Shull, F., Boehm, B., Carrigy, A., Clarke, L., Componation, P., Dagli, C., Lane, J.A., Layman, L., Miller, A. and O'Brien, S., 2009. Evaluation of Systems Engineering Methods, Processes and Tools on Department of Defense and Intelligence Community Programs-Phase 2 (No. SERC-2009-TR-002). SYSTEMS ENGINEERING RESEARCH CENTER HOBOKEN NJ.

17. http://www.businessdictionary.com/definition/matrix-organization.html

18. https://hbr.org/1978/05/problems-of-matrix-organizations

19. http://www.joelonsoftware.com/articles/fog0000000022.html

20. Mark, G., Gonzalez, V.M. and Harris, J., 2005, April. No task left behind?: examining the nature of fragmented work. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 321-330). ACM.

31

References

Images courtesy: http://kursevi.edukacija.rs/wp-content/uploads/2015/07/multitasking-efikasnost.jpg

http://mercercognitivepsychology.pbworks.com/f/1385057522/Multitasking.jpg

http://www.projectengineer.net/wp-content/uploads/2012/12/life_cycle.png

32