Stuck and Frustrated or In Flow and Happy: Sensing Developers’ Emotions and Progress

Sebastian Müller and Thomas Fritz

Stuck and Frustrated or in Flow and Happy: Sensing Developers’ Emotions and Progress

ICSE 2015, Florence, Italy

Benefits of Knowing Emotions and Progress

4

Support developers when stuck and making no progress

Avoid interruptions when in flow and making a lot of progress

Biometric Sensing ofEmotions & Progress

5

positive/negative emotions (e.g. Lin et al)

happiness, sadness (e.g. Li et al)

positive/negative emotions (e.g. Haag et al)

frustration (e.g. Scheirer et al)

positive/negative emotions (e.g. Muldner et al)

EEG(Electroencephalography)

frustration (e.g. Kapoor et al)

anxiety (e.g. Doehring)

Pupil sizes

Eye blinks

positive/negative emotions (e.g. Haag et al)

boredom, anxiety (e.g. Chanel et al)

positive/negative emotions (e.g. Drachen et al)

happiness (e.g. Steptoe et al)

EDA(Electro-dermal activity)

Skin temperature

HR(V)(Heart rate (variability))

Research Questions

6

Correlation of emotions and progress?

Aspects and practices that affect developers’ emotions and progress?

Biometric sensors to determine developers’ emotions and progress?

Study Participants & Tasks

17 participants: 6 professionals and 11 PhD students• on average 7.1 years of prof. development experience

• only minor differences between professionals and students

Two change tasks• Task 1: small program to interact with StackExchange API

• Task 2: new feature in JHotDraw

7

Lab Study Setup

8

Lab Study Setup

8

Neurosky headband

Empatica wristband

Eyetribe eye tracker

Study Method

9

Fish tank video as a baseline for biometric measurements

Study Method

9

Fish tank video as a baseline for biometric measurements

Work on two change tasks, each for 30 minutes

Study Method

9

Fish tank video as a baseline for biometric measurements

Work on two change tasks, each for 30 minutes

• progress• valence: ‘’very unpleasant’’ to ‘’very pleasant’’

arousal: ‘’very calm’’ to ‘’very excited’’• reasons for the current state of emotions / progress & aspects

that would help to improve

Assessments every 5 minutes or at signs of strong negative/positive emotions

Study Method

9

Fish tank video as a baseline for biometric measurements

Work on two change tasks, each for 30 minutes

After working on both change tasks: set of emotion inducing pictures

• progress• valence: ‘’very unpleasant’’ to ‘’very pleasant’’

arousal: ‘’very calm’’ to ‘’very excited’’• reasons for the current state of emotions / progress & aspects

that would help to improve

Assessments every 5 minutes or at signs of strong negative/positive emotions

Data Collection

10

biometric data from 3 sensors

213 data points• emotion and progress ratings

• answers to the questions we asked

2 emotion ratings for the set of emotion inducing pictures

Research Questions

11

Correlation of emotions and progress?

Aspects and practices that affect developers’ emotions and progress

Biometric sensors to determine developers’ emotions and progress

Correlation Emotions & Progress

12

Linear Mixed Model

Dependent variable: Self-reported progressFixed effects: Valence, arousalRandom effects: Participant, task, measurement time

Correlation Emotions & Progress

13

Linear Mixed Model

Dependent variable: Self-reported progressFixed effects: Valence, arousalRandom effects: Participant, task, measurement time

Random effects: Measurement time no effect, task medium effect and participant highest effect

Effect Estimate Deviance explained

Valence 0.66 28.03

Arousal 0.10 1.09

Correlation Emotions & Progress

14

Correlation Emotions & Progress

14

Subject 1

Significant correlation between progress and valence

Correlation Emotions & Progress

14

Subject 6Subject 1

Significant correlation between progress and valence

No significant correlation between progress and valence

Correlation Emotions & Progress

14

Subject 6Subject 1

Significant correlation between progress and valence

No significant correlation between progress and valence

significant correlations for 12 subjects, but not for the other 5 subjects

Research Questions

15

Correlation of emotions and progress?

Aspects and practices that affect developers’ emotions and progress?

Biometric sensors to determine developers’ emotions and progress

Top 3 Reasons for a Change inEmotions / Progress

16

Increase in emotions/progress # Cases # Subjects

localize relevant code 21 (11.3%) 14 (82.4%)

(better) understand parts of the code 18 (9.7%) 13 (76.5%)

next steps are clear 12 (6.5%) 9 (52.9%)

Top 3 Reasons for a Change inEmotions / Progress

16

Increase in emotions/progress # Cases # Subjects

localize relevant code 21 (11.3%) 14 (82.4%)

(better) understand parts of the code 18 (9.7%) 13 (76.5%)

next steps are clear 12 (6.5%) 9 (52.9%)

”I’ve found a starting point almost immediately. I have a good feeling that I’ll make significant progress very soon.” (S1)

Top 3 Reasons for a Change inEmotions / Progress

17

Decrease in emotions/progress # Cases # Subjects

difficulty in understanding how parts of the code / API works 33 (17.7%) 12 (70.6%)

difficulty in localizing relevant code 15 (8.1%) 8 (47.1%)

not being sure about next steps 9 (4.8%) 9 (52.9%)

Top 3 Reasons for a Change inEmotions / Progress

17

Decrease in emotions/progress # Cases # Subjects

difficulty in understanding how parts of the code / API works 33 (17.7%) 12 (70.6%)

difficulty in localizing relevant code 15 (8.1%) 8 (47.1%)

not being sure about next steps 9 (4.8%) 9 (52.9%)

”It’s going too slow. I think it’s very cumbersome when so much time is needed to understand the

project and find a starting point.” (S9)

Emotions and Progressare Often Intertwined

18

”I finally figured out how to do it. I’m really happy and I’m not feeling completely stuck

anymore.” (S6)

”I can’t make any progress. That’s annoying.” (S13)

Research Questions

19

Correlation of emotions and progress?

Aspects and practices that affect developers’ emotions and progress

Biometric sensors to determine developers’ emotions and progress?

Electro-dermal Activity (EDA)and Valence

201 2 3 4

Machine Learning Approach

21

Data Cleaning and Feature Extraction• Noise cleaning• Feature extraction following established methods

{Min, Max}Attention; {Min, Max}Meditation, …∆(α); ∆(β); ∆(γ); ∆(δ); ∆(θ), …

{Min, Max}PeakAmpl; ∆NumPhasicPeaks/Min, …

MinPupilSize; MaxPupilSize, …

EEG(Electroencephalography)

Pupil sizes

Eye blinks

EDA(Electro-dermal activity)

Skin temperatureHR(V)(Heart rate (variability))

MeanTemp; ∆MeanTemp, …

∆MeanHR; ∆VarianceHR, …

∆Eyeblinks/Min

Machine Learning Approach

22

Data Cleaning and Feature Extraction• Noise cleaning• Feature extraction following established methods

Data Labeling• binary split of emotions (positive/negative) and progress (low/high)

Machine Learning Approach

23

Data Cleaning and Feature Extraction• Noise cleaning• Feature extraction following established methods

Data Labeling• binary split of emotions (positive/negative) and progress (low/high)

Machine Learning• Leave-one-out approach• Classifier: decision tree S01

T1S17 T1

S01T2

S17T2

T1

...S01 S17

Test data Training data

......

T2

Machine Learning Results

24

Prediction Correct Impr. Precision Recall Features

Emotion 71.36% 18.76% 64.32% 82.03% α, β/θ, MinPupilSize, ∆MeanHR

Progress 67.70% 32.93% 67.85% 68.29%

α, β/θ, ∆MeanTempPeakAmpl,

MaxPupilSize, ∆MeanPupilSize,

∆MeanSCL

Contributions and Outlook

25

Study on developers’ emotions and progress with 17 participants• Emotions and perceived progress are strongly correlated• Biometric sensors can be used to predict emotions and progress

Future challenges• Individual differences• Privacy concerns

New opportunities for developer support• Avoid interruptions when in flow and making a lot of progress• Support when stuck and making no progress

Individual Machine Learning Results

26

Accu

racy

(%)

0

25

50

75

100

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17

Emotions Progress

Individual Machine Learning Results

27

Accu

racy

(%)

0

25

50

75

100

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17

Emotions Progress

Individual Machine Learning Results

28

Accu

racy

(%)

0

25

50

75

100

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17

Emotions Progress

Individual Machine Learning Results

29

Accu

racy

(%)

0

25

50

75

100

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17

Emotions Progress

Biometric Features

3030

EEG: {Min, Max}Attention; {Min, Max}Meditation; ∆{Mean, Stdev}Attention; ∆{Mean, Stdev}Meditation;∆(α); ∆(β); ∆(γ); ∆(δ); ∆(θ); ∆(α/β); ∆(α/γ); ∆(α/δ); ∆(α/θ); ∆(β/α); ∆(β/γ); ∆(β/δ); ∆(β/θ); ∆(γ/α); ∆(γ/β); ∆(γ/δ); ∆(γ/θ); ∆(δ/α); ∆(δ/β); ∆(δ/γ); ∆(δ/θ); ∆(θ/α); ∆(θ/β); ∆(θ/γ); ∆(θ/δ); ∆(θ/(α+β)); ∆(β/(α+θ))

EDA: {Min, Max}PeakAmpl; ∆NumPhasicPeaks/Min; ∆MeanPhasicPeakAmpl; ∆SumPhasicPeakAmpl/Min∆MeanSCL; ∆AUCPhasicSkin temperature: MeanTemp; ∆MeanTemp; {Max, Mean}TempPeakAmpl; SumTempPeakAmpl/MinHR(V): ∆MeanHR; ∆VarianceHR; SDNN; RMSSD; NN20; NN50; pNN20; pNN50

Pupil size: MinPupilSize; MaxPupilSize; ∆{Mean, Stdev}PupilSize; ∆NumPupilSizeOutliersEye blinks: ∆Eyeblinks/Min