Stuck and Frustrated or In Flow and Happy: Sensing Developers’ Emotions and Progress

Sebastian Müller and Thomas Fritz

Stuck and Frustrated or in Flow and Happy: Sensing Developers’ Emotions and Progress

ICSE 2015, Florence, Italy

Benefits of Knowing Emotions and Progress

4

Support developers when stuck and making no progress

Avoid interruptions when in flow and making a lot of progress

Biometric Sensing ofEmotions & Progress

5

positive/negative emotions (e.g. Lin et al)

happiness, sadness (e.g. Li et al)

positive/negative emotions (e.g. Haag et al)

frustration (e.g. Scheirer et al)

positive/negative emotions (e.g. Muldner et al)

EEG(Electroencephalography)

frustration (e.g. Kapoor et al)

anxiety (e.g. Doehring)

Pupil sizes

Eye blinks

positive/negative emotions (e.g. Haag et al)

boredom, anxiety (e.g. Chanel et al)

positive/negative emotions (e.g. Drachen et al)

happiness (e.g. Steptoe et al)

EDA(Electro-dermal activity)

Skin temperature

HR(V)(Heart rate (variability))

Research Questions

6

Correlation of emotions and progress?

Aspects and practices that affect developers’ emotions and progress?

Biometric sensors to determine developers’ emotions and progress?

Study Participants & Tasks

17 participants: 6 professionals and 11 PhD students• on average 7.1 years of prof. development experience

• only minor differences between professionals and students

Two change tasks• Task 1: small program to interact with StackExchange API

• Task 2: new feature in JHotDraw

7

Lab Study Setup

8

Lab Study Setup

8

Neurosky headband

Empatica wristband

Eyetribe eye tracker

Study Method

9

Fish tank video as a baseline for biometric measurements

Study Method

9


Work on two change tasks, each for 30 minutes

Study Method

9



• progress• valence: ‘’very unpleasant’’ to ‘’very pleasant’’

arousal: ‘’very calm’’ to ‘’very excited’’• reasons for the current state of emotions / progress & aspects

that would help to improve

Assessments every 5 minutes or at signs of strong negative/positive emotions

Study Method

9



After working on both change tasks: set of emotion inducing pictures

• progress• valence: ‘’very unpleasant’’ to ‘’very pleasant’’

arousal: ‘’very calm’’ to ‘’very excited’’• reasons for the current state of emotions / progress & aspects

that would help to improve

Assessments every 5 minutes or at signs of strong negative/positive emotions

Data Collection

10

biometric data from 3 sensors

213 data points• emotion and progress ratings

• answers to the questions we asked

2 emotion ratings for the set of emotion inducing pictures

Research Questions

11


Aspects and practices that affect developers’ emotions and progress

Biometric sensors to determine developers’ emotions and progress

Correlation Emotions & Progress

12

Linear Mixed Model

Dependent variable: Self-reported progressFixed effects: Valence, arousalRandom effects: Participant, task, measurement time


13

Linear Mixed Model

Dependent variable: Self-reported progressFixed effects: Valence, arousalRandom effects: Participant, task, measurement time

Random effects: Measurement time no effect, task medium effect and participant highest effect

Effect Estimate Deviance explained

Valence 0.66 28.03

Arousal 0.10 1.09


14


14

Subject 1

Significant correlation between progress and valence


14

Subject 6Subject 1


No significant correlation between progress and valence


14

Subject 6Subject 1


No significant correlation between progress and valence

significant correlations for 12 subjects, but not for the other 5 subjects

Research Questions

15


Aspects and practices that affect developers’ emotions and progress?

Biometric sensors to determine developers’ emotions and progress

Top 3 Reasons for a Change inEmotions / Progress

16

Increase in emotions/progress # Cases # Subjects

localize relevant code 21 (11.3%) 14 (82.4%)

(better) understand parts of the code 18 (9.7%) 13 (76.5%)

next steps are clear 12 (6.5%) 9 (52.9%)


16

Increase in emotions/progress # Cases # Subjects

localize relevant code 21 (11.3%) 14 (82.4%)

(better) understand parts of the code 18 (9.7%) 13 (76.5%)

next steps are clear 12 (6.5%) 9 (52.9%)

”I’ve found a starting point almost immediately. I have a good feeling that I’ll make significant progress very soon.” (S1)


17

Decrease in emotions/progress # Cases # Subjects

difficulty in understanding how parts of the code / API works 33 (17.7%) 12 (70.6%)

difficulty in localizing relevant code 15 (8.1%) 8 (47.1%)

not being sure about next steps 9 (4.8%) 9 (52.9%)


17

Decrease in emotions/progress # Cases # Subjects

difficulty in understanding how parts of the code / API works 33 (17.7%) 12 (70.6%)

difficulty in localizing relevant code 15 (8.1%) 8 (47.1%)

not being sure about next steps 9 (4.8%) 9 (52.9%)

”It’s going too slow. I think it’s very cumbersome when so much time is needed to understand the

project and find a starting point.” (S9)

Emotions and Progressare Often Intertwined

18

”I finally figured out how to do it. I’m really happy and I’m not feeling completely stuck

anymore.” (S6)

”I can’t make any progress. That’s annoying.” (S13)

Research Questions

19


Aspects and practices that affect developers’ emotions and progress

Biometric sensors to determine developers’ emotions and progress?

Electro-dermal Activity (EDA)and Valence

201 2 3 4

Machine Learning Approach

21

Data Cleaning and Feature Extraction• Noise cleaning• Feature extraction following established methods

{Min, Max}Attention; {Min, Max}Meditation, …∆(α); ∆(β); ∆(γ); ∆(δ); ∆(θ), …

{Min, Max}PeakAmpl; ∆NumPhasicPeaks/Min, …

MinPupilSize; MaxPupilSize, …

EEG(Electroencephalography)

Pupil sizes

Eye blinks

EDA(Electro-dermal activity)

Skin temperatureHR(V)(Heart rate (variability))

MeanTemp; ∆MeanTemp, …

∆MeanHR; ∆VarianceHR, …

∆Eyeblinks/Min


22


Data Labeling• binary split of emotions (positive/negative) and progress (low/high)


23


Data Labeling• binary split of emotions (positive/negative) and progress (low/high)

Machine Learning• Leave-one-out approach• Classifier: decision tree S01

T1S17 T1

S01T2

S17T2

T1

...S01 S17

Test data Training data

......

T2

Machine Learning Results

24

Prediction Correct Impr. Precision Recall Features

Emotion 71.36% 18.76% 64.32% 82.03% α, β/θ, MinPupilSize, ∆MeanHR

Progress 67.70% 32.93% 67.85% 68.29%

α, β/θ, ∆MeanTempPeakAmpl,

MaxPupilSize, ∆MeanPupilSize,

∆MeanSCL

Contributions and Outlook

25

Study on developers’ emotions and progress with 17 participants• Emotions and perceived progress are strongly correlated• Biometric sensors can be used to predict emotions and progress

Future challenges• Individual differences• Privacy concerns

New opportunities for developer support• Avoid interruptions when in flow and making a lot of progress• Support when stuck and making no progress

Individual Machine Learning Results

26

Accu

racy

(%)

0

25

50

75

100

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17

Emotions Progress


27

Accu

racy

(%)

0

25

50

75

100


Emotions Progress


28

Accu

racy

(%)

0

25

50

75

100


Emotions Progress


29

Accu

racy

(%)

0

25

50

75

100


Emotions Progress

Biometric Features

3030

EEG: {Min, Max}Attention; {Min, Max}Meditation; ∆{Mean, Stdev}Attention; ∆{Mean, Stdev}Meditation;∆(α); ∆(β); ∆(γ); ∆(δ); ∆(θ); ∆(α/β); ∆(α/γ); ∆(α/δ); ∆(α/θ); ∆(β/α); ∆(β/γ); ∆(β/δ); ∆(β/θ); ∆(γ/α); ∆(γ/β); ∆(γ/δ); ∆(γ/θ); ∆(δ/α); ∆(δ/β); ∆(δ/γ); ∆(δ/θ); ∆(θ/α); ∆(θ/β); ∆(θ/γ); ∆(θ/δ); ∆(θ/(α+β)); ∆(β/(α+θ))

EDA: {Min, Max}PeakAmpl; ∆NumPhasicPeaks/Min; ∆MeanPhasicPeakAmpl; ∆SumPhasicPeakAmpl/Min∆MeanSCL; ∆AUCPhasicSkin temperature: MeanTemp; ∆MeanTemp; {Max, Mean}TempPeakAmpl; SumTempPeakAmpl/MinHR(V): ∆MeanHR; ∆VarianceHR; SDNN; RMSSD; NN20; NN50; pNN20; pNN50

Pupil size: MinPupilSize; MaxPupilSize; ∆{Mean, Stdev}PupilSize; ∆NumPupilSizeOutliersEye blinks: ∆Eyeblinks/Min

Science

Stuck and Frustrated or In Flow and Happy: Sensing Developers’ Emotions and Progress