dB-SERC Mentor-Mentee Evidence-Based Teaching...

dB-SERCMentor-MenteeEvidence-BasedTeachingAward

AdamJ.LeeWilliamC.GarrisonIII

Designingandbuildingsecuresystemsishard!

2

“TheblackartofprogrammingSatan’scomputer”[1]

Longstandingdesigns andimplementations havebeenproveninsecure:

Needham-SchroederMan-in-the-middlediscovered

after17yearsinuse

OpenSSLHeartbleed vulnerabilitydiscovered

2yearsafterintroduced

Formalverificationisverydifficult,evenforexperiencedsoftwareengineers!

CS1653teachessecurityengineeringwithafocusonasemester-longgroupproject

3

CS1653:AppliedCryptographyandNetworkSecurity

Lecturespresentalgorithmsandprotocols,studentsapplytheseinaninterleavedsemesterproject

Inthisproject,studentsmust:• Workingroupsforthefullsemester• Proposetheirownsolutionstoadversarialtasks• Develop,maintain,andextendanon-trivialcodebase(~5klines)

Requiresbothdesign andcoding!

AsummaryoftheCS1653semesterproject

4

Studentsdevelopasecuredistributedfile-sharingsystem

Fivephases,eachconsideringadditionalsecuritythreats

Studentsmeetwithinstructortoproposesolutions,demowithTAaftersubmission

!"#$%&'

()*+,'-$).$)' /#"$'-$).$)'0'

/#"$'-$).$)'!'

!"

!"

!"

12%23$'/#"$4'

5+&6$%782&$'2%9'

12%23$'()*+,4'

Eventhebeststudentsrunintoproblemswiththisproject…

5

Themostcommonproblems:• Unevendistributionofwork• Lackofcommunicationamonggroupmembers• Procrastination,submittinglast-minute• Jugglingdesignandcode• Rushingthroughcode• Combiningcodewrittenbymultiplegroupmembers• Designandcodenotmatching,evolvingout-of-sync

Canusingaversioncontrolsystemhelpmitigatetheseissues?

Whydevelopcodeusingaversioncontrolsystem?

6

InaVCS,anychangetoacodebaseiscalledacommit

TheVCSmaintainsahistory of

previouscommitswithdescriptions

u1

u2u1

u2

mrgAcommitisrelative,toeasethemerging

ofworkfrommultipleusers

Commitlogsaretimeseriesdescribingdevelopmentatafinegranularity,andhavebeenusedforavarietyofexperiments:

• AdoptionofnewAPIsdoesnotkeeppacewiththeirdevelopment[2]• Programminglanguagedesignhasamodesteffectoncodequality[3]• Genderandtenurediversityarepositivepredictorsofproductivity[4]• Functionswithassertshavesignificantlyfewerdefects[5]• AskingquestionsonQ&Asitescatalyzesdevelopment(andviceversa)[6]

HowcanusingaVCSimprovetheCS1653projectforourstudents?

7

Stayorganized:studentsreviewtheirchangeswhencommitting

Commitlogsimprovecommunication:seewhatyourgroupmates havecompleted

Muchsimplermerging whenworkingsimultaneously:nomoreemailingcodeandmanuallycombining!

Continuoussubmission:workuntilthedeadline,committingasyougo

Whataboutusinganalytics?

VCSanalyticstoimprovethecourseproject

8

High-levelgoal:improvegroupperformance…how?

Duringthesemester• Useanalyticstodetectproblemsingroups• Allowtheinstructortointerveneasneeded

Betweensemesters• Useanalyticstodiscoverwhatmakessomegroupsmoresuccessful• Adjustcoursetoencouragebehaviorseen instronggroups

WecollecteddatafromSpring2015offeringofthecourse,andappliedthelessonslearnedtoSpring2016

• 2154:33students,14groups,4projectphases• 2164:33students,12groups,5projectphases

Whatanalyticscorrelatedwithgroupperformance?

9

Distributionoflabor

Per-weekworkcompleted

Others:Goodcommitmessages,workingondocumentationearly

ForSpring2016,ourchangeswereprimarilyinterventionary

10

Duringofficehoursmeetings,checkedlogsforindicators:• Earlyworkondocumentation• Balanceofcommitspermember• Descriptivecommitmessages

Whenconfrontedwithconcerns,studentshadarangeofresponses:

• Expressedregret,admittedtheyneededtoimprove• Defendedtheirgroupmates• Explainedspecialcircumstances

• “Johncouldn't commit,sochangeswentthroughme”• “Wemetatmyplaceandpair-programmed”

Inbetweenphases,offeredhelpmanaginggroupwork,etc.

Overall,studentsseemedto“bounceback”moresuccessfully

11

40

50

60

70

80

90

100

30 40 50 60 70 80

Gradeonnextprojectphaseafteragradeof80or below

Term2154 Term21640

10

20

30

40

50

60

70

80

90

100

MEAN MEDIAN

Gradeonnextprojectphaseafteragradeof80orbelow

Term2154 Term2164

Overall,studentsseemedto“bounceback”moresuccessfully

12

0

10

20

30

MEAN MEDIAN

Gradeincreaseonnextprojectphaseafteragradeof80orbelow

Term2154 Term2164

-10

0

10

20

30

40

50

30 40 50 60 70 80

Gradeincreaseonnextprojectphaseafteragradeof80or below

Term2154 Term2164

Projectgradesoverthecourseoftheterm

13

35

45

55

65

75

85

95

105

P1 P2 P3 P4 P5

2154GradesbyPhase

35

45

55

65

75

85

95

105

P1 P2 P3 P4 P5

2164GradesbyPhase

Futureimprovementstobemadeusingthesetechniques

14

Phase1seemedtobeharderduetoversioncontrol• HoldoffuntilPhase2?• ShorterassignmenttogetusedtoVC?• Grademoreleniently?

Phase3isstillthehardestoverall• Closertrackingofrepositories,evenoutsideofmeetings?• Givemoreguidance,leavelaterphasesmoreopen-ended?• Shorten,movesomematerialtoPhase4?

Outliergroupsneverrecover• Offermorepointedguidance?• Detectthistypeofgroup,breakupearly?

Questions?

15

References:1. RossJ.Anderson andRogerM.Needham,“Programming Satan’sComputer,” In

ComputerScienceToday:RecentTrendsandDevelopments,1995.2. TylerMcDonnell, Baishakhi Ray,Miryung Kim:AnEmpiricalStudyofAPIStability

andAdoption intheAndroid Ecosystem.ICSM2013:70-793. Baishakhi Ray,DarylPosnett,VladimirFilkov,Premkumar T.Devanbu:Alarge

scalestudyofprogramming languagesandcodequalityingithub.SIGSOFTFSE2014:155-165

4. Bogdan Vasilescu,DarylPosnett,Baishakhi Ray,MarkG.J.vandenBrand,AlexanderSerebrenik, Premkumar T.Devanbu,VladimirFilkov:GenderandTenureDiversityinGitHub Teams.CHI2015:3789-3798

5. Casalnuovo Casey,Devanbu Prem,OliveiraAbilio,Filkov Vladimir,andBaishakhiRay:AssertUseinGitHub Projects.ICSE2015

6. Bogdan Vasilescu,VladimirFilkov,AlexanderSerebrenik:StackOverflow andGitHub:AssociationsbetweenSoftwareDevelopment andCrowdsourcedKnowledge.SocialCom 2013:188-195

Thank you!