Test-Driven Learning in Early Programming Courses

David JanzenCalifornia Polytechnic State University

San Luis Obispo, CA

Hossein SaiedianUniversity of Kansas

Lawrence, KS

SIGCSE’08

I Have a Dream

• That all students will some day… – learn to design and test as they learn to program

• “... we have as many testers as we have developers. And testers spend all their time testing, and developers spend half their time testing.”

Bill Gates in Information Week, May 2002

SIGCSE’08

How to Accomplish the Dream?

• Test-Driven Development

• TDD is a design (and testing) approach involving short, rapid iterations of

RefactorUnit Test Code

SIGCSE’08

Forces programmer to consider use of a method before implementation of the method

TDD in Actionclass RectangleTest { @Test public void testArea() { Rectangle r = new Rectangle(5,8); assertEquals(40, r.getArea()); }}

class Rectangle { private length, width; public Rectangle(int l,int w) { length = l; width = w; } public int getArea() { return length * width; }}

@Test public void testPerim(){ Rectangle r = new Rectangle(5,8); assertEquals(26,r.getPerim()); }}

public int getPerim() { return (length * 2) + (width * 2); }}

SIGCSE’08

“beyond the visionary phase and into the early mainstream” IEEE Software 2007

Adoption EvolutionObjects

Research LabsDahl & Nygaard

IndustrySmalltalk, C++

Grad & Advanced UG

First Year CS

NASA ProjectMercury

Grad & Advanced UG

First Year CS

TDD

IndustryAny SW Process

IndustryXP

Predicting the Future

SIGCSE’08

Questions We Want To Answer

1. Can we teach automated unit testing in first year programming courses without sacrificing other first year topics?

2. Is it better to teach test-first (TDD) or test-last programming to first year students?

3. What do students think about TDD?

SIGCSE’08

How Do We Teach TDD?

• Test-Driven Learning involves teaching testing by example

System.out.println(“Result: ” + myObject.calculateResult());

assertEquals(25, myObject.calculateResult());

XXXXXXXXXXXXX

SIGCSE’08

Experiment Design

• Proposed TDD/TDL integration into CS1 and CS2 courses at University of Kansas in 2006 as part of PhD research

• We don’t always get what we ask for– Given permission to teach two labs and assign

two projects • Weeks 6 and 7 in a 16-week CS1 course

• Weeks 1 and 3 in a 16-week CS2 course

SIGCSE’08

TF/TL Instruction & Survey

Project 1 Project 2

CS1

CS2Test-First

Test-Last

Test-Last Test-First

Test-LastTest-First

Survey

Experiment Design

40

66

40

66

6

30

Project 1 solution with tests provided

Project 1 solution with tests provided

CS1 StudentsAssigned by ID

CS2 StudentsSelf-selected

SIGCSE’08

Experiment Context

• CS1 and CS2 both used C++, vi/emacs, and g++

Project Topics 1st Project 2nd Project

CS1 Arrays, functions, text I/O

Classes, text I/O

CS2 Classes, array-based list, text I/O

Classes, pointer-based list, text I/O

SIGCSE’08

Goal: Keep Unit Testing Simple#include <cassert> class Exams { . . . };int main(){

run_tests(); } void run_tests() { { //test 1 Minimum of empty list is 0

Exams exam1; assert(exam1.getMin() == 0);

} //test 1 { //test 2

Exams exam1; exam1.addExam(90); assert(exam1.getMin() == 90);

} //test 2 } SIGCSE’08

CS1 Results: Grades & Effort

• Project grades were virtually the same

• Time spent on project was ~10% higher for test-first programmers

Time (min) 1st Project 2nd Project

TF mean 285.29 343.48

TL mean 260.20 308.40

% diff 10% 11%

p-value .60 .47

SIGCSE’08

CS1 Results: Unit Tests

• Test-first programmers wrote more tests on the first project, but fewer on second

• Remember that students switched approaches# asserts 1st Project 2nd Project

TF mean 5.85 1.89

TL mean 3.85 3.10

% diff 52% -39%

p-value .11 .11

SIGCSE’08

CS1 Results: Confidence

• No difference in confidence on first project

• On second project, test-first programmers were more confident of their solutions (P1 tests)2nd Project Quality Changes Reuse

TF mean 3.98 3.90 3.69

TL mean 3.25 3.06 2.88

% diff 22% 27% 28%

p-value .05 .02 .02

SIGCSE’08

CS1 Results: Student OpinionsCS1 Programmer Opinions

0% 20% 40% 60% 80% 100%

FewerDefects

Simpler

Correctness

ThoroughTesting

BestApproach

Choice

Ch

arac

teri

stic

% Choosing

Test-First Test-Last

SIGCSE’08

CS2 Results: Grades

• Project grades were higher for test-first

• No significant difference in previous experience or gpa

Score 1st Project 2nd Project

TF mean 88.83 90.17

TL mean 79.47 72.83

% diff 12% 24%

p-value .09 .04

SIGCSE’08

CS2 Results: Effort

• Time spent on project was lower for test-first programmers

Score 1st Project 2nd Project

TF mean 12.42 16.00

TL mean 14.70 18.06

% diff -16% -11%

p-value .30 .42

SIGCSE’08

CS2 Results: Unit Tests

• Test-first programmers wrote more tests

• Not significant, high standard deviation

• Students not graded on tests# asserts 1st Project 2nd Project

TF mean 53.67 47.67

TL mean 13.43 25.03

% diff 300% 90%

p-value .08 .39

SIGCSE’08

CS2 Results: Confidence

• No significant difference in confidence on either project

Both Projects Quality Changes Reuse

TF mean 3.83 4.00 3.67

TL mean 3.21 3.89 3.71

% diff 16% 3% -1%

p-value .14 .87 .74

SIGCSE’08

CS2 Results: Student OpinionsCS2 Programmer Opinions

0% 20% 40% 60% 80% 100%

FewerDefects

Simpler

Correctness

ThoroughTesting

BestApproach

Choice

Ch

arac

teri

stic

% Choosing

Test-First Test-Last

SIGCSE’08

Reluctance to adopt test-firstdespite perceived benefits

11% vs 63% would choose test-first

SIGCSE’08

Validity Concerns

• Study too small to draw any conclusions• Lack of randomization in CS2• Confounding factors:

– New concepts introduced with each project

– Solutions with tests provided after 1st project

– Simple assert-based unit tests, no framework

– C++

• However, many of the results line up with results from industry and more advanced courses

SIGCSE’08

Conclusions

• Unit testing can be integrated into CS1/2 courses without taking time away from other topics

• Test-first programmers tend to write more tests• Test-first programmers may do as well (CS1) or

better (CS2) on projects• Test-first programmers may spend more time (CS1)

or perhaps less time (CS2) on projects• Early programmers are reluctant to use test-first,

even after positive experiences with it

SIGCSE’08

Future Work

• Currently conducting a study in CS1.5– Java and JUnit – TDD from beginning– TDD throughout 10-week quarter

• Early Results– TDD takes no extra instruction time– Existing course materials can be rewritten w/TDD– TDD could influence course refactoring

• e.g. delay coverage of I/O

SIGCSE’08

Questions

• Resources at http://www.simexusa.com/tdl/• If you are an early adopter of TDD in CS1/2,

I’d like to know, email me at djanzen (at) calpoly.edu

• Acknowledgements– SIGCSE Special Projects Grant – seed funding– Lockheed Martin – current (future work) study

SIGCSE’08