On-line auto-graded assignments in economics: A test of their

effectivenessBrendan Kennelly1, John Considine2 and

Darragh Flannery1

1Department of Economics, NUI Galway2Department of Economics, UCC

Structure of Presentation

• Background

• Experiment

• Descriptive statistics

• Econometric analysis

• Discussion

Background: What is Aplia?

• Designed by Paul Romer in 2000

• Course management tool

• Facilitates learning and assessment– Using diagrams– Practice Assignment– Instant Feedback

Income doubles for this consumer. Place new budget constraint on diagram.

Background: Previous Work

• Technology– Teaching/Learning– Cost to Instructor

• Aplia– Collins et al (2008)– O’Dea and Ring (2008)

• Aplia in Nat. Univ. of Ireland, Galway– Kennelly and Duffy (DEE 2007)

• All Assignments on Aplia• Evaluated By Student Survey

A New ‘Experiment’

Brendan Kennelly and David Duffy. 2007. Poster Presentation to Developments in Economic Education Conference. Evaluating the Use of Aplia as a Substitute to Alternative First Year Tutorials Using Student Survey

Darragh Flannery (econometrics) and John Considine (student opinion) replace David Duffy

Aplia being used as a complement to tutorials for Second Year students

Evaluate by Student Survey and Focus Groups

Evaluate by examining grade outcomes

A New ‘Experiment’

• 390 students took managerial economics• Analysis mainly about 254 Commerce

Students• 10 assignments; all did 1 and 2 on Aplia• Thereafter ¼ did a paper assignment

instead of Aplia• Exam in 8 sections corresponding to the

last 8 assignments• Students had to do 6 sections in the exam

Assignment grade based on best 8 out of 10

Examination grade based on 6 question out of 8

Variable Mean Standard Deviation

Minimum Maximum Scale

Overall Exam Score 116.8 46.27 24.75 254.75 0-300

Assignment 1 30.58 9.45 2.27 47.72 0-50

Assignment 2 26.59 10.46 3.12 50 0-50

Assignment 3 33.31 8.39 4.54 50 0-50

Assignment 4 32.52 8.40 7.17 50 0-50

Assignment 5 29.85 12.09 1.19 50 0-50

Assignment 6 30.42 10.60 4.16 49.30 0-50

Assignment 7 30.12 15.19 2.17 50 0-50

Assignment 8 34.25 10.42 4.41 50 0-50

Overall Assignment Mark 61.92 18.48 6 97 0-100

CAO points 438.85 37.45 390 590 390-600

1st Year Economics Result 57.83 10.76 30 88 0-100

Overall Course Total 178.73 58.91 43.5 348.75 0-400

Summary Statistics

Descriptive Statistics

• Table 2: completion rate was lower for the written assignments than for the Aplia ones.

• Table 3: no consistent pattern in the marks across assignment format (Aplia or written)

• Table 4: no consistent pattern in the average mark per examination section by each group

• Table 5: average mark varied considerably across exam section

A1 A2 A3 A4 A5 A6 A7 A8

0.90 0.94 0.95 0.94 0.67 0.94 0.92 0.94

0.92 0.88 0.93 0.86 0.96 0.74 0.91 0.91

0.93 0.92 0.86 0.85 0.91 0.94 0.80 0.95

0.93 0.88 0.92 0.75 0.93 0.94 0.92 0.69

Table 2: Percentage of Group Completing Assignment

Econometric analysis: overall exam performanceVariable Coefficient P-value

Constant -117.43 0

Overall Assignment Mark 1.4 0

CAO Points .2 0

Results in 1st Year Economics 1.85 0

Gender .28 .94

Variable Coefficient P-value

Constant -158.85 0

No. of Assignments Complete 8.8 0

CAO Points .23 0

Results in 1st Year Economics 2.2 0

Gender .33 .94

Observations 254

Overall exam performance

• The results are in line with expectations. We replaced overall assignment mark with a variable that measured the number of assignments attempted. This variable is also significant and has a very large coefficient. However there is relatively little variation in this variable.

Econometric analysis: section specific regressions

1 2 3 4 Constant 4.7

(.58) -14.42

(.1) -24.58 (.01)

-27.35 (.01)

CAO Points

-.0045 (.83)

.0166 (.48)

.055 (.03)

.075 (007)

1st Year Result

.227 (.007)

.5 (0)

.214 (.024)

.29 (.006)

Marks in Ass i

.144 (.094)

.04 (.58)

.244 (.03)

.08 (.48)

Gender 2.65 (.062)

-.38 (.784)

-1.15 (.48)

2.12 (.24)

Aplia/paper Dummy

.475 (0.778)

3.9 (.026)

1.42 (0.45)

-2.7 (.19)

Observations 221 174 172 159

R2 .1 .31 .195 .22 p-values in parentheses

Econometric analysis: section specific regressions

5 6 7 8 Constant -23.74

(.03) -23.74 (.03)

-32.93 (0)

-33.17 (0)

CAO Points

.04 (.09)

-.001 (.96)

.053 (.02)

.035 (.035)

1st Year Result

.28 (.005)

.575 (0)

.395 (0)

.36 (0)

Marks in Ass i .035 (0.66)

-.01 (.88)

.1 (.49)

.14 (.017)

Gender -1.01 (.54)

-.9 (.5)

.18 (.9)

.075 (.95)

Aplia/paper Dummy

-2.89 (.129)

2.79 (.122)

1.91 (.27)

.39 (.788)

Observations 105 162 170 180

R2 .22 .32 .34 .3 p-values in parentheses

Section specific regressions

• Aplia/paper dummy is statistically significant only in Section 2. In that case students who did the assignment by Aplia performed somewhat better in that section of the exam.

• The number of marks that a student received in an assignment is statistically significant with a positive sign in the regressions on Section 3 and Section 8 and is weakly significant in the regression on Section 1.

Panel analysis

• The variation across students and across sections in our dependent variable meant that our data could be analysed using panel data techniques

• Pooled data

• Fixed effects

• Random effects

Econometric analysis: Pooled DataVariable Coefficient P-value

Constant -16.9 0

Assignment Mark .09 0

Aplia/Paper Dummy .66 .288

Dummy for Section 2 3.0 0

Dummy for Section 3 -1.57 .1

Dummy for Section 4 3.85 0

Dummy for Section 5 -7.6 0

Dummy for Section 6 -3.6 0

Dummy for Section 7 -4.3 0

Dummy for Section 8 -3.03 0

CAO Points .033 0

Results in 1st Year Economics .35 0

Gender .23 .65

Observations 1343

Econometric analysis: fixed effectsVariable Coefficient P-value

Constant 21.56 0

Assignment Mark .009 .766

Aplia/Paper Dummy .57 0.334

Dummy for Section 2 3.1 0

Dummy for Section 3 -1.88 .04

Dummy for Section 4 3.22 .001

Dummy for Section 5 -7.25 0

Dummy for Section 6 -4.11 0

Dummy for Section 7 -5.01 0

Dummy for Section 8 -3.1 0

Rho .43

Observations 1343

Econometric analysis: Random EffectsVariable Coefficient P-value

Constant -17.1 0

Assignment Mark .07 .011

Aplia/Paper Dummy .64 .274

Dummy for Section 2 3.13 0

Dummy for Section 3 -1.65 .07

Dummy for Section 4 3.61 0

Dummy for Section 5 -7.52 0

Dummy for Section 6 -3.85 0

Dummy for Section 7 -4.6 0

Dummy for Section 8 -3.1 0

CAO Points .034 0

Results in 1st Year Economics .36 0

Gender .18 .77

Rho .136

Observations 1343

Discussion: Results

• Very little evidence that Aplia v Written format of the assignment matters

• Little evidence of a direct link between performance in an assignment and mark on the corresponding section in the exam.

• If there is no difference in learning outcome then– Cost (student & instructor) becomes important– Students’ preferences become important

Discussion: Issues

• Where might learning from Aplia take place?– Between practice assignment and actual

assignments?– Incrementally so that ALL students do a better

examination – all have used Aplia 6 times?

• Does the examination test learning from Aplia?• Is the experiment testing the benefits of

technology and/or assignments rather than Aplia?

• Is there cheating and/or cooperative learning?

Discussion: Future Work

• More analysis of existing data– Including two stage estimation

• Change Experiment– Do not allow practice assignments for some of

2009-10 assignments• Compare with other 2009-10 assignments• Compare with exact same 2008-9 assignment

• Is Aplia fit-for-purpose?– What purpose?