Figure 1. From professional development to student achievementWe predict that student achievement will improve by providing teachers with improved mathematics content and pedagogy. These treatments are applied to teachers in grades 4-8.
Modeling student achievement in K-12 math and science classrooms: An experimental approachM.J. Bryant, R.A. Cardullo, K. Bocian, K.A. Hammond. University of California, Riverside, CA 92521
Figure 2. Comparison of possible experimental designs
The treatment () can be applied to either entire schools or to individual classrooms within a school. Schools or classrooms that do not receive the treatment serve as controls (). Ignoring issues of human subjects, the treatment would be applied to half of the schools or classrooms. The bottom two panels illustrate and example of increasing the proportion of treatment to control groups which is desirable in an educational experiment.
Year 1 Year 2 Year 3 Year 4 Year 1 Year 2 Year 3 Year 4
Treatment Applied to Schools Treatment Applied to Classrooms
1 2 3 4 5 6 7 8 910 11 12 13 14 15 16
Tre
atm
ent:
Con
trol
Con
stan
t(S
choo
l)T
reat
men
t:C
ontr
ol I
ncre
ases
(Sch
ool)
1 2 3 4 5 6 7 8 910
11 12 13 14 15 16
0.00
0.05
0.10
0.15
0.20
0.0 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0
Alp
ha (
< 0
.05
cons
ider
ed s
igni
fica
nt)
Probability of P < Alpha
Treatment Applied to:SchoolsClassrooms
Classroom design has the most statistical power
School design has the most statistical power
The experimental design with the highest statistical power is dependent on where the variation is greatest.
Variation greatest among schools Variation greatest among classrooms
0.0
Figure 3. Monte Carlo based power analyses inform which design has the highest statistical power. The choice between schools or classrooms as the experimental unit is subject to many logistical considerations. The design impacts statistical power (the probability that a significant difference is found when a difference exists). Computer simulations reveal that statistical power is optimized as a function of where random variation lies. For example if all classrooms in a school are equal but variation exists among schools, randomly assigning treatment to classrooms has greater statistical power.
Figure 4. Assignment of treatment and control groups as a stratified random design.In educational settings, demographic factors (i.e. socio-economic status or SES) are known to have a great impact on student performance, but random assignment of treatment is essential. Random treatment assignment may lead to unintended pooling on either end of an SES continuum. By selecting schools at random from matched pairs, we can control for differences in SES. Here schools were matched by principal components analysis. Blue arrows connect matched pairs ( = Treatment, = Control). Perfect pairing may not be possible for all schools (▲ = schools with SES characteristics outside the range of the other schools).
-3
-2
-1
0
1
2
3
-5 -4 -3 -2 -1 0 1 2 3 4 5
Schools matched by demographic variablesfor stratified random assignment
Index Based on Aggregated Student Level Characteristics (i.e. % reduced or free lunch)
Inde
x B
ased
on
Cha
ract
eris
tics
of t
he S
choo
l (i
.e. C
lass
Siz
e)
Treatment
4th GradeTeachers
5th GradeTeachers
6th GradeTeachers
7th GradeTeachers
4th GradeStudents
5th GradeStudents
6th GradeStudents
7th GradeStudents
Time (2002-2006)
A core team consisting of mathematicians, scientists, and educators develops a treatment
The treatment is administered to teachers in the treatment group
Teachers teach with learned techniques
Efficacy of the treatment is measured through student test scores
This several year long study measures the growth in both teachers and students over time
INTRODUCTION AND STATEMENT OF PROBLEM
Mathematical Achievement through Collaboration with Teachers and Students (Mathematical ACTS) is a multi-faceted NSF-funded research project designed around specifically tailored initiatives for teachers and students in the 4th – 8th grades. ACTS seeks to raise achievement and close gaps in mathematics for all students in the Jurupa Unified School District through a series of programs (150 hours annually) for teachers and students. For the purpose of this project, student achievement in mathematics is defined as preparation for and success in Algebra I, the gatekeeper class for higher mathematics courses and entrance colleges and universities. We targeted students in the Jurupa Unified School District 4th through 9th grades, beginning in September 2003 through June 2007. Low mathematics performance in K-12 impedes progress in college mathematics and science and its toll is being felt at many institutions of higher education. At the University of California, Riverside the percentage of students who take remedial mathematics (defined as pre-calculus or below) has risen from 65% in Fall, 2000 to 73% in Fall, 2003 (UCR Learning Center Fall, 2003 report). In addition, the majority of under-represented high school students in the 66th State Assembly District are not prepared for college. Less than 15% of students from this region attend any university (California Postsecondary Education Commission Report, 2003) and the percentage of students that take any mathematics beyond Algebra I is exceedingly low. Left unchecked, any cohort of students that is not well prepared in both mathematics and science will not be prepared to function in a technologically advanced society.
The effect of professional development on teaching and the subsequent achievement of students are studied in a research design that includes:•Comparison of teacher baseline and post ACTS intervention knowledge and use of mathematics and pedagogy. •Assessment of the implementation of ACTS teaching strategies in the classroom.•Assessment of the value added by the treatment in the classroom by tracking the rate of student achievement growth in targeted and control (wait-listed) classrooms.•Case studies of teacher and school implementation within one school site.•Assessment of benchmarks of systemic change for both UCR and JUSD. The initial study group includes teachers and students from 16 elementary and 3 middle schools. In dealing with a small group a number of questions arise concerning the research design, the quality of assessments used for evaluating both content and pedagogy, and the types of analysis used to maximize statistical power in quantifying the effectiveness of treatments. Since scientifically designed and controlled experiments are rare in education, Mathematical ACTS provides an opportunity to quantify the effectiveness of various teacher interventions.
Question: How can student achievement in mathematics be improved in public schools so that a larger fraction of students is eligible for Algebra I by eighth grade?
Hypothesis: Since students receive mathematics training from teachers, improving student achievement in elementary and middle school math classes can be attained by improving the math proficiency and pedagogy of teachers.
Prediction: As a result of ACTS professional development, student achievement, as measured by standardized tests, will significantly increase resulting in a measurable increase in Algebra I enrollment.
Figure 5. Mathematics content knowledge and pedagogy are related but distinct.Primary and secondary school teachers tend to have higher pedagogical skills (as measured by a University of Michigan designed pedagogy assessment) when their mathematics content proficiency is higher (CSU/UC Algebra Readiness Test). Secondary school teachers have significantly higher mathematics pedagogical skills, perhaps due to the fact they are single subject teachers. Despite the correlation, there remains considerable variation in pedagogical skill for all levels of content knowledge. The secondary school teachers’ baseline data serves as benchmark goal for the primary schools teachers.
0
0.2
0.4
0.6
0.8
1.0
0 0.2 0.4 0.6 0.8 1
Relationship Between Mathematics Proficiency and Pedagogy
Score on Mathematics Proficiency Test
Sco
re o
n M
athe
mat
ics
Ped
agog
y T
est
Elementary School TeachersSecondary School Teachers
Effect of Teacher’s Grade level: P = 0.0002
Percentage of Elementary School Teachers Correctly Answering Question
Per
cent
age
of S
econ
dary
Sch
ool T
each
ers
Cor
rect
ly A
nsw
erin
g Q
uest
ion
1 to 1
5050
60
60
70
70 80
80
90
100
90 100
Question 39
Question 1
1) 3.01 + 0.301+ 30 = ?(A) 0.631(B) 3.611(C) 33.311(D) 33.32
39) In the right triangle ABC shown to the right, what is the length of AC ?
(A) 10 (B) 14 (C) 24 (D) 48
B
A
C
6
8
Sample Questions
Figure 6. Baseline data can serve to inform professional development content.Primary and secondary school teachers do not understand all mathematics subject areas equally. The data points above the 1 to 1 line suggest relatively higher proficiency on a given question by secondary school teachers. By examining the relative performance on each question by the different groups of teachers we can identify concepts where a need for professional development is not indicated (i.e. Question 1). Questions with a great disparity between the two groups can be used to fine tune professional development in areas of content knowledge where it is most needed (Question 39 builds off of State Standards from the grade 4 through grade 7).
CONCLUSIONS 1. Any experimental design in education must take into account both traditional scientific design
motifs as well as ethical considerations (i.e., ultimately providing treatments for all subjects during the study).
2. To optimize statistical power the choice of schools or classrooms as experimental units depends on the amount of variation within that unit.
3. Pairs of experimental and control groups must be established by taking into account multiple variables (i.e., socio-economic status, school test scores, etc.).
4. In our study, there are significant differences in both math content and pedagogy between elementary (Grades 4-6) and Middle School (Grades 7 & 8) teachers. Further, there are defined areas within mathematics where there are significant differences between these two groups.
ACKNOWLEDGEMENTS
1. This work is funded by a Math Science Partnership grant from the National Science Foundation.2. We would like to acknowledge the support of the ALPHA Center at the UCR for assisting in the preparation of
assessment materials and data collection.3. We thank the Jurupa Unified School District for their cooperation and assistance with this project.