EA12-514-2 Evaluation of the Reasoning Mind Mathematics Program

Dallas Independent School District

EVALUATION OF THE REASONING MIND MATHEMATICS PROGRAM

2011-12

EA12-514-2

DEPARTMENT OF EVALUATION AND ACCOUNTABILITY

Mr. Mike Miles Superintendent of Schools

Nancy Directo

EVA

App

Kihneman, Por – Program

ALUATION

proved Report

Ph.D. Evaluation

Dallas Indepe

Mr

Superinte

N OF THE R

EA

Robert J

t of the Depar

DaJ

endent Schoo

. Mike Miles

endent of Sch

REASONIN

2011-12

A12-514-2

J. Costello Ph

rtment of Eva

Cecilia Execu Evalua

allas, Texas July 2012

ol District

hools

NG MIND P

h.D.

aluation and A

a Oakeley, Phutive Directoration and Acc

PROGRAM

Accountability

h.D.

countability

y

i

Table of Contents

Page

ABSTRACT………………………………………………………………………………………… 1

PROGRAM DESCRIPTION………………………………………………………………………. 3

PURPOSE AND SCOPE OF THE EVALUATION……………………………………………… 3

MAJOR EVALUATION QUESTIONS AND RESULTS………………………………………… 3

2.1 What were the demographic characteristics of grade two students in the Reasoning Mind program?.............................................................................………………………

3

Methodology…………………………………………………………………………..………. 3

Results………………………………………………………………………………….……… 3

2.2 What were the demographic characteristics and credential of grade two mathematics teachers?...........................................................................................……………….…

4

Methodology………………………………………………………………………………….. 4

Results……………………………………………………………………………………….… 4

2.3 What was the level of implementation of the Reasoning Mind program and Teacher Training?........………

5

Methodology………………………………………………………………………………..… 5

Results………………………………………………………………………………..……..… 5

2.4 What was the level of performance of Reasoning Mind students on the ITBS?……… 9

Methodology………………………………………………………………………………..… 9

Results……………………………………………………………………………………..…. 9

SUMMARY AND RECOMMENDATIONS………………………………………………….….. 20

APPENDIX………………………………………………….……………………………………..

21

ii

List of Tables

Table Page

1 Demographic Characteristics of Second Grade Title I Mathematics Teachers in Reasoning Mind Program ........................................................................................... 5

2 Reasoning Mind Implementation Categories .............................................................. 6

3 Number and Percent of Schools by Problems Corrected ........................................... 6

4 Number and Percent of Schools by Reasoning Mind Hours Online ........................... 6 5 Average Reasoning Mind Hours online by Category .................................................. 7 6 Percent of Students at or Above the 40th Percentile on the ITBS Mathematics Total Within Student Group and Year ................................................................................. 9 7 Number of Students at or above the 80th Percentile on the ITBS Mathematics Total Within Student Group and Year .................................................................................. 11 8 Percent of Students at or above the 80th Percentile on the ITBS Mathematics Total Within Student Group and Year .................................................................................. 11 9 Percent of Students at or Above the 40th Percentile on the ITBS Mathematics Computation within Student Group and Year ............................................................ 13 10 Number of Students at or Above the 80th Percentile on the ITBS Mathematics Computation within Student Group and Year ............................................................ 14 11 Percent of Students at or Above the 80th Percentile on the ITBS Mathematics Computation within Student Group and Year ............................................................ 14 12 Differences in the Percent of Students at or Above the 40th Percentile on the ITBS Mathematic Total within Student Groups and Years ......................................... 14 13 Differences in the Number of Students at or Above the 80th Percentile on the ITBS Mathematic Total within Student Groups and Years ......................................... 17 14 Differences in the Percent of Students at or Above the 80th Percentile on the ITBS Mathematic Total within Student Groups and Years ......................................... 18 15 Percent of Students at or Above the 40th Percentile on ITBS Mathematic Total by Online Hours Quartile ............................................................................................. 19

iii

List of Figures

Figure Page

1 Percent Students at or Above 40th Percentile on ITBS Mathematics Total within Group and Year .............................................................................................................. 10


3 Percent Students at or Above 40th Percentile on ITBS Mathematics Computation within Group and Year ................................................................................................... 13


5 Differences in the Percent of Students at or Above 40th Percentile from First to Second Grade on ITBS Mathematics Total .................................................................. 16

6 Differences in the Number of Students at or Above 80th Percentile from First to Second Grade on ITBS Mathematics Total .................................................................. 18

7 Differences in the Percent of Students at or Above 80th Percentile from First to Second Grade on ITBS Mathematics Total .................................................................. 19

1

FINAL REPORT

EVALUATION OF THE REASONING MIND PROGRAM 2011-12

Project Evaluator: Robert J. Costello Ph.D.

ABSTRACT

Approximately 13,400 students in 149 Title I schools received instruction in the Reasoning Mind mathematics supplementary program at the second grade level in the Dallas ISD for the school year 2011-12. Data provided by Reasoning Mind shows that the average number of hours online for Dallas ISD students for the year was 30.5 hours. Data further indicates that students received 13.4 hours online in semester one and 17.1 hours online in semester two. The range of online instruction was from 0.0 hours to 168.9 hours. Only seven schools had more than 50 hours of instruction. Based on six six-weeks sessions for a total of 36 weeks in a school year the students received approximately 50 minutes of instruction per week. Hours varied per school due to implementation problems in the first semester and to principal’s decisions on how to implement the program. Results of the ITBS Mathematics Total by year shows that for district 2nd grade overall 58 percent of the students scored at or above the 40th percentile an increase of 1.2 percentage points over the previous year; however, the 2011 results showed an increase of 2.2 percentage points over the 2010 results. Although increases were reported for all student groups, with the exception of African American students, the increases were less than those reported for 2010 to 2011 (prior to 2nd grade implementation of RM). The goal for increasing the percent of students reaching the 80th percentile by 10 percentage points was not met by either. In general, Reasoning Mind did not achieve their stated goals of increasing mathematics achievement for second grade Dallas ISD students.

PROGRAM DESCRIPTION

Reasoning Mind (RM) is a Texas-based nonprofit organization that has developed an adaptive,

online mathematics curriculum for grades 2-6. In an RM classroom, every child sits at a computer and

works online in the RM system and learns mathematical concepts while solving problems at his or her

own pace. The system creates an individual path of learning tailored to each child’s needs based on its

assessment of that student’s strengths and weaknesses. Students log in to the program online and are

guided through grade-appropriate lessons and corresponding problems at their own pace. All problems

are ranked and tailored in difficulty level based on a student’s correct answers on past problems.

Students can request “hints” for problems, and regardless of right or wrong answers, are provided a

“Genie Solution” for the most thorough method for computing every problem. While students receive

individualized instruction online, the teacher is free to give children one-on-one support and attention.

2

Teachers use computers to monitor all students in the classroom and immediately see what questions

students are getting right or wrong on each topic. Students having difficulty on lessons receive one-on-

one interventions or small-group tutorials. As a result, the entire learning experience is structured to suit

individual student needs. The RM program is designed to supplement traditional mathematics instruction

in grades 2-4 and as a core mathematics curriculum in grades 5-6.

The rationale for inclusion of the Reasoning Mind program into the Dallas ISD second grade

classrooms was that during the time period from 2007-08 to 2009-10 second grade students have not

done well on the ITBS Mathematics Total. African American second grade students decreased from 53.0

percent of students at or above the 40th percentile to 47.4 percent (in 2010-11 the percentage was 50.6)

and Special Education second grade students decreased from 28.2 percent at or above the 40th

percentile to 21.8 percent in 2009-2010 (in 2010-11 the percentage was 24.8). There appeared to be a

need for supplemental mathematics material for second grade students as well as a need to train

teachers and supporting staff in the use of supplemental materials.

District-wide, the total number of African American kindergarten through second grade students

at or above the 80th percentile on the ITBS Mathematics Total decreased each year, from 2,107 in

2007-2008 to 1,726 in 2009-2010 ( in 2010-11 there were 1,933 students).

Program Goals. Second grade results on the ITBS Mathematics Total will be five percentage

points higher than those of the 2009-2010 results for students at or above the 40th percentile. The

number of second grade students on the ITBS Mathematics Total at or above the 80th percentile, in each

sub-population will increase by at least ten percentage points higher than the number of students in

2009-2010.

Second grade results on the ITBS Mathematics Computation will be five percentage points higher

than those of the 2009-2010 results for students at or above the 40th percentile. Although these expected

outcomes were obtained from the Title I workscope for the Reasoning Mind program it is believed that

comparisons should also be made between the 2011-12 outcomes and the 2010-11 results rather than

the 2009-10 results. Also it is assumed that since comparisons were made with the number of students

on the ITBS Mathematics Total at or above the 80th percentile that similar comparisons should be made

for students on the ITBS Mathematics Computation at or above the 80th percentile. One has to question

3

why goals were set for comparisons with two year old data rather than last year’s data. Given that district

ITBS scores for second grade students have been increasing since 2009 it would seem that Reasoning

Mind would be getting credit for increasing scores in 2010 when the program was not in Dallas ISD

second grade at the time.

PURPOSE AND SCOPE OF THE EVALUATION

The purpose of the evaluation was to determine the level of program effectiveness in

accomplishing the goals of the Reasoning Mind mathematics program. Mathematics achievement for

grade two was measured by the Iowa Tests of Basic Skills (ITBS) using both the mathematics total and

the mathematics computation subtests. In addition, the evaluation discusses the staff training received by

second grade teachers to implement the program and the implementation schedule.

MAJOR EVALUATION QUESTIONS AND RESULTS

2.1 What were the demographic characteristics of grade two students

in the Reasoning Mind program?

Methodology

The district’s student database was utilized to extract enrollment, demographic, and achievement

data. Current year enrollment data were extracted from the Dallas ISD student database on

November 1, 2011. Information on implementation, school schedules and student participants were

obtained from the Reasoning Mind coordinators. RM has students in several schools at the third and

fourth grades which were not included since this evaluation was for Title I funded projects, which in this

case were second grade students only.

Results

A total of 13,419 students were enrolled in second grade at 149 Title I schools; four elementary

schools chose not to implement Reasoning Mind (George Dealey, Gabe Allen, Harry Stone and Stevens

Park). Approximately 23 percent of the RM students were African American, 70 percent were Hispanic

and the remaining 7 percent were classified as Others. Fifty-two percent (52%) were males, while 48

percent were females. Fifty-one percent (51%) were limited English proficient (LEP) or English Language

4

Learners (ELL), eight percent were Special Education students and two percent were LEP Special

Education students. Economically disadvantaged students made up 90 percent of the second grade RM

population.

2.2 What were the demographic characteristics and credential status

of grade two mathematics teachers?

Methodology

Demographic and credential data were extracted from the Dallas ISD personnel and course file

databases on November 5, 2011. Job codes and course designations were used to identify elementary

mathematics teachers. Demographic information included gender, ethnicity, highest degree completed,

number of years of teaching experience in the Dallas ISD and type of certification.

Results

A total of 377 teachers were identified as the teacher of record for mathematics at grade 2 for

students at the 149 Title I schools. Of these teachers 82 percent were female and 18 percent were male.

In addition, 32 percent were African Americans, 40 percent were Hispanics, 26 percent were White and

2 percent were Others. Of the second grade mathematics teachers 65 percent had bachelor’s degrees,

30 percent had Master’s degrees, 1 percent had Doctorate’s and 5 percent had missing information

(Table 1). Thirty-nine percent (39%) of the teachers had five or less years of teaching experience in the

district. However, 13 percent had 21 or more years of district teaching experience. The median number

of years of experience in the district was eight years for second grade mathematics teachers. In addition,

three percent of the second grade mathematics teachers held alternative certifications.

5

Table 1

Demographic Characteristics of Second Grade Title I Mathematics Teachers in Reasoning Mind Program

Characteristics Number Percentage Highest Degree Earned

Bachelor’s 374 64.9 Master’s 170 29.5 Doctorate 6 1.0 Missing information 26 4.5

Years Teaching in the District

< 1 23 4.0 1-5 201 34.9 6-10 153 26.6 11-15 80 13.9 16-20 46 8.0 21+ 73 12.6

2.3 What was the level of implementation of the Reasoning Mind program and Professional Development?

Methodology

Implementation data for RM schools was collected and given to the Dallas ISD Accountability and

Evaluation department by the Reasoning Mind regional coordinator and the Dallas ISD Professional

Development department.

Results

Implementation Guidelines. Each campus received the following implementation guidelines:

1. All grade 2 students are expected to use the Reasoning Mind program.

2. Students should receive at least two hours per week in Reasoning Mind in a computer lab setting.

3. Reasoning Mind classes should be scheduled for at least 45 minutes per session.

4. The ideal Reasoning Mind schedules are three sessions per week for 45 minutes or two sessions per week for 60 minutes.

Once principals received the guidelines, they made the decision on how Reasoning Mind was

implemented on their campus. Toward the end of the first semester, Reasoning Mind placed schools in

one of four categorizes (Struggling, Fair, Good, and Excellent) according to the percent of problems

6

corrected and the number of RM hours online as of December 2011 (Table 2). For example a struggling

school was one that had between 0 percent and 50 percent of its basic problems corrected and/or

students received between 0.0 to 15.5 hours of RM online.

Table 2

Reasoning Mind Implementation Categories

Legend % of Basic Problems Corrected RM Hours Online Struggling 0.00 to 50.00 0.00 to 15.53 Fair 50.10 to 60.00 15.54 to 17.93 Good 60.10 to 70.00 17.94 to 20.32 Excellent 70.10 to 100.00 20.33 or greater

Table 3 shows that there was only 1 school in the Struggling category for percent of problems

corrected meaning that less than 50 percent of the basic problems were corrected at that school, while 25

(16.8%) of the schools had between 60 and 70 percent of the basic problem corrected and 122 (81.9%)

of the schools had 70 to 100 percent of the basic problem corrected.

Table 3

Number and Percent of Schools by Problems Corrected

Legend Number Percent Struggling 1 0.7 Fair 0 0.0 Good 25 16.8 Excellent 122 81.9

Table 4 shows that approximately 78 percent of the schools offered 18 hours or less of RM online

from September 6th through December 9th (Fair + Struggling schools).

Table 4

Number and Percent of Schools by Reasoning Mind Hours Online

Legend Number Percent Struggling 91 61.1 Fair 25 16.8 Good 10 6.7 Excellent 22 14.8

Schools rated Excellent for RM hours online showed an average of 26.8 hours of online

instruction (Table 5). Number of hours of online instruction for Excellent schools ranged from 20.7 at

7

Miller to 46.3 at Bonham. Of the 149 schools in the RM program 22 (14.8%) were classified as Excellent

for number of hours online.

Schools rated Good for RM hours online showed an average of 18.8 hours of online instruction.

Number of hours of online instruction ranged from 18.0 at Dorsey to 19.9 at Pershing. Of the 149 schools

in the RM program 10 (6.7%) were classified as Good for number of hours online.

Schools rated Fair for RM hours online showed an average of 16.7 hours of online instruction.

Number of hours of online instruction ranged from 15.6 at Holland to 17.9 at both Cowart and Turner. Of

the 149 schools in the RM program 25 (16.8%) were classified as Fair for number of hours online.

Schools rated Struggling for RM hours online showed an average of 8.46 hours of online

instruction. Number of hours of online instruction ranged from 0.0 at Alexander to 16.4 at Terry. Of the

149 schools in the RM program 91 (61.1%) were classified as Struggling for number of hours online.

Overall, the average number of hours of RM instruction for all second grade students for the first

semester was 13.4 hours. Given that only one school was in the Struggling category for problems related

to implementation it would seem then that the struggling group should have more RM hours online. The

range for the 149 Dallas ISD schools for RM online instruction was from 46 hours to 0 hours.

Table 5

Average Reasoning Mind Hours Online by Category

Legend Number Percent Average No. of Hours Online Instruction

Struggling 91 61.1 8.46Fair 25 16.8 16.66Good 10 6.7 18.81Excellent 22 14.8 26.80Total 1481 100.0 13.40 1 Data was unavailable for the Disciplinary Alternative Education Program (DAEP)

Data provided by Reasoning Mind shows that the average number of hours online for Dallas ISD

students for the year was 30.6 hours. That indicates that students received 13.4 hours online in semester

one and 17.2 hours online in semester two. Based on six six-weeks sessions for a total of 36 weeks in a

school year, the students received approximately 50 minutes of instruction per week. The range of online

instruction was from 0.0 hours to 168.9 hours. One has to question the validity of the hours online data.

For students to receive 160 or more hours of RM, they would have to receive 4 hours a week of online

8

instruction every week of the school year. In addition, variations within schools seem very unlikely,e.g.,

school A has a range of 124 hours of online instruction with students receiving as little as 27 hours of

instruction and as much as 151 hours (see Appendix A). Variations within schools such as the one just

cited were quite common. Given these ranges and other anomalies in the data the veracity of the hours

online data is highly questionable. A frequency distribution of hours online instruction showed that over

1,000 students received 10 hours or less of RM instruction for the year. It is this evaluator’s

understanding that the district pays for individual registration for each student, and that would amount to a

thousand registration fees for students receiving 17 minutes or less of instruction per week.

Reasoning Mind provided an extensive teacher professional development program to ensure that

teachers are prepared for their new role and familiar with the RM curriculum. The program consists of a

combination online and in-person Teacher Qualification Course, and continues with an additional

yearlong Certification Course which teachers take concurrently with their first year of RM teaching. These

courses are designed to expand teachers’ knowledge of mathematics content and pedagogy, but also to

support higher-quality classroom instruction. In addition, teachers receive ongoing professional

development through structured collaborations with Program Coordinators, as well as access to tutorials

and symposiums hosted by RM.

Reasoning Mind also provides ongoing support to teachers and students in schools using the

program. Program Coordinators meet regularly with teachers to monitor the progress of each classroom.

Program Coordinators also provide support and guidance to effectively integrate technology into their

classrooms.

Data available through the district’s professional development department showed that

Reasoning Mind had two sessions, a day 1 and a day 2 session for grade 2 teachers. Reasoning Mind's

training was a two-day, computer-based course designed to introduce teachers to basic system,

curriculum, and implementation knowledge needed to start a Reasoning Mind classroom in grade 2.

Since a stipend was paid for day 2, it could not be used to satisfy mandatory content credit/compensatory

time requirements. Day 1 sessions (Content Credit) were attended by 222 teachers and were held on

June 27, July 11, 13, 18, 20 & 25, and August 1, 3, 8, 10 & 11 of 2011. Day 2 sessions (Stipend Credit)

9

attended by 214 teachers were held on the following days: June 28, July 12, 14, 19, 21 & 26, and August

4, 9, 11 & 12 of 2011.

2.4 What was the level of performance of Reasoning Mind students on the ITBS?

Methodology

Mathematics subtests of the Iowa Tests of Basic Skills (ITBS) have been used to determine the

level of performance of Reasoning Mind students. Mathematics Total results used for analysis are

combinations of the Mathematics Concepts, Mathematics Problems, and Mathematics Computation

subtest results. Mathematics Computation statistics are provided separately. The following tables and

figures show the ITBS results for second grade students from 2007-08 to 2011-12. The number and

percent of students at or above the 40th and 80th percentile are reported for both Mathematics Total and

Mathematics Computation.

Results

Table 6 and Figure 1 report the results of the ITBS Mathematics Total by year and student group.

Overall, the district result of 58.0 percent of second grade students at or above the 40th percentile shows

an increase of 1.1 percentage points over the previous year; however, the 2011 results showed an

increase of 2.2 percentage points over the 2010 results. Although increases were reported for all student

groups, with the exception of African American students, the increases were less than those reported for

2010 to 2011 for all student groups. The White student group did perform at a higher level than other

student groups.

Table 6

Percent of Students at or Above the 40th Percentile on the ITBS Mathematics Total within Student Group and Year

School Student Groups Year Ending White

African American Hispanic

Econ Disadv

Special Educ LEP District

Number Tested

2008 75.0 53.0 57.9 56.2 28.2 54.8 57.5 13,407

2009 74.5 47.4 54.3 50.7 27.2 47.0 53.2 8,787

2010 76.1 49.7 54.7 53.1 21.8 52.7 54.6 13,231

2011 78.0 50.6 57.5 55.6 24.8 56.2 56.8 13,135

2012* 82.7 48.9 59.0 56.2 26.0 57.1 57.9 13,603 Source: MyData Portal Dallas ISD *School year when Reasoning Mind was implemented.

10

Figure 1 is a graphical presentation of the data presented in Table 6.

Figure 1 Percent students at or above 40th percentile on ITBS Mathematics Total within student group and

year.

Table 7 presents the number of students at or above the 80th percentile on the ITBS Mathematics

Total by year and student group. With the exception of the White subgroup all other student groups

showed a smaller gain in number from 2011 to 2012 than from the previous year. District totals show the

increase in number of students from 2010 to 2011 to be 262 students, while the increase from 2011 to

2012 was 70 students. In general, the number of students at or above the 80th percentile increased from

2011 to 2012, with the notable exception of African American students, but the increase is less than the

previous year.

White AfricanAm Hispanic Special

Ed LEP EconDisad District

2008 75 53 57.9 56.2 28.2 54.8 57.52009 74.5 47.4 54.3 50.7 27.2 47 53.22010 76.1 49.7 54.7 53.1 21.8 52.7 54.62011 78 50.6 57.5 55.6 24.8 56.2 56.82012 82.7 48.9 59 56.2 26 57.1 57.9

0

10

20

30

40

50

60

70

80

90

% Stude

nts a

t or A

bove

40th Pe

rcen

tile

11

Table 7

Number of Students at or Above the 80th Percentile on the ITBS Mathematics Total within Student Group and Year



Econ Disadv


Number Tested

2008 254 686 1,927 2,451 56 1,223 2,912 13,407

2009 275 556 803 1,289 47 259 1,686 8,787

2010 230 560 1,649 2,122 35 1,123 2,484 13,231

2011 252 565 1,857 2,343 45 1,282 2,746 13,135

2012 300 497 1,955 2,356 50 1,340 2,816 13,048 Source: MyData Portal Dallas ISD *School year when Reasoning Mind was implemented.

Table 8 reports the percentage of students at or above the 80th percentile on the ITBS

Mathematics Total by year and student group. Special Education students showed no gain while African

American students reported a loss from 2011 to 2012. District gains were 0.7 percentage point in 2012 a

decrease in gains for the 2011 year (2.1%). Yearly gains for student groups showed an increase for White

students from 2011 to 2012 over an increase from 2010 to 2011. All other student groups remained the

same, decreased (African Americans) or showed smaller gains from 2011-12 than in 2010-11.

Table 8

Percent of Students at or Above the 80th Percentile on the ITBS Mathematics Total within Student Group and Year



Econ Disadv


Number Tested

2008 1.9 5.1 14.4 18.3 0.4 9.1 21.7 13,407

2009 3.1 6.3 9.1 14.7 0.5 3.0 19.2 8,787

2010 1.7 4.2 12.5 16.0 0.3 8.5 18.8 13,231

2011 1.9 4.3 14.1 17.8 0.3 9.8 20.9 13,135


12


Figure 2. Percent students at or above 80th percentile on ITBS Mathematics Total within student group by

year.

Table 9 reports the percent of students at or above the 40th percentile on the ITBS Mathematics

Computation by student group and year. All student groups increase the percentage of students at the

40th percentile from the previous year, with the exception of African American students. District students

showed a 1.6 percentage point gain from 2011 to 2012 compared to a 0.6 percent gain from 2010 to

2011. LEP and Hispanic students showed gains after reporting decreases the previous year; also Special

Education students reported an increase in the percentage of students at or above the 40th percentile

from the previous year.



2008 1.9 5.1 14.4 0.4 9.1 18.3 21.72009 3.1 6.3 9.1 0.5 3 14.7 19.22010 1.7 4.2 12.5 0.3 8.5 16 18.82011 1.9 4.3 14.1 0.3 9.8 17.8 20.92012 2.3 3.8 14.9 0.3 10.3 18 21.6

02468

1012141618202224

% S

tude

nts

at o

r Abo

ve th

e 80

th P

erci

ntle

13

Table 9

Percent of Students at or Above the 40th Percentile on the ITBS Mathematics Computation within Student Group and Year



Econ Disadv


Number Tested

2008 70.7 57.2 65.9 63.2 41.1 65.3 64.0 13,384

2009 67.5 51.8 63.7 60.3 36.1 65.5 60.9 13,566

2010 65.4 52.8 64.2 60.9 33.1 64.9 61.4 13,270

2011 70.9 55.2 63.7 61.3 34.6 64.4 62.0 13,156



Figure 3. Percent students at or above 40th percentile on ITBS Mathematics Computation within student

group and year.

White AfricanAmerican Hispanic Econ

DisadvSpecialEduc LEP District

2008 70.7 57.2 65.9 63.2 41.1 65.3 642009 67.5 51.8 63.7 60.3 36.1 65.5 60.92010 65.4 52.8 64.2 60.9 33.1 64.9 61.42011 70.9 55.2 63.7 61.3 34.6 64.4 622012 75.6 52.8 66.1 62.6 38.9 66.8 63.6

0

10

20

30

40

50

60

70

80

% Stude

nts a

t or A

bove

40th Pe

rctile

14

Table 10 reports the number of students at or above the 80th percentile by student group and

year. The number of students in the African American, Economically Disadvantaged student groups and

for the district overall decreased from 2010-11 to the present year.

Table 10

Number of Students at or Above the 80th Percentile on the ITBS Mathematics Computation within Student Group and Year



Econ Disadv

Special Educ LEP

District (Title I)

Number Tested

2008 138 618 1,755 2,221 55 1,222 2,557 13,384

2009 222 787 2,758 3,422 88 2,059 3,822 13,566

2010 185 769 2,764 3,421 70 2,106 3,776 13,270

2011 225 737 2,756 3,419 72 2,111 3,810 13,156

2012* 256 635 2,812 3,356 86 2,615 3,781 13,012 Source: MyData Portal Dallas ISD *School year when Reasoning Mind was implemented.

Table 11 and Figure 4 showed the percent of students at or above the 80th percentile on the ITBS

Mathematics Computation by student group and year. Overall, district students showed a 0.1 percent gain

over last year. White and Economically Disadvantaged students showed losses from last year, while

Special Education and LEP students had the same gain from 2010 to 2011 as in 2011 to 2012. Hispanics

showed a slight increase in gains from the previous year, while African Americans showed a loss in the

percentage of students at the 80th percentile from the previous year.

Table 11

Percent of Students at or Above the 80th Percentile on the ITBS Mathematics Computation within Student Group and Year



Econ Disadv

Special Educ LEP

District (Title I)

Number Tested

2008 1.0 4.6 13.1 16.6 0.4 9.1 19.1 13,384

2009 1.6 5.8 20.3 25.2 0.7 15.1 28.2 13,566

2010 1.4 5.8 20.8 25.8 0.5 15.9 28.5 13,270

2011 1.7 5.6 21.0 26.0 0.6 16.1 29.0 13,156

2012* 1.9 4.9 21.6 25.8 0.7 16.3 29.1 13,012

15

Source: MyData Portal Dallas ISD *School year when Reasoning Mind was implemented.

Figure 4 is a graphical presentation of the data presented in Table 11

Figure 4 Percent students at or above 80th percentile on ITBS Mathematics Total within group and year.

Table 12 shows the difference in the percent of students at or above the 40th percentile on ITBS

Mathematics Total from first to second grade, starting in 2008 and ending in 2012. The difference

between the percentage of students in the first grade and the same group of students at the second

grade at or above the 40th percentile on the ITBS Mathematics Total shows a decrease in students for

2008-09 and increases for the following three year groupings. However, as Figure 5 shows those

increases are not consistent. For the grade grouping 2009-10 there was an increase of 4.2 percentage

points, 8.0 percentage points for 2010-11 and then in 2011-12 there was a decrease to only a 4.8

percentage point increase.



2008 1 4.6 13.1 0.4 9.1 16.6 19.12009 1.6 5.8 20.3 0.7 15.1 25.2 28.22010 1.4 5.8 20.8 0.5 15.9 25.8 28.52011 1.7 5.6 21 0.6 16.1 26 292012 1.9 4.9 21.6 0.7 16.3 25.8 29.1

0

5

10

15

20

25

30

35

% o

f Stu

dent

s at

or A

bove

80t

h Pe

rcen

tile

16

Table 12

Difference of Percent of Students at or Above the 40th Percentile on the ITBS Mathematics Total within Student Groups and Years



Econ Disadv

Special Educ LEP

District (Title I)

Number Tested

Grade 1 2008 77.2 51.2 55.3 53.8 32.5 52.2 55.5 14,263 Grade 2 2009 74.5 47.4 54.3 50.7 27.2 47.0 53.2 8,787 Difference -2.7 -3.8 -1.0 -3.1 -5.3 -5.2 -2.3 Grade 1 2009 75.2 45.1 50.7 47.7 30.1 44.0 50.4 8,446 Grade 2 2010 76.1 49.7 54.7 53.1 21.8 52.7 54.6 13,231 Difference 0.9 4.6 4.0 5.4 -8.3 8.7 4.2 Grade 1 2010 74.9 47.2 47.6 47.0 22.3 45.5 48.8 13,666 Grade 2 2011 78.0 50.6 57.5 55.6 24.8 56.2 56.8 13,135 Difference 3.1 3.4 9.9 8.6 2.5 10.7 8.0 Grade 1 2011 78.1 53.6 50.8 51.3 33.0 47.3 53.1 13,444 Grade 2 2012* 82.7 48.9 59.1 56.2 26.1 57.3 58.0 13,048 Difference 4.6 -4.7 8.2 4.9 -7.0 9.8 4.8 Source: MyData Portal Dallas ISD *School year when Reasoning Mind was implemented.


2008‐09 2009‐10 2010‐11 2011‐12District ‐2.3 4.2 8 4.8

‐4

‐2

0

2

4

6

8

10

ITBS

Differen

ce from

Grade

1 to

Grade

2

Mathematics Total

17

Figure 5. Differences in the Percent of Students at or Above 40th Percentile from First to Second Grade on ITBS Mathematics Total.

Table 13 shows the difference in the number of students at or above the 80th percentile on ITBS

Mathematics Total from first to second grade, starting in 2008 and ending in 2012. The difference

between the percentage of students in the first grade and the same group of students at the second

grade at or above the 80th percentile on the ITBS Mathematics Total shows a decrease in students for

2008-09 and increases for the following three year groupings. However, as Figure 6 shows those

increases are not consistent. For the grade grouping 2009-10 there was an increase of 907 students, a

decrease to 480 students for 2010-11 and then in 2011-12 there was a decrease to only 100 students.

Table 13

Difference of Number of Students at or Above the 80th Percentile on the ITBS Mathematics Total within Student Groups and Years



Econ Disadv

Special Educ LEP

District (Title I)

Number Tested

Grade 1 2008 300 667 1,776 2,305 50 1,079 2,800 14,263 Grade 2 2009 275 556 803 1,289 47 259 1,686 8,787 Difference -25 -111 -973 -1,016 -3 -820 -1,114 Grade 1 2009 253 570 718 1,199 36 218 1,577 8,446 Grade 2 2010 230 560 1,649 2,122 35 1,123 2,484 13,231 Difference -23 -10 931 923 -1 905 907 Grade 1 2010 262 559 1,390 1,855 36 922 2,266 13,666 Grade 2 2011 252 565 1,857 2,343 45 1,282 2,746 13,134 Difference 262 6 467 488 9 360 480 Grade 1 2011 310 660 1,669 2,243 47 1,016 2,716 13,444 Grade 2 2012* 300 497 1,955 2,356 50 1,340 2,816 13,048 Difference -10 -163 286 113 3 324 100 Source: MyData Portal Dallas ISD *School year when Reasoning Mind was implemented.

18


Figure 6. Difference in the number of students at or above the 80th percentile from first to second grade on

the ITBS Mathematics Total.

Converting the number of students at or above the 80th percentile (Table 13) to percents results in

Table 14.

Table 14

Differences in the Percent of Students at or Above the 80th Percentile on the ITBS Mathematics Total within Student Groups and Years

School Student Groups

Year Ending White African

American Hispanic Econ

Disadv Special Educ LEP

District (Title I)

Number Tested

Grade 1 2008 2.1 4.7 12.5 16.2 0.4 7.6 19.6 14,263 Grade 2 2009 3.1 6.3 9.1 14.7 0.5 2.9 19.2 8,787 Difference 1.0 1.7 -3.3 -1.5 0.2 -4.6 -0.4 Grade 1 2009 3.0 6.7 8.5 14.2 0.4 2.6 18.7 8,446 Grade 2 2010 1.7 4.2 12.5 16.0 0.3 8.5 18.8 13,231 Difference -1.3 -2.5 4.0 1.8 -0.2 5.9 0.1 Grade 1 2010 1.9 4.1 10.2 13.6 0.3 6.7 16.6 13,666 Grade 2 2011 1.9 4.3 14.1 17.8 0.3 9.8 20.9 13,134 Difference 0.0 0.2 4.0 4.3 0.1 3.0 4.3

2008‐09 2009‐10 2010‐11 2011‐12District ‐1,114 907 480 100

‐1,500

‐1,000

‐500

0

500

1,000

1,500

ITBS

Differen

ce from

Grade

1 to Grade

2

Mathematics Total

19

Grade 1 2011 2.3 4.9 12.4 16.7 0.3 7.6 20.2 13,444 Grade 2 2012* 2.3 3.8 15.0 18.1 0.4 10.3 21.6 13,048 Difference 0.0 -1.1 2.6 1.4 0.0 2.7 1.4 *School year when Reasoning Mind was implemented.

Figure 7 shows the percentage growth by year. The first to second grade growth in 2010 to 2011

was at a high of 4.3 percent compared to the previous grade groupings of 0.1 percent. However, growth

in the 2011 to 2012 grade grouping showed a decline to a 1.4 percent growth.


Figure 7. Differences in the percent of students at or above the 80th percentile from first to second grade on the ITBS Mathematics Total.

Students were assigned to quartiles according to the number of hours of online instruction in RM.

The quartiles were as follows; quartile 1 was 0.0 to 20.01 hours, quartile 2 was 20.02 to 30.61 hours,

quartile 3 was 30.62 to 41.28 hours and quartile 4 was greater than 41.29 hours of instruction. Table 15

shows the results of the percent of students at or above the 40th percentile on the ITBS Mathematics Total

by quartile. The results shows that the greater exposure to RM the higher the percent of students at or

above the 40th percentile. These results were based on data received from RM (See Appendix A). A

review of this data pointed out some questions regarding a wide range of hours online in some schools.

Therefore, the quartile results should not be interpreted as a definite conclusion that more hours online

led to higher percentages above the 40th percentile.

2008‐09 2009‐10 2010‐11 2011‐12District ‐0.4 0.1 4.3 1.4

‐1

0

1

2

3

4

5

ITBS

Differen

ces from Grade

1 to

Grade

2

Mathematics Total

20

Table 15

Percent of Students at or Above the 40th Percentile on ITBS Mathematics Total by Online Hours Quartile

Quartile 1 Quartile 2 Quartile 3 Quartile 4

46.6 55.4 59.5 62.8

Summary

The Reasoning Mind mathematics supplementary program was implemented at the second grade

level in the Dallas ISD for the school year 2011-12. Data provided by Reasoning Mind shows that the

average number of hours online for Dallas ISD students for the year was 30.5 hours. Data further

indicates that students received 13.4 hours online in semester one and 17.1 hours online in semester

two. The range of online instruction was from 0.0 hours to 168.9 hours. Only seven schools had more

than 50 hours of Instruction. Based on six six-weeks sessions for a total of 36 weeks in a school year the

students received approximately 50 minutes of instruction per week. Hours varied per school due to

implementation problems in the first semester and to a principal’s decision on how to implement the

program. The results of the ITBS Mathematics Total by year and student group shows that for district 2nd

grade overall, 58 percent of students scored at or above the 40th percentile, an increase of 1.1 percentage

points over the previous year; however, the 2011 results showed an increase of 2.2 percentage points

over the 2010 results. Although increases were reported for all student groups, with the exception of

African American students, the increases were less than those reported for 2010 to 2011 for all student

groups. Results of the quartile study of number of hours online during 2011-12 were not definitive.

Program goals were that second grade results on the ITBS Mathematics Total and Computation will be

five percentage points higher than those of the 2009-2010 results for students at or above the 40th

percentile. Even with the two years, the total percentage points gained were 3.3. The number of second

grade students on the ITBS Mathematics Total and Computation at or above the 80th percentile, in each

21

sub-population will increase by at least ten percentage points higher than the number of students in

2009-2010. One has to question why goals were set for comparisons with two year old data rather than

last year’s data. Given that district ITBS scores for second grade students have been increasing since

2009 it would seem that Reasoning Mind would be getting credit for increasing scores in 2010 when the

program was not in Dallas ISD second grade at the time.

In general, Reasoning Mind did not achieve their stated goals of increasing mathematics

achievement by the proposed amount for second grade Dallas ISD students. This is a costly program,

approximately $1,500,000 for the 2011-12 school year. Program results did not indicate significant

improvements for students served. Program implementation was not a major focus of this report;

however, there were indications that fidelity of implementation may be an issue. Therefore, since the

district has funded and expanded the program for 2012-13, evaluation should focus on both achievement

and implementation results. There should be oversight by district personnel to ensure appropriate

implementation.

22

Appendix A Grade 2 Students by School with Hours of Reasoning Mind On-line Instruction

School TEA #

Number of Students

Range # of Hours

Minimum # of Hours

Max # of Hours

Mean # of Hours

101 101 73.0 8.6 81.6 41.7 103 1 0.0 13.6 13.6 13.6 104 115 60.0 0.2 59.8 27.3 105 104 59.6 5.2 64.8 37.4 106 35 54.8 5.6 60.4 42.9 108 88 35.1 1.9 37.1 17.6 109 102 31.4 1.4 32.8 12.1 110 133 24.7 0.0 24.7 1.9 111 41 53.4 35.2 88.6 64.4 112 92 47.2 0.6 47.8 32.0 114 53 65.5 0.5 66.0 23.4 115 109 56.5 0.1 56.6 32.0 116 178 39.6 1.6 41.2 20.5 117 112 39.2 2.3 41.5 16.9 118 40 26.4 0.5 26.9 14.8 119 67 19.5 8.2 27.7 16.3 120 108 78.2 5.3 83.5 50.9 121 57 61.8 0.1 61.9 24.1 122 71 26.4 6.5 32.9 21.7 124 92 40.7 0.4 41.1 17.4 125 118 48.6 0.6 49.1 32.0 127 33 52.5 3.3 55.8 36.5 128 29 40.5 2.1 42.6 27.1 129 100 63.1 0.4 63.4 26.6 130 111 65.3 7.1 72.4 46.0 131 59 42.3 3.9 46.2 28.8 133 79 39.6 7.9 47.5 25.9 135 61 28.3 8.9 37.3 29.2 136 72 21.9 8.7 30.7 17.4 137 80 60.1 2.5 62.6 33.7 139 55 63.3 0.1 63.5 29.6 140 32 64.1 7.2 71.3 46.6 141 53 68.5 4.3 72.8 56.3 142 102 57.6 0.1 57.8 26.1 143 38 124.1 26.6 150.7 59.5 144 76 28.5 2.3 30.7 14.4 145 119 60.5 20.7 81.2 47.7 146 34 40.8 5.5 46.3 21.8 147 125 88.5 0.5 89.0 50.9 148 66 92.2 0.6 92.8 50.5 149 81 22.5 8.1 30.6 15.4 150 29 27.5 8.0 35.4 22.5 152 72 58.5 4.0 63.0 44.9 153 96 73.5 5.2 78.7 29.2 154 132 108.8 0.1 108.8 37.0 155 99 55.0 0.7 55.8 28.8 156 70 39.3 13.1 52.5 35.5 157 36 37.7 8.3 46.0 37.3 158 66 44.4 5.5 49.8 34.9

23

School TEA #

Number of Students

Range # of Hours

Minimum # of Hours

Max # of Hours

Mean # of Hours

159 154 72.8 0.5 73.3 40.5 160 44 26.2 3.3 29.5 16.0 161 106 26.5 1.2 27.6 14.5 162 107 52.7 7.6 60.3 34.3 163 73 91.3 6.8 98.1 41.9 164 118 35.9 18.4 54.3 32.1 166 107 48.7 0.8 49.5 37.4 167 132 164.0 4.9 168.9 58.5 168 102 78.9 6.1 85.0 34.9 169 92 73.3 14.9 88.2 25.4 170 106 41.2 0.3 41.5 17.5 171 130 60.8 13.2 74.0 37.1 172 88 70.7 11.8 82.5 38.3 173 75 12.6 0.5 13.1 7.7 174 34 21.1 25.2 46.3 35.1 175 95 55.4 0.1 55.5 40.6 176 98 87.7 0.4 88.1 40.3 177 75 15.3 10.1 25.4 20.4 178 54 43.4 0.7 44.1 26.2 180 70 55.7 3.9 59.6 42.6 181 64 59.9 14.1 74.0 53.7 182 138 58.9 6.5 65.4 22.1 183 85 30.4 1.3 31.7 15.5 184 41 22.1 30.8 52.9 36.9 185 53 53.2 2.4 55.6 41.2 186 63 63.6 9.6 73.2 36.9 187 99 48.7 2.5 51.2 30.5 188 132 63.1 8.6 71.7 40.2 189 55 84.5 2.0 86.5 35.2 190 80 40.1 8.7 48.8 37.2 191 100 51.2 1.1 52.3 22.7 192 67 79.1 4.5 83.6 48.4 193 81 65.7 2.2 67.9 30.3 194 84 60.1 2.3 62.4 36.6 195 60 37.4 8.5 45.8 27.4 196 50 26.0 5.0 31.0 12.5 197 90 22.7 2.2 25.0 10.1 198 78 49.9 4.1 54.0 22.0 199 95 40.3 0.7 41.0 30.9 200 61 43.9 2.9 46.8 30.8 201 87 148.0 0.6 148.5 37.7 202 75 18.8 8.3 27.1 15.3 203 67 53.7 4.9 58.6 37.6 205 122 55.6 0.8 56.3 26.9 206 73 44.2 0.3 44.5 35.7 207 111 89.9 0.1 90.0 35.0 208 143 104.9 0.3 105.1 49.4 209 107 57.2 0.9 58.1 43.6 210 126 51.8 1.4 53.3 39.0 211 5 22.3 0.3 22.6 9.8 212 42 36.8 16.1 52.8 25.6 213 66 43.9 2.4 46.3 25.4

24

School TEA #

Number of Students

Range # of Hours

Minimum # of Hours

Max # of Hours

Mean # of Hours

214 23 28.9 7.1 36.0 27.5 215 66 43.3 7.9 51.3 22.6 216 127 127.6 0.7 128.3 35.5 218 164 58.1 0.1 58.2 33.9 219 58 44.2 15.1 59.3 40.0 220 58 40.6 1.2 41.7 25.1 222 111 75.9 6.6 82.5 38.4 224 44 74.4 10.6 85.0 51.0 225 92 55.9 3.8 59.6 41.7 226 79 95.6 0.4 96.0 41.0 227 24 75.8 0.6 76.4 44.3 228 46 38.3 3.2 41.5 13.8 229 129 33.4 14.8 48.2 31.3 230 65 48.1 7.5 55.6 38.8 232 115 15.8 2.0 17.7 8.4 233 72 91.0 13.0 104.0 36.0 234 112 54.8 1.4 56.2 31.6 235 56 30.2 1.8 32.1 19.2 236 73 50.1 4.8 55.4 30.4 237 111 51.9 0.8 52.7 38.5 239 73 24.3 0.3 24.6 14.2 240 106 43.8 0.9 44.7 23.8 241 2 3.9 12.3 16.2 14.3 250 83 68.9 0.1 69.0 26.7 260 64 40.3 24.5 64.8 41.9 263 52 57.2 1.1 58.3 36.2 264 141 49.8 0.3 50.2 28.6 265 38 41.6 10.4 52.0 40.2 266 88 57.4 3.7 61.1 24.9 268 92 52.1 10.1 62.2 39.4 269 47 102.8 4.4 107.2 38.0 271 148 43.7 0.7 44.3 16.8 272 91 50.0 22.6 72.5 49.4 273 89 56.7 1.0 57.7 30.1 274 132 56.7 3.9 60.1 26.6 275 40 80.9 0.9 81.8 64.2 276 121 73.6 1.2 74.8 49.6 277 71 44.0 8.0 52.0 29.4 278 78 45.9 1.8 47.6 28.9 279 90 69.9 8.2 78.1 33.1 280 171 56.4 7.0 63.4 43.5 281 56 38.8 1.6 40.4 19.7 283 59 37.7 4.4 42.1 29.2 284 136 44.9 0.8 45.6 27.1 286 72 48.2 0.8 49.1 19.3 287 100 44.3 0.7 45.0 25.0 288 23 24.1 8.9 33.0 19.6 289 75 11.3 7.2 18.5 13.2 301 123 43.7 0.3 44.0 18.3 304 97 47.6 1.0 48.6 25.5 305 86 37.3 6.0 43.4 23.0 997 21 40.0 0.4 40.3 17.0

25

Documents

EA12-514-2 Evaluation of the Reasoning Mind Mathematics Program