Upload
mcgraw-hill-education
View
234
Download
0
Embed Size (px)
Citation preview
ResearchRonald C. Martella, Ph.D.
Nancy E. Marchand-Martella, Ph.D.
confidential and proprietary
confidential and proprietary 2
confidential and proprietary 3
confidential and proprietary 4
confidential and proprietary
Research Base:White Papers
5
confidential and proprietary 6
confidential and proprietary 7
confidential and proprietary 8
confidential and proprietary 9
confidential and proprietary 10
confidential and proprietary 11
confidential and proprietary 12
confidential and proprietary 13
confidential and proprietary
DATA
14
confidential and proprietary
Learner Verification Study
15
• Main purpose to provide feedback using pre-publication version of program.
• Imperfect implementation—full program not completed, starts and stops throughout, changes in program and technology improvements.
• Even with focus of study, positive results were seen across diverse settings—non-special education, special education, after-school.
• Social Validation: Teachers and students liked program and reported effects.
• Statistically significant results seen on TOWRE-2.• Positive results on GRADE; statistically significant effects for Grade 5
Total and Comp; statistically significant effects for Vocab at Washington site.
• Increasing trend on DIBELS.
confidential and proprietary 16
Background • Designed by Department of Academic Research as an
Exploratory study• Incorporates 141 students across grades 3 through 8
attending schools in GA, SC, TN, TX, and WA• All students performing at least two years below grade level
in reading • Demographics
– Higher percentage of males (58%) than females – 65% identified as Caucasian– Over half identified with special needs, most of these SLD– 86% free/reduced lunch
confidential and proprietary 17
Implementation
• **Study utilized pre-published version of materials **
• Staggered implementation– Late September: SC– October: GA, TN, WA– Early November: TX
• Setting– GA and TX in non-special education classrooms– TN and SC in special education classrooms – WA in an ‘after-school’ setting
confidential and proprietary 18
Implementation, cont.• Consultants conducted on-site visits to monitor
implementation throughout the study– Strong implementation for SC, TN, and TX
• All program components; 90 – 120 minutes per day for instruction– Slight deviations observed in GA
• All program components; 90 – 120 minutes per day for instruction – Partial implementation in WA
• Digital and Print; 45 minutes – 1 hour for instruction– Initial difficulties with technology/connectivity in WA which
impacted implementation of the Digital Experience
confidential and proprietary 19
Social Validity Questions of Interest• What are teachers’ and students’ attitudes and
perceptions of the program?– Online survey for teachers, end of study– Student interview, end of the study
• What effect does FLEX Literacy have on students’ perceived reading skill and interest in reading?– Pre- and post-program administration of a 9 item
survey
confidential and proprietary 20
Results
• Teacher Survey
confidential and proprietary 21
Teacher Survey
• All teachers (100%) reported that students were engaged during the Print Experience , Digital Experience, and the Project Experience.
• All teachers (100%) reported that routines were easy to follow, students felt successful, and content standards were addressed in the Print Experience.
• All teachers (100%) indicated that the Teacher’s Guide is well-organized, students felt successful, and content standards are addressed in the Project Experience.
confidential and proprietary 22
Teacher Survey, cont.
• Responses on items addressing the Digital Experience varied across sites and largely reflect issues with technology/connectivity that have since been corrected.
• Despite this, the majority of teachers cited the Digital Experience and the opportunity for students to work at their own pace as particular strengths of SRA FLEX Literacy.
confidential and proprietary 23
• What four words would you use to describe SRA FLEX Literacy?
• Individual• Challenging• High-expectations• Collaborative• Intense• Flexible• Thought provoking• Life time skills• Interesting• Relevant• Deliberate• Engaging• A very different format
confidential and proprietary 24
• 100% of teachers indicated they would recommend SRA FLEX Literacy to a colleague.
• Yes. I already have. Teachers are looking for an individualized program for students struggling in reading and this program (mainly the digital) is equivalent to a student who is sitting one on one with a reading tutor. The teacher knows exactly where the student is and 30 students can work at once on their level, rather than hiring a tutor for 30 minutes of small group once or twice a week. More students benefit from individualized instruction resulting in less being labeled with a disability.
confidential and proprietary 25
Results
• Student Interview
confidential and proprietary 26
How did SRA FLEX Literacy help you become a better reader?
• 87% of students reported that they felt SRA FLEX Literacy helped them become a better reader.
Read/Pronounce Words
Read Aloud Asks Questions Other0
10
20
30
40
50
60
70
80
confidential and proprietary 27
90% of students’ responses were positive, indicating they enjoyed the program or thought it improved their reading skills.
• What would you tell other students about SRA FLEX Literacy?Fun/Cool Read Better Easy/Hard DK
0
10
20
30
40
50
60
confidential and proprietary 28
59%
16%
7%
6%
12%
Digital ExperiencePrint ExperienceProject ExperienceLearningDK
What is the best thing about SRA FLEX Literacy?
confidential and proprietary 29
• The majority of students responded that they enjoyed the Project Experience.– Roughly one-third reported that their favorite part
was “working in teams”– Other responses varied considerably, but included
“researching,” “coloring/creating,” “making PowerPoint slides,” and simply, “all of it.”
confidential and proprietary 30
Results
• Student Survey
confidential and proprietary 31
• Short, nine-item survey administered before and after participation in FLEX
• Designed to capture student perceived reading skill and interest in reading– I am a good reader– I like to read at home
• Response categories ranged from Yes! to Not At All
confidential and proprietary 32
All Students
• After participation in FLEX, more students selected Yes!, or Mostly in responding to the statements:– I am a good reader (15 % increase)– When reading, I can easily read all of the words
(14% increase)– When reading, I know what all of the words mean
(10% increase)
confidential and proprietary 33
By Subgroup
• I am a good reader– Grade 5 (25% increase)
• I can read as well as other students– Grade 5 (8% increase)
• When reading, I know what all of the words mean– Grade 5 (20% increase)
• I would enjoy receiving a book as a present– Grade 8 (25% increase)
• When reading, I can easily read all of the words– Grade 8 (33% increase)
confidential and proprietary 34
Results
• Achievement Measures
confidential and proprietary 35
Questions of Interest, cont.
• Association with basic reading, vocabulary, and comprehension skills– Pretest and Posttest using published, norm-
referenced tests
• Association with oral reading fluency skills– Repeated administrations of the Oral Reading
Fluency (ORF) subtest of the DIBELS
confidential and proprietary 36
Quantitative Measures
• Test of Word Reading Efficiency, Second Edition (TOWRE-2)– 2 subtests: Sight Word Efficiency and Phonemic Decoding
Efficiency • Group Reading Assessment and Diagnostic Evaluation
(GRADE)– Comprehension and Vocabulary Composites along with a
Total Score• Oral Reading Fluency (ORF) subtest from the DIBELS
– One-minute reading assessment of reading fluency (words per minute)
confidential and proprietary 37
TOWRE-2 and GRADE
• Group administered• Contain parallel forms for pretest and posttest• High reliability/validity• Give norms for off-grade level assessment• Provide standard scores/are norm-referenced– Allow for comparison across various tests and groups– Can be manipulated for use in statistical tests– Easily interpretable – Often utilized in the identification of students with
disabilities
confidential and proprietary 38
A word about “statistical significance”
• Indicates whether or not there is a difference between groups simply due to chance
• Is impacted in part by sample size– If treatment effects are small (and present), larger
samples are more powerful in detecting them.– With smaller samples, treatment effects (if
present) must be relatively larger to be detected.
confidential and proprietary 39
…and Effect Size• Describes the magnitude of the difference• Represents the average difference between groups,
expressed in standard deviation units – For example, an effect size of .30 means “three-tenths of a
standard deviation”• Effect size is descriptive for that particular group• Cohen’s d
– ‘Small’: .20– ‘Medium’: .50– ‘Large’: .80 and greater– For educational research, .25 is considered minimal level
confidential and proprietary 40
Learner Verification Study (continued)
• Effect sizes are measures of magnitude—i.e., how large are the effects.• Effect sizes were positive. Note that .25 effect size is the minimal level for
educational significance (Adams & Engelmann, 1996); effect sizes over 1.00 are relatively rare in education.
Some example results:Grade 5
Effect size = .42 on GRADE Comp Effect size = .38 on GRADE Total
Grade 8Effect size = 1.49 on TOWRE-2
TennesseeEffect size = 1.37 on TOWRE-2
WashingtonEffect size = .73 on TOWRE-2 Effect size = .52 on GRADE Vocab
confidential and proprietary 41
All Students• Statistical significance
achieved for TOWRE-2, effect size is small.
• Non-statistically significant increases for the GRADE.
• For all measures, pretest and posttest scores fall within the Low Average range.
confidential and proprietary 42
Student Performance by Grade Level
• Grades 3, 4, 6, and 7– Increases noted
• Grade 5– Statistically significant increases for GRADE
Comprehension (d = .42, small) and GRADE Total (d = .38, small effect).
• Grade 8– Statistically significant difference for TOWRE-2 (d =
1.49, large effect)
confidential and proprietary 43
Grade 5• Statistical significance
achieved for GRADE Total and GRADE Comp; effect size is small.
• Non-statistically significant increases for the TOWRE-2 and GRADE Vocab.
• The average comprehension score increased from the Low range to the Low Average range.
confidential and proprietary 44
Grade 8• Statistical significance
achieved for TOWRE-2,; effect size is large.
• Stable performance for the GRADE.
• The average TOWRE-2 score increased from the Low Average range to the Average range.
confidential and proprietary 45
Student Performance By Site
• Increases shown in Georgia, South Carolina, and Texas
• Tennessee– Statistically significant increases for the TOWRE-2 (d = 1.37,
large effect)
• Washington– Statistically significant increases for the TOWRE-2 (d = .73,
medium effect) and GRADE Vocabulary (d = .52, medium effect).
confidential and proprietary 46
Tennessee• Statistical significance
achieved for TOWRE-2; effect size is large.
• Non-statistically significant increases for the GRADE.
• The average TOWRE-2 score increased from the Low Average range to the Average range.
confidential and proprietary 47
Washington• Statistical significance
achieved for TOWRE-2; effect size is medium.
• Statistical significance achieved for GRADE Vocabulary; effect size is medium.
confidential and proprietary 48
SLD Category• Statistical significance
achieved for the TOWRE-2, effect size is large.
• Non-statistically significant increases are noted on the GRADE Total and GRADE Comprehension.
• Stable performance on GRADE Vocabulary.
• The average TOWRE-2 score increased from the Low range to the Low Average range.
confidential and proprietary 49
Results
• DIBELS• Oral Reading Fluency
confidential and proprietary 50
Oral Reading Fluency
• General upward trend across grade levels• The average gain in words per minute by
grade level ranged from a 4% to a 59% increase
• Strongest results for grade 5
confidential and proprietary 51
GRADE 5ORF
• At baseline, students read an average of 63 words per minute.
• Students read an average of 100 words per minute on the Spring benchmark.
• Students gained, on average, 37 words per minute. This reflects a 59% increase over baseline levels.
confidential and proprietary 52
Results
• LEXILE Measures
confidential and proprietary 53
confidential and proprietary 54
confidential and proprietary 55
Because the total number of lessons students receive may have an effect on their scores, the correlation between Lexile® growth and percentage of lessons completed for the program was calculated.
The correlation between Lexile® growth and percentage of yearly lessons completed for the program was statistically significant, r = .559, n = 69, p < .001 (two-tailed). This correlation indicates there was a statistically significant, positive relationship between Lexile® growth and the percentage of yearly lessons completed.
confidential and proprietary 56
Summary
• The majority of students and teachers reported that they enjoyed using SRA FLEX Literacy, would recommend it to others, and felt successful while using it.
• There were some positive shifts in perceived reading skill and interest in reading, for several groups.
• Pre-post program differences on the TOWRE-2 were determined significant for several groups.
• Oral Reading Fluency scores show an upward trend, over time.
confidential and proprietary
Washington School District Study
57
• Large study in Eastern Washington School District.• 55%-75% free and reduced price lunch.• Primary purpose to determine student academic gains.• Full program implemented for entire year.• Up to 150 students across Grades 6, 7, and 8.• Intervention students.• Multiple measures including Washington State MSP (Measurements of
Student Progress) test results, AIMSWeb, TOWRE-2, GRADE, in-program assessments.
• Possible math test results on MSP.• Compare with students who are not in intervention program but with similar
demographics.