55
0 antrell, Carter & Rintamaa, 2012 C 2012 Susan C. Cantrell, Janis C. Carter, and Margaret Rintamaa Collaborative Center for Literacy Development University of Kentucky March, 2012 Striving Readers Cohort II Evaluation Report: Kentucky

Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

  • Upload
    others

  • View
    0

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

0antrell,Carter&Rintamaa,2012

C

2012

SusanC. Cantrell,Janis

C. Carter,and

MargaretRintamaa

CollaborativeCenterfor

LiteracyDevelopment

UniversityofKentucky

March,2012

StrivingReadersCohortIIEvaluation Report:

Kentucky

Page 2: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving
Page 3: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

1SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

Acknowledgments

Theauthorsgratefullyacknowledgethecontributionsofthemanyindividualswhofacilitatedthisevaluation. ThisevaluationcouldnothavebeensuccessfulwithoutthesupportofCindyParker,RobinHebert,MaryRuddandMonicaOsbornefromtheKentuckyDepartmentofEducation(KDE)andadministratorsandstaffinthenineStrivingReadersschools. WewanttothanktheStrivingReadersinterventionistswhoworkdiligentlywithstudentsandteacherseachdayandtowhomweturnregularlyforinformationtofurtherthiswork. Furthermore,wegratefullyacknowledgethemanystudentswhoparticipatedinthisstudy.

Additionalmembersoftheevaluationteamwhodeservecreditfortheircontributions

tothisreportincludethefollowingindividualstowhomweareimmenselygrateful:

DataAnalystDeepshikhaSigdel

ResearchAssistantsLauraDudneyAmandaGoodwinJessicaGreweMelissaMurphyJenniePahl

StaffSupportElisha ComerJudyJohnson

Finally,weextendourgratitudetothestaffoftheCollaborativeCenterforLiteracyDevelopment(CCLD)fortheirongoingsupportofthiswork,AbtAssociatesInc.forthetechnicalassistancetheyprovidedduringthestudy,andtheU.S.DepartmentofEducationforfundingthisproject.

Page 4: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

2SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

TableofContents

ExecutiveSummaryofFindings:ImplementationandImpact……………………………………. 4KCLMTargetedIntervention…………………….……………………………………………………………….. 5

Implementation….……………………………………………………………………………………………….. 5Impacts.…………………………………………………………………………………………………………….... 5

IntroductionandStudyBackground…………………………………………………………………………….. 5DescriptionoftheInterventionModel….…………………………………………………………………… 5FrameworkoftheSupplementalLiteracyInterventionClass..………………………………….. 6ComponentsofKCLM.……………………………………………………………………………………………… 7TargetedStudents…………………………………………………………………………………………………….. 8SelectionProcessforInterventionists……………………………………………………………………….. 9DesiredCharacteristicsoftheInterventionClassroom……………………………………………… 9LogicModelforKentuckyCognitiveLiteracyModelIntervention………………………………. 10PlannedTrainingModel…………………………………………………………………………………………….. 11PlannedClassroomInstructionalModel…………………………………………………………………….. 12PlannedExperiencesforControlStudents…………………………………………………………………. 12UnitPlanningTemplate…………………………………………………………………………………………….. 13KeyEvaluationDesignFeatures…………………………………………………………………………………. 14

EvaluationoftheImplementation…………………………………………….….……………………………… 14SummaryoftheDesignoftheImplementationStudy.…..……………………………………….…. 14ImplementationDataCollectionandAnalysis..…………………………………………………………. 15

SummerTraining…………………………………………………………………………………………………. 15CoachMentoring…………………………………………………..……..……………………………………… 15ClassroomInstruction…………………....………………………..……………………………………….... 15

KCLMImplementationResults……………………………………..…………………………………………….. 16CharacteristicsofInterventionists……………………………………………………………………..……… 16ImplementationofProfessionalDevelopmentModel…………………….………………………… 16

Interventioniststraining….………………………………………………………………………………….. 16Professionaldevelopmentinputs………………………………………………………………………… 17Professionaldevelopmentforadministrators……………………………………………………… 18

ImplementationofClassroomModel……………………………………………………………………….. 18Classsize,intensity,andduration……………………………………………………………………….. 18Classroomimplementationresults……………………………………………………………………… 18Experiencesforcontrolstudentsduringinterventionperiod………………………………. 20Additionalreadingprograms………………………………………………………………………………. 20Implicationsforimpactanalysis…………………………………………………………………………… 20

EvaluationofImpact……………………………………….…………………………………………………………… 21StudyDesign……………………………………………………………………………………….…………………….. 21

Page 5: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

3SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

Samplingselectionprocess…...……………………………………………………….……………………. 21Samplesize……………….……………………………………………………………………………………….. 21ConsortDiagramReadingAchievement…………………………………………………..………… 22ConsortDiagramWritingAchievement……………………………………………………………… 23ConsortDiagramStudentEfficacyandMotivation….…………………………………………… 24MetacognitiveAwarenessofReadingStrategiesInventory(MARSI).…………….…….. 25AdolescentMotivationSurvey………………...…………………………………………………….……. 25

SummaryofAnalyticApproach………………...………………………………………………………………. 26Level-1HLM:StudentLevel………………..……………………………………………………………….. 26Level-2HLM:SchoolLevel………………………………………………………………………..………… 26

DescriptionoftheFirstYearSample………….……………………………………………………………… 26ImpactsonStudents………………..………………………………………………………………………………… 27

DiscussionandConclusions……………………………………………………………..…………………………… 29References………………………….………………………………………………………………………………………. 30

APPENDICESA:SummaryofAnalyticProcedures………………………………………………………………………….. 36B:StudyMeasures…………………………………………………………………………………………………….. 38

StudentSurvey…………………………………………………………………………………………………….. 39TeacherObservationProtocol……………………………………………………………………………… 48

EXHIBITS

1:ConsortDiagramReadingAchievement…………………………………………………..……………. 222:ConsortDiagramWritingAchievement…………………………………………………………………. 233:ConsortDiagramStudentEfficacyandMotivation….…………………………………………….. 24

FIGURES

1:ComponentsofKCLM……………………………………………………………………………………………. 72:KCLMInterventionLogicModel…………………………………………………………………………………… 103:UnitPlanningTemplate…………………………………………………………………………………………. 13

TABLES

1.1FrameworkoftheSupplementalLiteracyInterventionClass…..…………………………… 61.2PlannedPDActivities…………………………………………………………………....……………………. 112.1 Contentanddeliveryofprofessionaldevelopmenttraining(KCLM)……….…………… 172.2 MostWidelyImplementedModelComponents……………………………………….……….. 192.3 MeanScoresbyTeacherforQualityofKCLMcomponent....................................... 202.4 InterventionandControlStudentDemographics(andProportions)....................... 272.5 ImpactoftheTargetInterventiononStudentReadingAchievement,WritingAchievement,

StrategyUseSelfEfficacy,andMotivation…….……………………………… 28

Page 6: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

4SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

StrivingReadersCohortII: Kentucky

ExecutiveSummaryofFindings:ImplementationandImpact

ThisStrivingReadersevaluationexaminedtheimpactofatargetedinterventionforstrugglingadolescentreadersinparticipatingschools. Thisstudywasconductedinninehighschoolsinnineschooldistrictsservinglargepercentagesofat-riskstudentsinKentucky. ThetargetedinterventionforstrugglingreaderswastheKentuckyCognitiveLiteracyModel(KCLM)developedbytheKentuckyDepartmentofEducation.

TheeffectivenessoftheKCLMinterventionwasdeterminedthrougharandomized

controlfieldtrialutilizingatreatmentandcontrolgroupdesign.TheKCLMwasasupplementtotheregularcurriculumwhereinstudentsinthetargetedinterventionparticipatedinareadingclassinplaceofanelectiveaspartoftheirregularschoolday. Thecontrol-groupconditionwas“businessasusual,”whereinstudentsinthecontrolgrouptakearegularelectivesuchasband,theaterarts,civics,orphysicaleducation. ThisstudyexaminedtheimplementationofKCLManditsimpactonstrugglingninth-gradestudents’readingandwritingachievement,self-efficacywithreadingstrategies,andmotivationforreading.

InthisStrivingReadersproject,eachschoolemployedaninterventionteacherwhowas

responsibleforteachingthetargetedinterventiontostrugglingreaders.Theimpactresearchquestionsthatmotivatedthestudydesignandanalysisplanare:

• WhatistheimpactofKentucky’sCognitiveLiteracyModel(KCLM)onthereading

achievementoflow-achievingreaders?• WhatistheimpactofKCLMonthewritingachievementoflowachievingreaders?• WhatistheimpactofKCLMontheperceivedreadingstrategyuseoflowachieving

readers?• WhatistheimpactofKCLMonthemotivationandengagementoflowachieving

readers?Inaddition,thefollowingimplementationquestionswillbeanswered:

• What is the state-level implementation of theprofessional development training and

supportforinterventionteachersintheproject?• WhichcomponentsofKCLMwereimplementedmostfrequentlybyteachersin

classrooms?• WhatwasthequalityofKCLMimplementationinclassrooms?

Page 7: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

5SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

KCLMTargetedIntervention

ThisstudyexaminedboththeimplementationandimpactoftheKCLMoverthecourseofthefirstyearoftheStrivingReadersproject.

Implementation. Duringtheyearoftheproject,KCLMteachersparticipatedintrainingandon-sitesupport,andtrainingwasprovidedforschooladministrators. Duringtheyear,KCLMteacherswereprovided11daysprofessionaldevelopmenttrainingandupto66hourssupportfromvisitsbyKDEliteracystaff. Schooladministratorswereprovidedonedayoftrainingandatleasttwosupportmeetingsregardingtheinterventionfortheyear. Overall,participationintheprofessionaldevelopmenttheKCLMteachersandadministratorswashigh,with100%ofteachersandadministratorsparticipatingfullyinthetrainingeitherthroughtheformaltrainingdatesorthroughmakeuptraining. Classroomimplementationfidelitywasmeasuredthroughclassroomobservations. ObservationsindicatedthatteachersimplementedsomecomponentsoftheKCLMmodelmorereadilythanothercomponents.Theinterventionwasimplementedwithadequatequalityin3of9classrooms.

Impacts. Thisstudyusedhierarchicallinearmodeling(HLM)analysestomeasuretheimpactsoftheKCLMonninth-grade(highschool)students’readingachievement,writingachievement,perceivedstrategyuse,andmotivation.InthisStrivingReadersstudy,therewerenoimpactsonstudents’readingachievementasmeasuredbytheGroupReadingAssessmentandDiagnosticEvaluationandnoimpactsonwritingachievementasmeasuredbytheKentuckyStateWritingAssessment. Studentsurveyresultsindicatedsignificanteffectsoftheinterventiononparticipatingstudents’self-efficacyforstrategyuseandonstudents’readingmotivation.

IntroductionandStudyBackground

DescriptionoftheInterventionModel

ThetargetedinterventionfortheKentuckyprojectwastheKCLM,developedbytheKentuckyDepartmentofEducation(KDE). ThepurposeoftheKCLMwastoassiststudentswhoweresignificantlybehindgradelevelinreadingbyprovidingthemwiththesupportstobesuccessfulinlearningacrossthecurriculum. Table1.1illustratesthestrandsoftheintervention.

Page 8: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

6SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

Table1.1

FrameworkoftheSupplementalLiteracyInterventionClass

Motivationand

Engagement• Thematicinstructionwithproject/inquiry-basedlearningproduces

studentswhoaremorefullyengagedandmotivatedtolearn.• Literacyisasocialaccomplishment(Bloome,1986;Dyson,1992).

Strategic

Processing• Strategy—adeliberatecognitiveprocessofselecting,enacting,

monitoringandregulatingbehavior. Includescomprehensionstrategy

instructionandfoundationalreadingskills.• Skill—amentalactivitythatcanbeappliedtospecificlearningsituations.• Metacognition—keytostrategicprocessingbecauseitenablesstudents

tomonitorprogresstowardachievingtheirgoals(Flavell,1979).Instructional

Strategiesfor

ContentLearning

• RobertMarzano’scharacteristicsofeffectivevocabularyinstruction• Marzano,etal.strategiesforlearningsuchascues,questions,and

advanceorganizers;non-linguisticrepresentations;identifyingsimilaritiesanddifferences;summarizingandnotetaking

Communication

Skills• Includesreading,writinganddiscussionoutcomesthataddressavariety

ofapproachestoessentialquestionsandtexts.• Writinghelpsreadersclarifymeaningandprovidesopportunitiesfor

authenticengagementandcommunication.• Exchangingideas,especiallythroughextendeddiscussionofmeaning

andinterpretationoftext,isessentialtoalearningcommunity.

Itisimportanttonotethatwhilethesearecategorizedintospecificstrands,theyalsoareembeddedacrossstrandstointegrateatotalliteracyexperienceforstudents.ThefourcomponentsoftheKCLMframeworkweretiedtogetherbycontent-relatedthemessuchassuccess,theenvironment,andproblemsolving.

Page 9: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

7SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

Figure1showsthemajorcoursecomponentsandspecificdimensionsofeachcomponent.

SupplementalInterventionModelComponentsMotivation&Engagement

StrategicProcesses InstructionalStrategies

Communication

Theme-basedInstructionConnectionsbetweenlearningactivitiesandrealworldissuesVariedinstructionalformatStudentpriorknowledge,interestandbackgroundusedindeterminingcontentTechnologyisusedtofacilitatelearningGoalsettingandprivatefeedbackFocusonproblem-solvingprocessesAutonomyforlearningandmeaningfulchoices

Explicitcomprehensionstrategyinstruction(modeling,explanation,practice,andreflection)Explicitinstructioninfoundationalreadingskills

Instructionaltoolsforcomprehensionandvocabularylearning.Vocabularyprocessingthroughvisual,auditory,physicaland/oremotionalexperiences;opportunitiestousetheirownwordsornon-linguisticrepresentationstodefinenewwords;teacherexplanationsandexamplesofnew,keytermsAfocusonsummarizingandidentifyingsimilaritiesanddifferencesHigherlevelquestioning

Pre-,during-,andpost-textbaseddiscussionstrategiesWritingtolearnactivitiesExplicitinstructioninwritingstrategiesExplicitinstructioninfoundational writingskills

Figure1:ComponentsofKCLM.

Page 10: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

8SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

SRCohortIIEvaluationReport:Kentucky

IntheKentuckyproject,ninth-gradestudentswhoscoredtwogradelevelsormorebelowgradelevelinreadingreceivedaminimumof225minutesperweekandamaximumof375minutesperweekofsupplementalreadinginstructioninatargetedinterventionclasstaughtbyaninterventionteacher. Studentswereplacedinthiscourseinadditiontotheirregularreading/languageartsclassesforanentireschoolyear.

Overthecourseoftheproject,theprofessionaldevelopmentmodelforthetargetedinterventionincludedsummerandfollow-uptrainingsandon-sitesupportfromKDEliteracystaff.Tolearnhowtoimplementthetargetedintervention,teachersparticipatedinasummerworkshop,whichwasledbyKDEliteracystaff. Duringtheschoolyear,thetrainersledtheteachersinfollow-upworkshops. Acrosstheproject,KCLMteachersreceived11daysofworkshoptraininginthetargetedinterventionintotal. Tosupporttheirongoinglearninganddevelopment,teachersalsoparticipatedinsitevisitsandregulardistancesupportbyKDEliteracystaff.

Theprofessionaldevelopmentmodelincludedtrainingandsupportforadministrators,aswell. SchooladministratorsattendedaonedaymeetinginthesummertolearnabouttheKCLMmodelandthewaysinwhichtheschoolsshouldsupporttheintervention. Additionally,KDEliteracystaffparticipatedinon-sitemeetingswithadministratorsuptosixtimesduringtheschoolyear.Topicsatthosemeetingsincludedgrantrequirements,evidence-basedcomponentsofthetargetedintervention,schedulingissues,observationsofinterventionists,literacyleadership,literacyplanning,andmeetingtheneedsofstrugglingadolescentlearners. AdministratorsalsoreceivedKCLMupdatenewslettersfromKDEfourtimesduringthecourseoftheyear.TargetedStudents

Thisprojectwasdesignedtoservelowachievingreadersintheninthgrade. Inthespringoftheireighth-gradeyear,studentsinmiddleschoolsthatfeedintothenineparticipatinghighschoolsweregiventhemiddleschoolformofthespringGroupReadingAssessmentandDiagnosticEvaluation(GRADE). StudentsthatscoredanNCEof40orlowerontheGRADEweredefinedaslowachievingreaders,andwereplacedintheeligibilitypool.

Studentsinalldayresourceclasseswerenoteligible;allotherstudentswereeligible. Theevaluationteamdirectedthefacultyatallfeedermiddlesschoolstoidentifystudentsinalldayresourceclasses,andinstructedtheschoolsnottogivethespringGRADEtothesestudents. Inaddition,evaluatorsdirectedthefacultyatthenineparticipatinghighschoolstoidentifystudentsthatwereplacedinalldayresourceclassesaftertheyenrolledinninth-gradethisfallandaskedthemtoprovideevaluatorswiththenamesofthesestudentsforremovalfromthestudy.Finally,middleschoolssenthomepassiveconsentformswithalleighth-gradestudentswhowouldbetestedforparticipationinthestudy. Thestudywasdescribed,andparentsweredirectedtocontacttheevaluatorsiftheychosetonotallowtheirstudenttoparticipateinthestudy. Noparentdeclinedpermissionfortheirchild’sparticipationinthestudy.

Twothousandtwohundredfourstudentswerelistedontheschoolregistersheetsasenrolledatthefeedermiddleschools. Evaluatorsreceivedanadditionaltwenty-fivestudent

Cantrell,Carter,&Rintamaa(2012)

8

Page 11: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

9SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

GRADEsheetsthatwerenotontheenrollmentlists.Therewere637studentsthatmetthestudy’scriteriononthespringGRADEforeligibility.SelectionProcessforInterventionists

Interventionteacherswererecruitedandhiredbyindividualschools. Advertisementsfortheinterventionteacherpositionincludedthefollowingcriteria: experiencedclassroomteacher,respectedbyfacultyandadministration;familiaritywithand/orinterestininterdisciplinaryandproject-basedlearning;willingnesstolearnandapplynewskillsandknowledge;planning/reflectingskills;strongleadershipability;adaptabilityandproblemsolvingskills;presentationskills;collaborationskills;abilitytomediatebetweentheschoolandcommunityorganizations;andpersonalcommunicationskills.DesiredCharacteristicsoftheInterventionClassroom

Classesweretobenolargerthantwentyninth-gradestudents. Theinterventionclasswastomeetdailyforatleast45minutesfortheentireyear.

Page 12: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

10SRCohortIIEvaluationReport:Kentucky

SRCohortIIEvaluationReport:Kentucky

Figure2:KCLMInterventionLogicModel Cantrell,Carter&Rintamaa,2012

10

Page 13: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

SRCohortIIEvaluationReport:Kentucky

PlannedTrainingModel

Duringthesummerof2010,theinterventionteachersfromeachschoolweretoreceivefivedaysofinitialtrainingonthecorestrands. ThistrainingwastobedevelopedandprovidedbytheKentuckyDepartmentofEducationliteracystaff. Schooladministratorswererequiredtoattendonedayofsummertrainingtolearnaboutthegoalsoftheinterventionandtheexpectationsforsupportoftheinterventionistandstudents. Bytheendoftheinitialtraining,theinterventionistsweretocreateaninstructionalplanfordirectstrategyinstruction,andintegrationofstrategiesintostandards-basedunitsofstudy. Eachparticipantwastoleaveunderstandinghowtoteach,use,andassesswithintheKCLMframework,andwithaplantobeginimmediateimplementationasthe2010-2011schoolyearbegan.

Throughouttheyearofimplementation,theinterventionistsweretoreceiveon-going

trainingandsupportthroughface-to-facemeetingsandanonlineKentuckyVirtualSchools(KYVS)learningcommunity,whichwastoincludebookstudies,Webinars,adiscussionforum,andrepositoriesforsharingresources.KDEliteracystaffplannedsitevisitstotheschoolsinthefallandagaininthespringtoprovideon-sitecoachingandsupportforupto50additionalhours.

Administratorsweretoengageinprofessionaldevelopmentfocusingontheguidefrom

theNationalAssociationofSecondarySchoolPrincipals,CreatingaCultureofLiteracy:AGuideforMiddleandHighSchoolPrincipalsaswellasaKDEDVDresource,LiteracyLeadership:StoriesofSchoolwideSuccess.Thissharedleadershipnetworkwasintendedtohelpadministratorsandcoachesdevelopacloseworkingrelationshipcriticalinsupportinginstructionalimprovementefforts.

Table1.2

PlannedPDActivities

PDhrs. Activities Date Attendees*40 Trainingoncorestrandsofintervention;develop

instructionalplanandunitsofstudy8 Trainingongoalsforinterventionandon

providingsupporttointerventionists50+ OngoingPD: Onlinelearningcommunity,

includingbookstudies,webinars,discussionforum,sharingnetwork

July2010 I

July2010 A

2010 I

16 Onsitevisits/coaching 2010 I

*I=Interventionist;A=Administrator

Cantrell,Carter&Rintamaa,2012

11

Page 14: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

SRCohortIIEvaluationReport:KentuckyPlannedClassroomInstructionalModel

TheKentuckyDepartmentofEducationcreatedaUnitPlanningTemplate(UPT)toassistinterventionteachersincreatingunitsofinstructionfortheinterventionclass. OnecomponentoftheUPTlistedthestrategiesandactivitiesthatcomprisetheKCLM.Teachersselectedstrategiestoteachineachunit.Figure3showsthestrategiesandactivitiesaslistedontheUPT.Teacherswerefreetoselectthestrategiestheytaughtbasedontheirassessmentofstudents’needs.Aspartoftheinterventionclass,studentsassignedtotheinterventionwilltaketheSAT-10online(reading)andTOWL-4(writing)diagnosticassessmentstwiceperyear.

Inaddition,KCLMisdesignedtoengagestudentsinreadingavarietyoflevel-appropriatetextsrelatedtoessentialquestion(s)foreachcontent-relatedunit,includingnonfiction,informationalandproceduraldocuments,andnarrativetexts.PlannedExperiencesforControlStudents

Expectationsforthecontrolstudentswere“businessasusual,”whereinstudentsinthecontrolgrouptookaregularelectivesuchasband,theaterarts,civics,orphysicaleducation.

Cantrell,Carter,&Rintamaa(2012)

12

Page 15: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

SRCohortIIEvaluationReport:Kentucky

Figure3:UnitPlanningTemplate

Cantrell,Carter&Rintamaa,2012

13

Page 16: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

SRCohortIIEvaluationReport:KentuckyKeyEvaluationDesignFeatures

Theevaluationisdesignedtomeasuretheimpactofthetargetedinterventiononstudentoutcomesteacherefficacy. Theimpactstudywasguidedbythefollowingresearchquestions:Theimpactresearchquestionsthatmotivatedthestudydesignandanalysisplanare:

• WhatistheimpactofKentucky’sCognitiveLiteracyModel(KCLM)onthereadingachievementoflowachievingreaders?

• WhatistheimpactofKCLMonthewritingachievementoflowachievingreaders?• WhatistheimpactofKCLMontheperceivedreadingstrategyuseoflowachieving

readers?• WhatistheimpactofKCLMonthemotivationandengagementoflowachievingreaders?

Studentoutcomemeasuresareasfollows:

• GroupReadingAssessmentandDiagnosticEvaluation(GRADE)• TheKentuckyStateWritingAssessment• MetacognitiveAwarenessofReadingStrategiesInventory(MARSI)• StudentMotivationSurvey

Inaddition,thefollowingimplementationquestionswereanswered:

• Whatisthestate-levelimplementationoftheprofessionaldevelopmenttrainingandsupportforinterventionteachersintheproject?

• WhichcomponentsofKCLMwereimplementedmostfrequentlybyteachersinclassrooms?

• WhatwasthequalityofKCLMimplementationinclassrooms?

EvaluationofImplementation

SummaryoftheDesignoftheImplementationStudy

Theresearchquestionsthatguidedtheimplementationstudyofthetargetedinterventionare:

• Whatisthestate-levelimplementationoftheprofessionaldevelopmenttrainingandsupportforinterventionteachersintheproject?

• WhichcomponentsofKCLMwereimplementedmostfrequentlybyteachersinclassrooms?

• WhatwasthequalityofKCLMimplementationinclassrooms?

Cantrell,Carter&Rintamaa,2012

14

Page 17: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

15SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

ImplementationDataCollectionandAnalysis

Summertraining.Duringthesummerof2010,theselectedinterventionteachersfromeachschoolreceivedfivedaysofinitialtrainingonthecorestrands. ThistrainingwasdevelopedandprovidedbytheKentuckyDepartmentofEducationliteracyconsultants.Attendancerecordswerekeptateachtrainingsession,andindividualteacherattendancewascomputedattheendofthetrainingintermsofpercentageofdaysattended.Schooladministratorswillberequiredtoattendonedayofsummertrainingtolearnaboutthegoalsoftheinterventionandtheexpectationsforsupportoftheinterventionistandstudents. Participationwillbeassessedandscoredforadequacyandfidelitythroughattendancerecordsprovidedbythedevelopers.

Researchassistantsattendedeachtrainingsessionandtookdetailedfieldnotesinfiveminuteintervals. AcodelistwasdevelopedrelatedtoContentandDelivery.ContentcodesrelatedtothekeycomponentsofKCLM.Deliverycodesrelatedtovarioustrainingformatssuchaswholegroup,smallgroup,discussion,andlecture.Toestablishreliability,threeresearchassistantsindependentlycodedonedayoffieldnotes.Agreementwas90%forcontentand90%fordelivery.Afterthecodingtheresearchassistantsagreedthatsomeofthecodesshouldbefurtherbrokendownforamoreaccuratedescriptionofwhatwasgoingonduringthetimeinterval.Researchassistantsandinvestigatorsdiscussedaddingadditionalcodesforcontentanddelivery.Thefollowingcodeswereaddedtocontent:Collaboration,Planning-working,andPlanning-discussion.Thefollowingcodeswereaddedtodelivery:Wholegroup-lectureandWholegroup-discussion.Theresearchassistantscontinuedindependentlycodingthenextfourdaysoftheinterventionteachertraining.Researchassistantsmetagaintocheckagreement.Agreementachievedforcontentwas95%and100%agreementfordelivery.

Coachingandmentoring.Interventionistsreceivedon-goingtrainingandsupportthroughanonlinelearningcommunitythatincludedbookstudies,Webinars,adiscussionforum,andrepositoriesforsharingresources.KDEconsultantsmadesitevisitstotheschoolsinthefallandagaininthespringtoprovideon-sitecoachingandsupport. Participationwasassessedandscoredforadequacyandfidelitythroughattendancerecordsprovidedbythedevelopers.

Classroominstruction.Implementationfidelityforthetreatmentconditionwasestablishedthroughclassroomobservationsusingastandardizedobservationprotocolforinterventionclasses. Readinginterventionteacherswereobservedtwiceduringtheyear. Thestandardizedobservationprotocolforinterventionteachersincludedtwocomponents:achecklistofessentialKCLMcomponentsandaqualityrubricforassessingteachers’implementationquality.Inthefall2010,tworesearchassistantsmettocreatealistofmodelcomponentsbasedonthetrainingsfromKDE. ThislistwassenttoKDEforvalidation. KDEaddedafewcomponentstomakethelistcomplete. Toidentifythemodelfeaturesthatweremostcriticaltoprogramsuccess,evaluatorsaskedthedeveloperstorankeachsetoffeaturesforeachcomponentinorderofimportance. Toconfirmtheserankings,evaluatorssoughtfeedbackfromthreeexpertscholarswhoconfirmedandelaboratedontheshortenedlistforeachfeature.TheleadevaluatorandKCLMtrainerviewedtwovideosofKCLMinstructionandindependentlycompletedtheprotocol.Then,theydiscussedtheirscores,reachedconsensusonscoring,andmademinorrevisionsto

Page 18: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

16SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

somewordingontheprotocol.Fortraininginusingtheprotocol,researchassistantsviewedonevideo,ratedtheinstructionusingtheprotocol,anddiscusseddisagreements. Toestablishinter-observeragreement,theresearchassistantsviewedthesecondvideoandindependentlycompletedaprotocol. AgreementwiththeleadevaluatoronthechecklistofkeyaspectsofKCLMcomponentsforthisprotocolwas74.4%.Thelargestareaofdisagreementwasrelatedtovocabularyinstructionunderthe“InstructionalStrategies”component. Theleadevaluatorprovidedadditionaltrainingaroundvocabularyinstructiontoclarifymisunderstandings.Agreementonthequalitydescriptorsforthesecondvideowas91.7%.

Itwasexpectedthatteacherswouldimplementsomeaspectofeachforthefourmodelcomponents(motivationandengagement,strategicprocesses,instructionalstrategies,andcommunication)duringeachclassperiodbutitwasnotexpectedthatteacherswouldimplementeveryaspectofeachcomponenteachclassperiod.Theprotocolyieldedinformationonwhichaspectsofeachcomponentwereimplementedduringobservationsandwhetherthequalityofimplementationwas(a)developing,(b)adequate,or(c)exemplary.Percentagesofobservationsthatincludedeachcomponentwerecomputed,andtheproportionofobservationsratedateachqualitylevelwasprovided.

KCLMImplementationResultsCharacteristicsofInterventionists

Nineinterventionistpositionswerefilledby11teachersduringtheyear. Interventionteacherswereselectedandhiredbyindividualschooldistrictsandmettheplannedcharacteristics,accordingtoschoolpersonnel. Atotalofnineinterventionistswerehiredinthesummerof2010. Duringthecourseoftheschoolyear,twointerventionistsleftandwerereplaced,foratotalof11interventionists.TheseinterventionistsimplementedtheKCLMtargetedinterventionswithinthenineschools.Oneinterventionistwasmaleand10werefemale.Allinterventionistswerewhite.Alloftheinterventionistshadamastersdegreeorhigher,andthree(27%)werecertifiedasreadingspecialists.Interventionistshadanaverageof12.9yearsofexperience.ImplementationofProfessionalDevelopmentModel

Interventionisttraining.Trainingwasprovidedtointerventionistsforfivedaysinthesummer,andthreedaysduringtheschoolyear. Table2.1showsthecontentamountanddeliveryoftheprofessionaldevelopmentmodel.

Page 19: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

17SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

Table2.1Contentanddeliveryofprofessionaldevelopmenttraining(KCLM)

Numberof Minutestotal

Percentoftraining

Content Housekeeping 70 2.77Overviewofgrant 220 8.71Collaboration 275 10.89Readingstrategies&Strategicprocessing 640 25.30Communication 175 6.93Motivationandbehaviormanagement 225 8.91Foundationalreading–basicskills 60 2.38Assessment 360 14.26Planningwork 280 11.09Planningdiscussion 115 4.55Projectbasedlearning 110 4.36

Delivery

FormatWholegrouplecture 790 31.23Wholegroupdiscussion 1035 40.91Smallgroup 340 13.44Individual 365 14.43

Professionaldevelopmentinputs.Duringthesummertraining,sevenofthenineinterventionistsattendedallfivedays. Oneinterventionistattendedfourdaysandwasabsentonedayduetoillness. Thismaterialwascoveredwiththeabsenteeduringthefollowingday’strainingduringlunchandbreaks. Anotherinterventionistdidnotattendthesummertraining.Thatinterventionistwastrainedone-on-oneduringthreedaysattheinterventionist’sschool.Duringtheschoolyeartherewerethreeregulartrainingdates. Allinterventionistsattendedtwoofthesedates,andeightattendedthethird. TheabsentinterventionistmadeupthemissedtrainingdateviaSkypewithKDEstaff. InterventionistsalsoattendedtheKentuckyReadingAssociationconference. Allnineinterventionistsattendedallthreedaysoftheconference.Attendanceandparticipationwasadequateforallinterventionists.

Inadditiontosummertraining,interventionistsreceivedsitevisitsupportfromKDEstaff

throughouttheyear. Thenumberofvisitsrangedfromfourdaysto11daysperinterventionistatsixhoursperdaywithanaverageof5.56days. Whenextrasupportwasneeded,additionalvisitswerescheduled.Itemsdiscussedduringthevisitswerestrategyimplementation,project-basedlearning,datacollection,studentengagementandmotivation,assessmentproceduresandothervarioustopicsdependingontypeofsupportneededrelatedtotheKCLMmodel.

InterventionistsreceivedsupportthroughtheuseofaNING,Skype,emailandphonecalls.

Phonecalls,emailandSkypewereusedforongoingsupportandprofessionaldevelopment. The

Page 20: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

18SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

NINGwasusedforbookstudies,discussionforums,bloggingandsharingresources.KDEstaffestimatethateachinterventionistwassupportedanestimatedthreetimesperweekviaphonecallsandemails,anestimatedonceperweekviaNING,andoncepermonthviaSkype.

Allparticipationintheprofessionaldevelopmenttrainingandsupportwasconsideredtobeadequateforallinterventionists.

Professionaldevelopmentforadministrators. Administratorswereprovidedwithonetrainingdayinthesummerof2010todiscusstherequirementsofthegrantandwaystheycouldsupporttheintervention. Administratorsfromallnineschoolsattended.

Duringtheschoolyear,KDEstaffmetwithschooladministratorstodiscussgrantrequirements,evidence-basedcomponentsofthetargetedintervention,schedulingissues,observationsofinterventionists,literacyleadership,literacyplanning,andmeetingtheneedsofadolescentlearners—especiallylowachievingstudents. Thenumberofmeetingsrangedfromtwotosixperschoolwithanaverageof3.11meetingsperschool. KDEstaffalsosharedaKCLMUpdatenewsletterwithalladministratorsfourtimesthroughouttheyear. Alladministratorparticipationwasconsideredtobeadequate.ImplementationofClassroomModel

Classsize,intensity,andduration. Classsizesvariedthroughouttheyearduetostudentattrition(i.e.,transferringschools,droppingout,etc.). Classesrangedfromasfewas12studentstoasmanyas20.Theinterventionclassmeteverydaythroughoutthecourseoftheyear,andrangedfrom45minutesto75minutesdaily.

ClassroomImplementationResults. TheobservationprotocolincludedratingsforbothpresenceofindicatorsforeachcomponentforKCLM,andqualityofeachcomponentofKCLM.Thenextfourtablesshowthenumberoftimesinterventionistswereobservedincorporatingindicatorsofmotivationandengagement,strategicprocessing,instructionalstrategies,andcommunicationskillsintheirlessons.Table2.2indicatesthemodelcomponentsmostwidelyimplementedbyteachersandthenumbersandpercentsoflessonsthatincludedthosecomponents.Othermodelcomponentswereobservedinfewerthan50%oflessons.

Page 21: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

19SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

Table2.2

MostWidelyImplementedModelComponents

ModelComponent Round1N=18

% Round2 %N=18

Motivation&EngagementTeachermakesconnectionsbetweenlearningactivitiesandrealworldissuesTeachervariesinstructionalformat,i.e.groupwork,lecture,andpartnerworkStudentpriorknowledge,interestandbackgroundusedindeterminingcontent

StrategicProcessingTeacherexplainscognitivestrategiesforcomprehensionTeachermodelsusingcognitivestrategiesforcomprehensionTeacherencouragesandprovidesopportunitiesforstudentstopracticeusingcognitivestrategiesforcomprehension

InstructionalStrategiesTeacherexplicitlyincorporateshigherlevelquestionsTeacherusesinstructionaltoolstosupportstudentcomprehensionorvocabularyTeacherdescribes,explains,andprovidesanexampleofnewkeyterms

CommunicationSkillsTeacherusesduringtext-baseddiscussionstrategiesTeacherincludeswritingtolearnactivities

ForeachKCLMcomponent,interventionistsfocusedonsomecomponentsmorethanothers.Formotivationandengagement,interventionistsweremorelikelytomakeconnectionsbetweenthelessonandrealworldissues,tovaryinstructionalformat,andtotiethecontentintostudentexperiences. Interventionistswerenotaslikelytoincorporatetechnology,providestudentfeedback,facilitateproblemsolvingandprovidestudentswithchoices.Forstrategicprocessing,interventionistsweremorelikelytoexplain,modelandencouragestudentpracticeofcognitivestrategies. Interventionistswerenotaslikelytoaskstudenttoreflectoncognitiveprocessingandreadingcomprehensionorprovideexplicitinstructioninfoundationalreadingskills.Overall,interventionistswerenotaslikelytoincorporateinstructionalstrategiesintotheirlessons. Inthefirstroundofobservations,interventionistsdiduseinstructionaltoolstosupportstudentcomprehensionorvocabulary,andinthesecondroundtheywerelikelytodescribe,

13 72 9 50

9 50 9 50

11 61 12 67

14

78

9

50

12 67 10 56

15 83 13 72

7

39

10

56

12 67 6 33

8 44 11 61

11

61

6

33

15 83 9 50

Page 22: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

20SRCohortIIEvaluationReport:Kentucky

explainandgiveexamplesofnewvocabularywords.Communicationskillswerealsounder-utilizedbyinterventionists. Therewassomeevidenceofteacherusesduringtext-baseddiscussionstrategies,andwritingtolearnactivities.

ClassroomobservationsalsoincludedaqualityscoringforeachKCLMcomponent. Table2.3showseachteacher’squalityscorebyKCLMcomponentforeachroundofobservation. Eachroundrepresentsanaveragebetweenscoresforeachclassperiodobserved.Table2.3showsthemeanscoresbyteacherforqualityofeachKCLMcomponentandtheoverallscoreforthemodel.Intermsofoverallqualityofimplementation,implementationofKCLMwasadequatein3ofthe9classrooms.

Table2.3

MeanScoresbyTeacherforQualityofKCLMcomponent

Classrooms

123456789

Note.1-1.4=Developing,1.5-2=Adequate,2.1-3=Exemplary

Experiencesforcontrolstudentsduringinterventionperiod. Studentswhowereselectedforthecontrolgroupreceivedaregularelectiveaspartoftheirfreshmanprogram.Awiderangeofelectivesweretakenincludingband,chorus,civics,andphysicaleducation.

Additionalreadingprograms.Intwoschools,additionalreadingassistanceprogramswere

providedtostudentswhoqualified. AtoneschoolstudentsfromboththeinterventionandcontrolgroupstookanunstructuredreadingclasswheretheteacherhadaccesstomaterialsfromStudyIslandandDiscoveryEducationaswellasRead180andSystem44.Inthesecondschool,studentsinbothinterventionandcontrolgroupstookaonesemesterclasscalledReadingRevisited,ahighlystructuredclassrelyingheavilyonvocabularyworkbookexercises.

Implicationsforimpactanalysis.Twofactorsrelatedtoimplementationshouldbe

consideredwheninterpretingtheimpactanalysistofollow.First,teacherswerelearningtoimplementtheinterventionastheywereimplementingit,andthisresultedinavariationin

MotivationEngagement

Mean

StrategicProcessingMean

InstructionalStrategiesMean

CommunicationSkillsMean

OverallscoreMean

1 1 1 1 11.5 1.5 1 1.5 1.382.25 2 2 2 2.061 1 1 1 12 2 2 1.5 1.882 2 2 1.5 1.881 1 1 1 11.5 1 1 1 1.131 1 1 1 1

Page 23: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

20SRCohortIIEvaluationReport:Kentucky

implementationquality.Teacherswereadheringtothemajorcomponentsoftheintervention,buttheydidnotnecessarilyimplementthosecomponentsastheyweredesignedtobeimplemented.Therefore,impactfindingsshouldnotbedirectlyattributedtotheinterventionmodelasitisdesigned.Second,thatsomeschoolscontinuedtoimplementreadingprogramstargetedatlowachievingreaderspresentsaconfoundingfactorthatmayhaveinfluencedoutcomesforbothtreatmentandcontrolgroups.

StudyDesign

EvaluationofImpact

Samplingselectionprocess. Evaluatorsimplementedastratifiedrandomsamplingprocedureforstudentswithineachschoolusingfourdemographicvariables: specialeducationstatus,free/reducedlunchstatus,ethnicity,andgender. Withineachschoolstudentsweresortedbydemographicvariablescreatingsubgroupsofstudents,andstudentswithineachsubgroupwerethensortedbyassessmentscore. Usingarandomnumbergeneratortoassignthefirststudenttoeithertheinterventionorcontrolgroup,thestudentswerethenalternatelyassignedsequentiallytotheinterventionorcontrolgroup.

Evaluatorsprovidedalistofstudentsthatqualifiedfortheinterventiongrouptothenineparticipatinghighschoolsduringthesummerof2011. Thesestudentswerescheduledtobeintheinterventionclass,andstudentsandparentswerenotifiedwhentheclassschedulesweregiventoallstudentsatthebeginningofthefallsemester. Schoolswerenotprovidedthenamesofthestudentsinthecontrolgroup.

Astudentwasremovedfromthestudypost-randomassignmentifhe/shedroppedoutofschoolormoved/transferredtoaschoolnotparticipatingintheStrivingReadersprogram.Also,ifthestudentdidnottaketheposttestinthespringof9thgrade,theywereremoved.Finally,astudentwasineligiblepost-randomassignmentif,afterthestudentenrolledintheninth-grade,thehighschoolplacesthestudentinalldayresourceclasses. Theschoolswereinstructedtoinformtheevaluatorsattheendofthestudyifacontrolstudentwasplacedinalldayresourceclasses. Thusthecriterionthehighschoolusedtoassignastudenttoalldayresourcewouldnotbeaffectedbytheresultsoftherandomassignment,andwasappliedequallytointerventionandcontrolstudents. Therewere13interventionstudentsthatwereassignedtoalldayresourceclassespost-randomassignment,andareineligibletobeinthestudy. Thereweresixteencontrolstudentswhowereplacedinalldayresourceclassesafterrandomassignment.

Samplesize.Exhibit1showsthesamplesizeresultsforreadingachievementtest,GRADE,dividedbycondition. Outofthepopulationof2,229eighthgradestudents,637wereidentifiedaslowachievingreaders(NCEof40orlower). Threehundrednineteenstudentswereassignedtothetreatmentgroup,and318studentswereassignedtothecontrolgroup.Afterrandomassignment,13studentsinthetreatmentgroupand16studentsinthecontrolgroupwereassignedtoalldayresourceclassesafterenrollinginhighschoolsowereineligibleforthestudy. Additionally,22studentsinthetreatmentgroupdidnotenrollinaparticipating

Page 24: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

22SRCohortIIEvaluationReport:Kentucky

highschool. Thenumberofstudentsinthecontrolgroupthatdidnotenrollinaparticipatinghighschoolisnotavailable.

Therewere306targeted,eligiblestudentsatbaselineintheinterventiongroupand302

studentsinthecontrolgroup. Attritionintheinterventiongroupattheendoftheschoolyeartotaled74students(36studentsmovedand38studentsdidnottakethespringtest),resultinginananalyticsampleof232interventionstudents. Attritioninthecontrolgrouptotaled49students(34studentseitherdidnotenrollintheparticipatinghighschoolormovedduringtheyearand15studentsdidnottakethespringtest),resultinginananalyticsampleof253controlstudents.

Page 25: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

23SRCohortIIEvaluationReport:Kentucky

Exhibit2showsthesamplesizeresultsforthewritingachievement,bycondition. Thetargetednumberofeligiblestudentsatbaselineintheinterventiongroupwas306,with302studentsinthecontrolgroup. Attritionintheinterventiongroupduringtheschoolyeartotaled134students(36studentsmovedand98studentsdidnottakethespringwritingtest),resultinginananalyticsampleof172interventionstudents. Attritioninthecontrolgrouptotaled109students(34studentseitherdidnotenrollintheparticipatinghighschoolormovedduringtheyear,and75studentsdidnotcompletethespringsurvey),resultinginananalyticsampleof193controlstudent.

Page 26: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

24SRCohortIIEvaluationReport:Kentucky

Exhibit3showsthesamplesizeresultsforstudentsurvey,bycondition. Thetargetednumberofeligiblestudentsatbaselineintheinterventiongroupwas306,with302studentsinthecontrolgroup. Attritionintheinterventiongroupduringtheschoolyeartotaled152students(36studentsmovedand116studentsdidnotcompletethespringsurvey),resultinginananalyticsampleof154interventionstudents. Attritioninthecontrolgrouptotaled129students(34studentseitherdidnotenrollintheparticipatinghighschoolormovedduringtheyear,and95studentsdidnotcompletethespringsurvey),resultinginananalyticsampleof173controlstudent

Page 27: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

25SRCohortIIEvaluationReport:Kentucky

ImpactMeasuresandDataCollection

ThefollowingmeasureswereusedtoascertaintheimpactoftheKCLMinterventiononstudents’readingandwritingachievement,readingstrategyuse,andreadingmotivation.

GroupReadingandDiagnosticEvaluation(GRADE).TheGRADEisanationallynormedreadingassessment,anditincludesnormalizedscoresforoverallreadingachievement. NormalCurveEquivalentscores(NCEs)includingvocabularyandcomprehensionitemscloselyalignstudentoutcomeswiththegoalsoftheintervention.Inthespringoftheplanningyear,theGRADE(levelM)wasgiventoalleighth-gradestudentsinparticipatingfeederschoolsasapretest. Inthefollowingspring,theGRADE(levelH)wasgiventoallninth-gradestudentsasapostest.

KentuckyStateWritingAssessment. TheKentuckystatewritingassessmentprovidesaholisticscorebasedonanalyticcategorieswhichcloselyalignstudentoutcomeswiththegoalsoftheintervention. TheStateWritingAssessmentwasgiventoalleighth-gradestudentsinspringoftheplanningyear,andallninth-gradestudentsinspringofthefollowingyear. Thetestisdesignedsothatstudentsgetachoicebetweentwowritingtaskswhichincludethreepossiblemodesofwriting(inform,narrateforapurpose,orpersuade)andfourpossibleresponseformats(article,editorial,letterorspeech). Inadditiontoawritingtask,eachstudentisalsopresentedwithdraftversionsofthreepiecesofwriting. Fourmultiple-choicequestionsdealingwithediting/revisingareprovidedwitheachofthethreedraftsso,eachstudentalsorespondsto12multiple-choicequestions.

MetacognitiveAwarenessofReadingStrategiesInventory(MARSI).TheMARSI(Mokhtari&Reichard,2002)isastudentself-reportmeasuredesignedspecificallytoassessadolescents’perceiveduseofreadingstrategiesduringacademicreading.Forthisstudy,theMARSIwasadaptedtomeasureself-efficacywithstrategyuse.“Ican”wasaddedtothebeginningofeachstrategystatementinthesurvey.Thesurveywasgiventoallstudentsinfallandspringofninthgrade. Thesurveyitemsarepresentedonascaleof1to5,where1isequalto“notatallconfident”and5isequalto“completelyconfident.” Exampleitemswouldbe“IamabletohaveapurposeinmindwhileIread”or“IcantakenoteswhilereadingtohelpmeunderstandwhatIread.”

AdolescentMotivationSurvey. Inthespringofeighthgradeandinthespringofninthgrade,allstudentsinparticipatingschools(andeighth-gradefeederschools)completedasurveymeasuringseveraldimensionsofintrinsicreadingmotivation(challenge,curiosity,intrinsictaskvalue,attainment),extrinsicreadingmotivation(extrinsictaskvalue,compliance),readingrelatedself-beliefs(expectancy,difficulty)andleisurereading.Thesurveyitemsarepresentedonascaleof1to5andthevaluevariesdependingonthedimensionmeasured. Asampleitemforthechallengedimensionis“Ilikehard,challengingbooks”where1equals “notatalltrueand5equals“verytrue.” Asampleitemfromtheleisuredimensionis“Howmuchtimehaveyouspentreadingamagazinethisweek?”with1equaling“none,”and5equaling“morethanfivehours.” Reliabilityforsubscaleswere>.70withtheexceptionofleisure

Page 28: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

26SRCohortIIEvaluationReport:Kentucky

(a=.585)andextrinsictaskvalue(a=.60).Thesescalesweretakenfromexistingmeasures(Eccles&Wigfield,1995;Hopper,2005;Wigfield&Guthrie,1997).SummaryofAnalyticApproach

HierarchicalLinearModels(HLMs)wereusedtoestimatetheimpactoftheKCLMonstudentachievement,motivation,andreadingstrategiesoutcomes.TheGRADENormalCurveEquivalents(NCEs)wereusedtoestimatetheimpactoftheKCLMinterventiononachievement.HolisticscoresfromthestatewritingassessmentwereusedtoestimatetheimpactoftheKCLMonwritingachievement.TheaverageoftheitemsontheMARSIwasusedtoestimatetheimpactonreadingstrategyuse,andtheaverageoftheitemsontheAdolescentMotivationSurveywasusedtoestimatetheimpactonmotivation.

Atwo-levelHLMmodel(studentsassignedtointerventionorcontrolgroupwithinschools)willbeusedtodeterminetheimpactofKCLM.Fourhypotheseswillbetested:

H1: TheKCLMinterventionhasnoimpactonstudentachievement.H2: TheKCLMinterventionhasnoimpactonstudentwritingachievement.H2: TheKCLMinterventionhasnoimpactonselfefficacyofstudentstrategyuse.H3: TheKCLMinterventionhasnoimpactonstudentmotivation.

Level-1HLM:StudentLevel.Atthestudentlevel,thespringoutcomevariable(reading

achievement,writingachievement,strategyuse,ormotivation)willbemodeledasafunctionoffalloutcomevariables(covariate),intervention/controlstatusandfourdemographicvariables:gender,ethnicity,free/reducedlunchstatus,andspecialeducation.

Level-2HLM:SchoolLevel.Thisanalysiswillbeperformedonninth-gradestudents’scoresfromninehighschools. InadditiontothebaseyearReadingKCCTscore,otherschoollevelvariablesthatwillbeincludedaretheschoolpercentofstudentsqualifyingforfreeorreducedlunchfees,schoolpercentofwhitestudentsintheschool,andschoolpercentofblackstudents,andthepercentofstudentswithdisabilities.DescriptionoftheFirstYearSample

Ninehighschoolsgeographicallydistributedacrossthestateparticipatedinthestudy.HighschooldemographicdatawascollectedfromtheKentuckyDepartmentofEducationwebsiteforthe2009-10academicyear. Theaveragenumberof10th-12thgradestudentsenrolledattheparticipatingschoolswas454,rangingfrom301to803students. TheaveragepercentofWhitestudentswas91.58%(53.9%,98.8%),andtheaveragepercentBlackstudentswas5.76%(0%,37.7%). Theaveragepercentofstudentsreceivingfree/reducedlunchwas62.25%(40.7%,83.5%),andtheaveragenumberofstudentsenrolledinspecialeducationclasseswas12.73%,rangingfrom7.2%to16.5%.

Studentdemographicdatawascollectedfromthemiddleschoolsforeveryenrolledeighth-gradestudent.Schoolswerecontactedforstudentdemographicinformationifthere

Page 29: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

27SRCohortIIEvaluationReport:Kentucky

werestudentsthatwerenotontheselistsbutcompletedtheeighth-grade.Table2.4showsdemographiccharacteristicsofstudentsintheinterventionandcontrolgroups.StudentsintheStrivingReadersclassesweretypicallyWhitemales,receivingfree/reducedlunchservicesandarenotassignedtospecialeducationclasses. Studentsthatreceivedtheinterventionwereverysimilarindemographicsascomparedtothecontrolgroup,withthepossibleexceptionofgender,whereaslightlyhigherpercentofmalesreceivedtheintervention.

Table2.4

InterventionandControlStudentDemographics(andProportions)

Gender Ethnicity Lunch Special EducationGroup Male Female White Minor-

ityReg

PayFree/Red

NotIn

SpecialEd.

Interv

136

96

205

27

40

192

161

71

(.59) (.41) (.88) (.12) (.17) (.83) (.69) (.31)

Contl 140 113 219 34 45 208 179 74 (.55) (.45) (.87) (.13) (.18) (.82) (.71) (.29)

Total

276

109

424

61

85

400

340

145

(.57) (.43) (.88) (.12) (.38) (.62) (.73) (.16)

ImpactsonStudents

Table2.5belowshowstheresultsonstudentreadingachievement,writingachievement,readingstrategyuseselfefficacyandreadingmotivationforinterventionandcontrolstudentsafteroneyearofintervention. Theunadjustedmeansandstandarddeviationsforeachmeasureisdisplayed,andthemeansadjustedfortheHLMresultsaredisplayed. Theestimatedimpactoftheintervention,theeffectsize,andthesignificancelevelareshown.

Page 30: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

28SRCohortIIEvaluationReport:Kentucky

Table2.5ImpactoftheTargetInterventiononStudentReadingAchievement,WritingAchievement,StrategyUseSelfEfficacy,andMotivation

Unadjusted HLM-adjusted

Means Means Control Tx Control Tx Estimated

ImpactEffectSize

p

ReadingAchievementSpringNCE 36.7 35.6 37.2 36.4 -0.79 -0.059 .439 (13.49) (13.48)

No.ofstudents 253 232

WritingAchievement

No.ofstudents 193 172

StrategyUseSelfEfficacySpring score 3.3 3.5 3.4 3.5 0.154 0.250 .012*

(0.62) (0.61)No.ofstudents 173 154

MotivationSpringscore 2.9 3.0 2.9 3.0 .128 0.217 .015*

(0.59) (0.62)

No.ofstudents 173 154

Note. Standarddeviationsarepresentedinparenthesis. Effectsizecalculatedastheimpactdividedbythecontrolgroupstandarddeviation.*Designatesstatisticalsignificanceatthe.05levelofsignificance.

Thereisnosignificanteffectoftheinterventionatthe.05levelonthereadingand

writingachievementafterthefirstyearoftheprogram. However,asignificanteffectoftheinterventionisshownforstrategyuseselfefficacyandreadingmotivation.

Springscore 826.3 827.2 826.9 827.6 0.62 0.066 .481 (9.41) (11.78)

Page 31: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

29SRCohortIIEvaluationReport:Kentucky

DiscussionandConclusions

TheimpactresultsfromoneyearofstudydonotrevealsignificanteffectsoftheKCLMinterventiononstudents’readingorwritingachievement,butthereweresignificantimpactsonstudents’self-efficacyforstrategyuseandstudents’readingmotivation.Developmentofdemonstrableimprovementsinreadingperformancemayrequiregreaterlengthsoftimetogaincomfortwithflexiblestrategyuseandtoreapbenefitsofincreasedmotivation.Whileitseemsthestudentsinthisstudyreportedincreasedconfidencewithusingreadingstrategies,theymaynothaveinternalizedandpracticedstrategyusetoasufficientenoughextenttoachievepurposefulflexibleuseunderawiderangeofconditions.Nevertheless,theimpactsoftheinterventiononstudents’strategyuseandmotivationarenoteworthygiventheemphasisplacedonthesedimensionsoflearninginrecommendationsforimprovingadolescents’literacyachievement(Biancarosa&Snow,2006;Kamil,Borman,Dole,Kral,Salinger,&Torgesen,2008).

Althoughtheplannedprofessionaldevelopmentmodelwasimplementedathighlevels,implementationoftheclassroommodelwaslower.ElementsoftheKCLMmodelwereevidentineachclassroomobservation,buttheteachersinthisprojectimplementedsomecomponentsoftheinterventiontoagreaterextentthanotherimportantcomponents.Also,themajorityofteachersimplementedtheinterventionatdevelopinglevelsofquality.Higherlevelsofimplementationmayhaveresultedinhigherlevelsofachievementforstudents.Itisprobablethatteacherswouldhavebeenabletoachievehigherlevelsofimplementationinfutureyearswithongoingsupporthadtheprojectextendedforthefullplannedprojectduration.

Page 32: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

31SRCohortIIEvaluationReport:Kentucky

References

Allington,R.(2007).Effectiveteachers,effectiveinstruction.InBeers,K.,Probst,R.E.,&Rief,L.(Eds.),Adolescentliteracy:Turningpracticeintopromise(pp.273-288).Portsmouth,NH:Heinemann.

Almasi,J.(2003).Teachingstrategicprocessesinreading.NewYork:TheGuilfordPress.

Anderson,V.,&Hidi,S.(1988/1989).Teachingstudentstosummarize.EducationalLeadership,46,26-28.Retrievedfromhttp://www.ascd.org/publications/educational-leadership.aspx

Armbruster,B.B.,Anderson,T.H.,&Ostertag,J.(1987).Improvingcontent-areareadingusing

instructionalgraphics.ReadingResearchQuarterly,26(4),393-416.Retrievedfromhttp://www.reading.org/general/publications/journals/rrq.aspx

Bandura,A.(1993).Perceivedself-efficacyincognitivedevelopmentandfunctioning.

EducationalPsychologist,28(2),117-148.doi:10.1207/s15326985ep2802_3

Bandura,A.(1997).Theexerciseofcontrol.NewYork,NY:Longman.

Biancarosa,G.,&Snow,C.E. (2006). Readingnext–Avisionforactionandresearchinmiddleandhighschoolliteracy: AreportfromtheCarnegieCorporationofNewYork.Washington,DC: AllianceforExcellentEducation.

Blank,W.(1997).Authenticinstruction.InW.E.Blank&S.Harwell(Eds.),Promisingpractices

forconnectinghighschooltotherealworld(pp.15-21).Tampa,FL:UniversityofSouthFlorida.(ERICDocumentReproductionServiceNo.ED407586)

Bloome,D.(1986).Readingasasocialprocessinamiddleschoolclassroom.InD.Bloome(Ed.),

Literacyandschooling(pp.123-149).Ablex:Norwood,NJ.Bransford,J.,Brown,A.,&Cocking,R.(1999).Howpeoplelearn:Brain,mind,experience,and

school.Washington,DC:NationalAcademyPress.Brewster,C.andFager,J.(2000).IncreasingStudentEngagementandMotivation:FromTime-

on-TasktoHomework.NorthwestRegionalEducationalLaboratory.RetrievedJuly9,2009,fromtheWorldWideWeb:http://www.nwrel.org/request/oct00/textonly.html.

Brookbank,D.,Grover,S.,Kullberg,K.,&Strawser,C.(1999).Improvingstudentachievement

throughorganizationofstudentlearning.Chicago:Master'sActionResearchProject,SaintXavierUniversityandIRI/Skylight.

Caine,R.N.,&Caine,G.(1994).Makingconnections:Teachingandthehumanbrain.Menlo

Page 33: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

30SRCohortIIEvaluationReport:Kentucky

Park,CA:Addison-WesleyCampbell,D.T.&Stanley,J.C.(1963).Experimentalandquasi-experimentaldesignsfor

research.Chicago:Rand-McNally.Carney,P.(2007).KentuckyReadingFirstEvaluation.UniversityofKentucky.Cartwright,K.(2008).Literacyprocesses:Cognitiveflexibilityinlearningandteaching.New

York:GuilfordPress.Chen,Z.(1999).Children'sanalogicalproblemsolving:Theeffectsofsuperficial,structural,and

proceduralsimilarities.JournalofExperimentalChildPsychology,62(3),410-431.doi:10.1006/jecp.1996.0037

Cook,T.D.,&Campbell,D.T.(1979).Quasi-experimentation:Designandanalysisfor

fieldsettings.Boston:HoughtonMifflin.Creswell,J.W.&PlanoClark,V.L.(2007).Designingandconductingmixedmethodsresearch.

ThousandOaks,CA:Sage.Creswell,J.W.(1998).Qualitativeinquiryandresearchdesign:Choosingamongfive

traditions.ThousandOaks,CA:Sage.Deshler,D.,Palincsar,A.,Biancarosa,G.,&Nair,M.(2007).Informedchoicesforstruggling

adolescentreaders.Newark,Delaware:InternationalReadingAssociation.Dev,P.C.(1997).Intrinsicmotivationandacademicachievement.Whatdoestheirrelationship

implyfortheclassroomteacher?RemedialandSpecialEducation,19(1),12-19.doi:10.1177/074193259701800104

Durkin,D.(1978-79).Whatclassroomobservationsrevealaboutreadingcomprehension

instruction.ReadingResearchQuarterly,15,481-533.doi:10.1598/RRQ.14.4.2Dyson,A.H.(1992).WhistleforWillie,lostpuppies,andcartoondogs:Thesociocultural

dimensionsofyoungchildren'scomposing.JournalofReadingBehavior,24(4),433-462.Eccles,J.S.,&Wigfield,A.(1995).Inthemindoftheactor:Thestructureofadolescents’

achievementtaskvaluesandexpectancy-relatedbeliefs. PersonalityandSocialPsychologyBulletin,21(3),215-225.doi:10.1177/0146167295213003

Fink,R.(2006).WhyJaneandJohncouldn’tread—andhowtheylearned:Anewlookatstriving

readers.Newark,DE:InternationalReadingAssociation.Fisher,D.,Frey,N.,&Rothenberg,C.(2008).Content-areaconversations:Howtoplan

Page 34: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

32SRCohortIIEvaluationReport:Kentucky

discussion-basedlessonsfordiverselanguagelearners.Alexandria,VA: AssociationforSupervisionandCurriculumDevelopment

Flavell,J.H.(1979).Metacognitionandcognitivemonitoring:Anewareaofcognitive-

developmentalinquiry.AmericanPsychologist,34,906-911.doi:10.1037/0003-066X.34.10.906

Fugate,M.&Waterman,K.(2004).Groupreadingassessmentanddiagnosticevaluation

reliabilityregardingeducationaldecisionmaking.JournalofEducationalPsychology,96(2),187-201.

Gambrell,L.B.,Palmer,B.M.,Codling,R.M.,&Mazzoni,S.A.(1996).Assessingmotivationto

read.TheReadingTeacher,49,518-533.doi:10.1598/RT.49.7.2Graham,S.,&Perin,D.(2007).Writingnext:Effectivestrategiestoimprovewritingof

adolescentsinmiddleandhighschools.AreporttoCarnegieCorporationofNewYork.Washington,DC:AllianceforExcellentEducation.

Guthrie,J.T.,&Wigfield,A.(2000).Engagementandmotivationinreading.InM.L.Kamil,P.B.

Mosenthal,P.D.Pearson,&R.Barr(Eds.),Handbookofreadingresearch:VolumeIII(pp.403-422).NewYork:Erlbaum.

Hopper,R.(2005).Whatareteenagersreading?Adolescentfictionreadinghabitsandreading

choices.Literacy39(3),113-120.doi:10.1111/j.1467-9345.2005.00409.xcKamil,M,L.,Borman,G.D.,Dole,J.,Kral,C.C.,Salinger,T.S.,&Torgesen,J.(2008). Improving

adolescentliteracy:Effectiveclassroomandinterventionpractices. Washington,D.C.:U.S.DepartmentofEducation,InstituteofEducationSciences,NationalCenterforEducationEvaluationandRegionalAssistance.

Kane,S.(2007).Literacyandlearninginthecontentareas.Scotsdale,AZ:HolcombHathaway.Kushman,J.W.,Sieber,C.,&Heariold-Kinney,P.(2000).Thisisn'ttheplaceforme:School

dropout.InD.Capuzzi&D.R.Gross(Eds.),Youthatrisk:Apreventionresourceforcounselors,teachers,andparents(3rded.,pp.471-507).Alexandria,VA:AmericanCounselingAssociation.

Langer,J.A.(2001).Guidelinesforteachingmiddleandhighschoolstudentstoreadandwrite

well.Albany,NY:NationalResearchCenteronEnglishLearningandAchievement,StateUniversityofNewYorkatAlbany.

Lapp,D.&Fisher,D.(2009).Introduction.InLapp,D.&Fisher,D.(Eds.),Essentialreadingsin

comprehension.Newark,DE:InternationalReadingAssociation.

Page 35: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

33SRCohortIIEvaluationReport:Kentucky

Lee,V.E.,&Smith,J.B.(1996).Collectiveresponsibilityforlearninganditseffectsongainsinachievementforearlysecondarystudents.AmericanJournalofEducation,104(2),103-147.doi:10.1086/444122

Lenz,B.K.,&Hughes,C.A.(1990).Awordidentificationstrategyforadolescentswithlearning

disabilities.JournalofLearningDisabilities,23(3),149-158.doi:10.1177/002221949002300304

Marzano,R.(1998).ATheory-BasedMeta-AnalysisofResearchonInstruction.Mid-Continent

RegionalEducationalLab.,Aurora,CO.Marzano,R.(2004).Buildingbackgroundknowledgeforacademicachievement:Researchon

whatworksinschools.Alexandria,VA:AssociationforSupervisionandCurriculumDevelopment.

Marzano,R.J.,Pickering,D.J.,&Pollock,J.E.(2001).Classroominstructionthatworks:

Research-basedstrategiesforincreasingstudentachievement.Alexandria,VA:AssociationforSupervisionandCurriculumDevelopment.

Mastropieri,M.A.,Scruggs,T.E.,Hamilton,S.L.,Wolfe,S.,Whedon,C.,&Canevaro,A.(1996).

Promotingthinkingskillsofstudentswithlearningdisabilities:Effectsonrecallandcomprehensionofexpositoryprose.Exceptionality,6(1),1-11.doi:10.1207/s15327035ex0601_1

Matsumura,L.,Garnier,H.,Pascal,J.,andValdes,R.(2002).Measuringinstructionalqualityin

accountabilitysystems:Classroomassignmentsandstudentachievement.EducationalAssessment,1532-6977,8(3),207–229.doi:10.1207/S15326977EA0803_01

Midgley,C.,Kaplan,A.,Middleton,M.,Maehr,M.L.,Urdan,T.,Anderman,L.H.,Anderson,E.,

&Roeser,R.(1998). Thedevelopmentandvalidationofscalesassessingstudents‘achievementgoalorientations.ContemporaryEducationalPsychology,23,113-131.doi:10.1006/ceps.1998.0965

Moats,L.C.(1998).Teachingdecoding.AmericanEducator,Spring/Summer,42-49.Retrieved

fromhttp://www.aft.org/newspubs/periodicals/ae/Mokhtari,K.&Reichard,C.A.(2002).Assessingstudents‘metacognitiveawarenessof

readingstrategies.JournalofEducationalPsychology,94(2),249-259.doi:10.1037/0022-0663.94.2.249

Moore,D.W.,&Readence,J.E.(1984).Aquantitativeandqualitativereviewofgraphic

organizerresearch.JournalofEducationalResearch,78(1),11-17.Retrievedfromhttp://www.tandf.co.uk/journals/titles/00220671.asp

Page 36: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

34SRCohortIIEvaluationReport:Kentucky

NationalCouncilofTeachersofMathematics.(2000).Principlesandstandardsforschoolmathematics.Reston,VA.

Patrick,H.,Ryan,A.M.,&Kaplan,A.(2007).Earlyadolescents‘perceptionsoftheclassroom

socialenvironment,motivationalbeliefs,andengagement.JournalofEducationalPsychology,99,83-98.doi:10.1037/0022-0663.99.1.83

Pearson,P.D.,Roehler,L.R.,Dole,J.A.,&Duffy,G.G.(1992).Developingexpertiseinreading

comprehension.InJ.Samuels.,&A.Farstrup,(Eds.),Whatresearchhastosayaboutreadinginstruction.Newark,Delaware:InternationalReadingAssociation.

Pitcher,S.M.,Albright,L.K.,DeLaney,C.J.,Walker,N.T.,Seunarinesingh,K.,Mogge,S.,

Headley,K.N.,Ridgeway,V.G.,Peck,S.,Hunt,R.,&Dunston,P.J.(2007).Assessingadolescents‘motivationtoread.JournalofAdolescentandAdultLiteracy,50,378-396.doi:10.1598/JAAL.50.5.5

Pressley,M.,Brown,R.,El-Dinary,P.B.,&Afflerbach,P.(1995).Thecomprehensioninstruction

thatstudentsneed:Instructionfosteringconstructivelyresponsivereading.LearningDisabilitiesResearchandPractice,10,215–224.Retrievedfromhttp://mc.manuscriptcentral.com/ldrp

Raudenbush,S.,Spybrook,J.,Liu,X.,&Congdon,R.(2004)Optimaldesignforlongitudinaland

multilevelresearch:DocumentationfortheOptimalDesignsoftware.AnnArbor:UniversityofMichigan.

Redfield,D.,&Rousseau,A.(1981).Ameta-analysisofexperimentalresearchonteacher

questioningbehavior.ReviewofEducationResearch,51,237-246.doi:10.3102/00346543051002237

Ross,J.A.(1988).Controllingvariables:Ameta-analysisoftrainingstudies.Reviewof

EducationalResearch,58(4),405-437.doi:10.2307/1170280Rozzelle,J.,Searce,C.(2009).PowerToolsforAdolescentLiteracy:StrategiesforLearning.

Retrievedfromhttp://go.solution-tree.com/literacy/

Seidman,I.E.(1998).Interviewingasqualitativeresearch.NewYork:TeachersCollegePress.

Silver,H.,Strong,R.,Perini,M.(2007).TheHiddenSkillsofAcademicLiteracy.Alexandria:ThoughtfulEducationPress.

Smith,M.W.,&Wilhelm,J.D.(2006).Goingwiththeflow.Portsmouth,NH:Heinemann.Snider,VE.(1989).Readingcomprehensionperformanceofadolescentswithlearning

disabilities.LearningDisabilityQuarterly,12,87-96.doi:10.2307/1510724

Page 37: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

35SRCohortIIEvaluationReport:Kentucky

StanfordAchievementTestSeries,TenthEdition(SAT-10).PearsonforEducation.

http://pearsonassess.com/HAIWEB/Cultures/en-us/Productdetail.htm?Pid=SAT10C.RetrievedJuly28,2009.

Stone,C.L.(1983).Ameta-analysisofadvancedorganizerstudies.JournalofExperimental

Education,51(7),194-199.Retrievedfromhttp://www.tandf.co.uk/journals/titles/00220973.asp

Strong,R.,Silver,H.,andPerini,M.(2001).Teachingwhatmattersmost:Standardsand

strategiesforraisingstudentachievement.Alexandria:AssociationforSupervisionandCurriculumDevelopment.

Strong,R.,Silver,H.,Perini,M.,Tuculescu,G.(2002).ReadingforAcademicSuccess.Thousand

Oaks:Corwin.TestofWrittenLanguage-Intermediate,4thEdition(TOWL-4).PearsonPsychCorp.Retrieved

fromtheWorldWideWebJuly28,2009:http://psychcorp.pearsonassessments.com/HAIWEB/Cultures/en-us/Productdetail.htm?Pid=PAa19045.

NorthCentralRegionalEducationalLaboratory.(n.d.)Thethinkingcurriculum.OakBrook,IL:

Author.[On-line].AvailableInternet:http://www.ncre.org/sdrs/areas/rpl_esys/thinking.htm

Tomlinson,C.(2001).Howtodifferentiateinstructioninmixed-abilityclassrooms.Alexandria:

ASCD.White,T.G.,Sowell,J.,&Yanagihara,A.(1989).Teachingelementarystudentstouseword-part

clues.TheReadingTeacher,42,302-309.Retrievedfromhttp://www.reading.org/general/publications/journals/rt.aspx

Wigfield,A.,&Guthrie,J.T.(1997).Relationsofchildren’smotivationforreadingtotheamount

andbreadthoftheirreading.JournalofEducationalPsychology,89(3),420-432.doi:10.1037/0022-0663.89.3.420

Wiggins,G.,&McTighe,J.(1998).Understandingbydesign.Alexandria,VA:Associationfor

SupervisionandCurriculumDevelopment.Williams,K.(2001).Groupreadingassessmentanddiagnosticevaluationtechnical

manual.CirclePines,MN:AmericanGuidanceServices.Woods,E.G.(1995).Reducingthedropoutrate.InSchoolImprovementResearchSeries(SIRS):

Researchyoucanuse(Close-upNo.17).Portland,OR:NorthwestRegionalEducationalLaboratory.

Page 38: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

36SRCohortIIEvaluationReport:Kentucky

AppendixA

SummaryofAnalyticApproach

HierarchicalLinearModels(HLMs)wereusedtoestimatetheimpactoftheinterventiononstudentachievementinreadingandwriting,andmotivation,andself-efficacyoutcomes.TheGRADENormalCurveEquivalents(NCEs)wereusedtoestimatetheimpactoftheinterventiononreadingachievement,andthestandardwritingscoreswereusedtomeasuretheimpactonwriting. Theaveragesurveyscoreswereusedtoestimatetheimpactonself-efficacyinreadingstrategyuseandimpactonmotivation.

Atwo-levelHLMmodel(studentsassignedtointerventionorcontrolgroupwithinschools)wasusedtodeterminetheimpactofthetargetedintervention. Atthestudentlevel,thespringoutcomevariable(readingachievement,writingachievement,self-efficacyinstrategyuse,ormotivation)wasmodeledasafunctionoffalloutcomevariables,intervention/controlstatus,andfourdemographicvariables:gender,ethnicity,free/reducedlunchstatus,andspecialeducation.Level-1Model:StudentOutcomes(achievement,readingstrategies,ormotivation)

M

Yij=� 0j+

where

� 1j(Y*ij)+� 2j(Tij)+

� mj mij+ ijm=3

Yij isthespringstudentoutcome(post-test)scoreforstudentiatschoolj;� 0jisthemeanstudentoutcome(post-test)scoreforcontrolstudentsat schoolj;Y*ijisthefallstudentoutcome(pre-test)scoreforstudenticenteredat schoolj;� 1jistheaveragestudentoutcome(pre-test)slopeforstudentsatschoolj;Tij=1ifstudentiisassignedtoLSCinterventionatschoolj,and0ifcontrol;� 2jisthemeandifferenceofstudentoutcomepre-postgainbetweeninterventionand

controlstudentsatschoolj; mij areadditionalcovariatesrepresentingdemographiccharacteristicsofstudentiat

schoolj(gender,ethnicity,free/reducedlunch,andspecialeducationstatus);� mjarecoefficientscorrespondingtostudentdemographiccovariates(gender,

ethnicity,free/reducedlunch,specialeducationstatus),and ij istherandomeffectrepresentingthedifferencebetweenstudentij’sscoreandthe

predictedmeanscoreforschoolj. Theseresidualeffectsareassumednormallydistributedwithmean0andvariancea 2.

Level-2Model: StudentAchievement–SchoolLevel

Thisanalysiswasperformedondatafrom9thgradestudentscollectedforoneyear. Thecovariatesinthismodelpertaintotheconcurrentyearthestudentwasintheinterventionor

Page 39: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

37SRCohortIIEvaluationReport:Kentucky

controlgroupwiththeexceptionoftheReadingKentuckyCoreContentTest(KCCT)score,forwhichthescoreforthebaseyear,spring,2010,wasused. InadditiontothebaseyearReadingKCCTscore,otherschoollevelcovariatesincludedenrollment,percentofwhitestudentsintheschool,percentofAfricanAmericanstudents,percentofstudentsqualifyingforfreeorreducedlunchfeesandpercentofstudentswithdisabilities..

Q

� 0j=

� 1j=� 2j=� mj

=

y 00+

y 10y 20y m0

y oqWqj

+q

µ 0j

where

y 00 isthemeanstudentoutcome(post-test)scoreof9thgradecontrolstudents

inKentuckyStrivingReadersmiddleschoolsWqj areschoollevelcovariatesincludingbaseyearReadingKCCT(spring,2010),and

averageschoolpercentfree/reducedlunch,percentwhitestudents,percentblackstudents,andpercentdisability;

y oqarecoefficientscorrespondingtoschool-levelcovariates;µ 0j istheuniqueeffectofschooljonmeanstudentoutcome,holdingWqjconstant(or

conditioningonWqj)-thiseffectisassumednormallydistributedwithmean0andvarianceT 2;

y 10isthefallstudentoutcome(pre-test)slope;y 20istheoveralltargetinterventiontreatmenteffectonspringstudentoutcome

(post-test)scores;y m0 isthefixedmthstudentcovariateeffect(gender,ethnicity,free/reducedlunch,special

educationstatus) onthespringoutcomevariable.

SelectionofCovariates. TherandomassignmentprocedureincludedallstudentdemographicvariablesintheHLMmodel,sowereincludedregardlessofsignificance.Interactioneffectswerenotconsidered.

Page 40: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

38SRCohortIIEvaluationReport:Kentucky

AppendixB

StudyMeasures

StudentSurvey

Itemsforthestudentsurveywereadaptedfromthefollowingpre-existinginventories:Eccles,J.S.,&Wigfield,A.(1995).Inthemindoftheactor:Thestructureofadolescents’

achievementtaskvaluesandexpectancy-relatedbeliefs.PersonalityandSocialPsychologyBulletin,21(3),215-225.doi:10.1177/0146167295213003

Hopper,R.(2005).Whatareteenagersreading?Adolescentfictionreadinghabitsandreadingchoices.Literacy39(3),113-120.doi:10.1111/j.1467-9345.2005.00409.xc

Mokhtari,K.&Reichard,C.A.(2002).Assessingstudents‘metacognitiveawarenessofreadingstrategies.JournalofEducationalPsychology,94(2),249-259.doi:10.1037/0022-0663.94.2.249

Wigfield,A.,&Guthrie,J.T.(1997).Relationsofchildren’smotivationforreadingtotheamountandbreadthoftheirreading.JournalofEducationalPsychology,89(3),420-432.doi:10.1037/0022-0663.89.3.420

Page 41: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

39SRCohortIIEvaluationReport:Kentucky

Page 42: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

41SRCohortIIEvaluationReport:Kentucky

Page 43: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

40SRCohortIIEvaluationReport:Kentucky

Page 44: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

42SRCohortIIEvaluationReport:Kentucky

Page 45: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

43SRCohortIIEvaluationReport:Kentucky

Page 46: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

44SRCohortIIEvaluationReport:Kentucky

Page 47: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

45SRCohortIIEvaluationReport:Kentucky

Page 48: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

46SRCohortIIEvaluationReport:Kentucky

Page 49: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

47SRCohortIIEvaluationReport:Kentucky

Page 50: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

49SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

TeacherObservationProtocol

TeacherFirstName:

TeacherLastName:_

School:

Date(includingdayoftheweek):_

Time:_

Observer:_

Notes:

Page 51: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

48SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

Page 52: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

51SRCohortIIEvaluationReport:Kentucky

Page 53: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

50SRCohortIIEvaluationReport:Kentucky

Page 54: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

51SRCohortIIEvaluationReport:Kentucky

Page 55: Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort II: Kentucky Executive Summary of Findings: Implementation and Impact This Striving

Cantrell,Carter,&Rintamaa(2012)

52SRCohortIIEvaluationReport:Kentucky