Striving Readers Cohort II Evaluation Report: Kentucky · 2018-03-26 · Striving Readers Cohort...

Preview:

Citation preview

0antrell,Carter&Rintamaa,2012

C

2012

SusanC. Cantrell,Janis

C. Carter,and

MargaretRintamaa

CollaborativeCenterfor

LiteracyDevelopment

UniversityofKentucky

March,2012

StrivingReadersCohortIIEvaluation Report:

Kentucky

1SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

Acknowledgments

Theauthorsgratefullyacknowledgethecontributionsofthemanyindividualswhofacilitatedthisevaluation. ThisevaluationcouldnothavebeensuccessfulwithoutthesupportofCindyParker,RobinHebert,MaryRuddandMonicaOsbornefromtheKentuckyDepartmentofEducation(KDE)andadministratorsandstaffinthenineStrivingReadersschools. WewanttothanktheStrivingReadersinterventionistswhoworkdiligentlywithstudentsandteacherseachdayandtowhomweturnregularlyforinformationtofurtherthiswork. Furthermore,wegratefullyacknowledgethemanystudentswhoparticipatedinthisstudy.

Additionalmembersoftheevaluationteamwhodeservecreditfortheircontributions

tothisreportincludethefollowingindividualstowhomweareimmenselygrateful:

DataAnalystDeepshikhaSigdel

ResearchAssistantsLauraDudneyAmandaGoodwinJessicaGreweMelissaMurphyJenniePahl

StaffSupportElisha ComerJudyJohnson

Finally,weextendourgratitudetothestaffoftheCollaborativeCenterforLiteracyDevelopment(CCLD)fortheirongoingsupportofthiswork,AbtAssociatesInc.forthetechnicalassistancetheyprovidedduringthestudy,andtheU.S.DepartmentofEducationforfundingthisproject.

2SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

TableofContents

ExecutiveSummaryofFindings:ImplementationandImpact……………………………………. 4KCLMTargetedIntervention…………………….……………………………………………………………….. 5

Implementation….……………………………………………………………………………………………….. 5Impacts.…………………………………………………………………………………………………………….... 5

IntroductionandStudyBackground…………………………………………………………………………….. 5DescriptionoftheInterventionModel….…………………………………………………………………… 5FrameworkoftheSupplementalLiteracyInterventionClass..………………………………….. 6ComponentsofKCLM.……………………………………………………………………………………………… 7TargetedStudents…………………………………………………………………………………………………….. 8SelectionProcessforInterventionists……………………………………………………………………….. 9DesiredCharacteristicsoftheInterventionClassroom……………………………………………… 9LogicModelforKentuckyCognitiveLiteracyModelIntervention………………………………. 10PlannedTrainingModel…………………………………………………………………………………………….. 11PlannedClassroomInstructionalModel…………………………………………………………………….. 12PlannedExperiencesforControlStudents…………………………………………………………………. 12UnitPlanningTemplate…………………………………………………………………………………………….. 13KeyEvaluationDesignFeatures…………………………………………………………………………………. 14

EvaluationoftheImplementation…………………………………………….….……………………………… 14SummaryoftheDesignoftheImplementationStudy.…..……………………………………….…. 14ImplementationDataCollectionandAnalysis..…………………………………………………………. 15

SummerTraining…………………………………………………………………………………………………. 15CoachMentoring…………………………………………………..……..……………………………………… 15ClassroomInstruction…………………....………………………..……………………………………….... 15

KCLMImplementationResults……………………………………..…………………………………………….. 16CharacteristicsofInterventionists……………………………………………………………………..……… 16ImplementationofProfessionalDevelopmentModel…………………….………………………… 16

Interventioniststraining….………………………………………………………………………………….. 16Professionaldevelopmentinputs………………………………………………………………………… 17Professionaldevelopmentforadministrators……………………………………………………… 18

ImplementationofClassroomModel……………………………………………………………………….. 18Classsize,intensity,andduration……………………………………………………………………….. 18Classroomimplementationresults……………………………………………………………………… 18Experiencesforcontrolstudentsduringinterventionperiod………………………………. 20Additionalreadingprograms………………………………………………………………………………. 20Implicationsforimpactanalysis…………………………………………………………………………… 20

EvaluationofImpact……………………………………….…………………………………………………………… 21StudyDesign……………………………………………………………………………………….…………………….. 21

3SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

Samplingselectionprocess…...……………………………………………………….……………………. 21Samplesize……………….……………………………………………………………………………………….. 21ConsortDiagramReadingAchievement…………………………………………………..………… 22ConsortDiagramWritingAchievement……………………………………………………………… 23ConsortDiagramStudentEfficacyandMotivation….…………………………………………… 24MetacognitiveAwarenessofReadingStrategiesInventory(MARSI).…………….…….. 25AdolescentMotivationSurvey………………...…………………………………………………….……. 25

SummaryofAnalyticApproach………………...………………………………………………………………. 26Level-1HLM:StudentLevel………………..……………………………………………………………….. 26Level-2HLM:SchoolLevel………………………………………………………………………..………… 26

DescriptionoftheFirstYearSample………….……………………………………………………………… 26ImpactsonStudents………………..………………………………………………………………………………… 27

DiscussionandConclusions……………………………………………………………..…………………………… 29References………………………….………………………………………………………………………………………. 30

APPENDICESA:SummaryofAnalyticProcedures………………………………………………………………………….. 36B:StudyMeasures…………………………………………………………………………………………………….. 38

StudentSurvey…………………………………………………………………………………………………….. 39TeacherObservationProtocol……………………………………………………………………………… 48

EXHIBITS

1:ConsortDiagramReadingAchievement…………………………………………………..……………. 222:ConsortDiagramWritingAchievement…………………………………………………………………. 233:ConsortDiagramStudentEfficacyandMotivation….…………………………………………….. 24

FIGURES

1:ComponentsofKCLM……………………………………………………………………………………………. 72:KCLMInterventionLogicModel…………………………………………………………………………………… 103:UnitPlanningTemplate…………………………………………………………………………………………. 13

TABLES

1.1FrameworkoftheSupplementalLiteracyInterventionClass…..…………………………… 61.2PlannedPDActivities…………………………………………………………………....……………………. 112.1 Contentanddeliveryofprofessionaldevelopmenttraining(KCLM)……….…………… 172.2 MostWidelyImplementedModelComponents……………………………………….……….. 192.3 MeanScoresbyTeacherforQualityofKCLMcomponent....................................... 202.4 InterventionandControlStudentDemographics(andProportions)....................... 272.5 ImpactoftheTargetInterventiononStudentReadingAchievement,WritingAchievement,

StrategyUseSelfEfficacy,andMotivation…….……………………………… 28

4SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

StrivingReadersCohortII: Kentucky

ExecutiveSummaryofFindings:ImplementationandImpact

ThisStrivingReadersevaluationexaminedtheimpactofatargetedinterventionforstrugglingadolescentreadersinparticipatingschools. Thisstudywasconductedinninehighschoolsinnineschooldistrictsservinglargepercentagesofat-riskstudentsinKentucky. ThetargetedinterventionforstrugglingreaderswastheKentuckyCognitiveLiteracyModel(KCLM)developedbytheKentuckyDepartmentofEducation.

TheeffectivenessoftheKCLMinterventionwasdeterminedthrougharandomized

controlfieldtrialutilizingatreatmentandcontrolgroupdesign.TheKCLMwasasupplementtotheregularcurriculumwhereinstudentsinthetargetedinterventionparticipatedinareadingclassinplaceofanelectiveaspartoftheirregularschoolday. Thecontrol-groupconditionwas“businessasusual,”whereinstudentsinthecontrolgrouptakearegularelectivesuchasband,theaterarts,civics,orphysicaleducation. ThisstudyexaminedtheimplementationofKCLManditsimpactonstrugglingninth-gradestudents’readingandwritingachievement,self-efficacywithreadingstrategies,andmotivationforreading.

InthisStrivingReadersproject,eachschoolemployedaninterventionteacherwhowas

responsibleforteachingthetargetedinterventiontostrugglingreaders.Theimpactresearchquestionsthatmotivatedthestudydesignandanalysisplanare:

• WhatistheimpactofKentucky’sCognitiveLiteracyModel(KCLM)onthereading

achievementoflow-achievingreaders?• WhatistheimpactofKCLMonthewritingachievementoflowachievingreaders?• WhatistheimpactofKCLMontheperceivedreadingstrategyuseoflowachieving

readers?• WhatistheimpactofKCLMonthemotivationandengagementoflowachieving

readers?Inaddition,thefollowingimplementationquestionswillbeanswered:

• What is the state-level implementation of theprofessional development training and

supportforinterventionteachersintheproject?• WhichcomponentsofKCLMwereimplementedmostfrequentlybyteachersin

classrooms?• WhatwasthequalityofKCLMimplementationinclassrooms?

5SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

KCLMTargetedIntervention

ThisstudyexaminedboththeimplementationandimpactoftheKCLMoverthecourseofthefirstyearoftheStrivingReadersproject.

Implementation. Duringtheyearoftheproject,KCLMteachersparticipatedintrainingandon-sitesupport,andtrainingwasprovidedforschooladministrators. Duringtheyear,KCLMteacherswereprovided11daysprofessionaldevelopmenttrainingandupto66hourssupportfromvisitsbyKDEliteracystaff. Schooladministratorswereprovidedonedayoftrainingandatleasttwosupportmeetingsregardingtheinterventionfortheyear. Overall,participationintheprofessionaldevelopmenttheKCLMteachersandadministratorswashigh,with100%ofteachersandadministratorsparticipatingfullyinthetrainingeitherthroughtheformaltrainingdatesorthroughmakeuptraining. Classroomimplementationfidelitywasmeasuredthroughclassroomobservations. ObservationsindicatedthatteachersimplementedsomecomponentsoftheKCLMmodelmorereadilythanothercomponents.Theinterventionwasimplementedwithadequatequalityin3of9classrooms.

Impacts. Thisstudyusedhierarchicallinearmodeling(HLM)analysestomeasuretheimpactsoftheKCLMonninth-grade(highschool)students’readingachievement,writingachievement,perceivedstrategyuse,andmotivation.InthisStrivingReadersstudy,therewerenoimpactsonstudents’readingachievementasmeasuredbytheGroupReadingAssessmentandDiagnosticEvaluationandnoimpactsonwritingachievementasmeasuredbytheKentuckyStateWritingAssessment. Studentsurveyresultsindicatedsignificanteffectsoftheinterventiononparticipatingstudents’self-efficacyforstrategyuseandonstudents’readingmotivation.

IntroductionandStudyBackground

DescriptionoftheInterventionModel

ThetargetedinterventionfortheKentuckyprojectwastheKCLM,developedbytheKentuckyDepartmentofEducation(KDE). ThepurposeoftheKCLMwastoassiststudentswhoweresignificantlybehindgradelevelinreadingbyprovidingthemwiththesupportstobesuccessfulinlearningacrossthecurriculum. Table1.1illustratesthestrandsoftheintervention.

6SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

Table1.1

FrameworkoftheSupplementalLiteracyInterventionClass

Motivationand

Engagement• Thematicinstructionwithproject/inquiry-basedlearningproduces

studentswhoaremorefullyengagedandmotivatedtolearn.• Literacyisasocialaccomplishment(Bloome,1986;Dyson,1992).

Strategic

Processing• Strategy—adeliberatecognitiveprocessofselecting,enacting,

monitoringandregulatingbehavior. Includescomprehensionstrategy

instructionandfoundationalreadingskills.• Skill—amentalactivitythatcanbeappliedtospecificlearningsituations.• Metacognition—keytostrategicprocessingbecauseitenablesstudents

tomonitorprogresstowardachievingtheirgoals(Flavell,1979).Instructional

Strategiesfor

ContentLearning

• RobertMarzano’scharacteristicsofeffectivevocabularyinstruction• Marzano,etal.strategiesforlearningsuchascues,questions,and

advanceorganizers;non-linguisticrepresentations;identifyingsimilaritiesanddifferences;summarizingandnotetaking

Communication

Skills• Includesreading,writinganddiscussionoutcomesthataddressavariety

ofapproachestoessentialquestionsandtexts.• Writinghelpsreadersclarifymeaningandprovidesopportunitiesfor

authenticengagementandcommunication.• Exchangingideas,especiallythroughextendeddiscussionofmeaning

andinterpretationoftext,isessentialtoalearningcommunity.

Itisimportanttonotethatwhilethesearecategorizedintospecificstrands,theyalsoareembeddedacrossstrandstointegrateatotalliteracyexperienceforstudents.ThefourcomponentsoftheKCLMframeworkweretiedtogetherbycontent-relatedthemessuchassuccess,theenvironment,andproblemsolving.

7SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

Figure1showsthemajorcoursecomponentsandspecificdimensionsofeachcomponent.

SupplementalInterventionModelComponentsMotivation&Engagement

StrategicProcesses InstructionalStrategies

Communication

Theme-basedInstructionConnectionsbetweenlearningactivitiesandrealworldissuesVariedinstructionalformatStudentpriorknowledge,interestandbackgroundusedindeterminingcontentTechnologyisusedtofacilitatelearningGoalsettingandprivatefeedbackFocusonproblem-solvingprocessesAutonomyforlearningandmeaningfulchoices

Explicitcomprehensionstrategyinstruction(modeling,explanation,practice,andreflection)Explicitinstructioninfoundationalreadingskills

Instructionaltoolsforcomprehensionandvocabularylearning.Vocabularyprocessingthroughvisual,auditory,physicaland/oremotionalexperiences;opportunitiestousetheirownwordsornon-linguisticrepresentationstodefinenewwords;teacherexplanationsandexamplesofnew,keytermsAfocusonsummarizingandidentifyingsimilaritiesanddifferencesHigherlevelquestioning

Pre-,during-,andpost-textbaseddiscussionstrategiesWritingtolearnactivitiesExplicitinstructioninwritingstrategiesExplicitinstructioninfoundational writingskills

Figure1:ComponentsofKCLM.

8SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter&Rintamaa,2012

SRCohortIIEvaluationReport:Kentucky

IntheKentuckyproject,ninth-gradestudentswhoscoredtwogradelevelsormorebelowgradelevelinreadingreceivedaminimumof225minutesperweekandamaximumof375minutesperweekofsupplementalreadinginstructioninatargetedinterventionclasstaughtbyaninterventionteacher. Studentswereplacedinthiscourseinadditiontotheirregularreading/languageartsclassesforanentireschoolyear.

Overthecourseoftheproject,theprofessionaldevelopmentmodelforthetargetedinterventionincludedsummerandfollow-uptrainingsandon-sitesupportfromKDEliteracystaff.Tolearnhowtoimplementthetargetedintervention,teachersparticipatedinasummerworkshop,whichwasledbyKDEliteracystaff. Duringtheschoolyear,thetrainersledtheteachersinfollow-upworkshops. Acrosstheproject,KCLMteachersreceived11daysofworkshoptraininginthetargetedinterventionintotal. Tosupporttheirongoinglearninganddevelopment,teachersalsoparticipatedinsitevisitsandregulardistancesupportbyKDEliteracystaff.

Theprofessionaldevelopmentmodelincludedtrainingandsupportforadministrators,aswell. SchooladministratorsattendedaonedaymeetinginthesummertolearnabouttheKCLMmodelandthewaysinwhichtheschoolsshouldsupporttheintervention. Additionally,KDEliteracystaffparticipatedinon-sitemeetingswithadministratorsuptosixtimesduringtheschoolyear.Topicsatthosemeetingsincludedgrantrequirements,evidence-basedcomponentsofthetargetedintervention,schedulingissues,observationsofinterventionists,literacyleadership,literacyplanning,andmeetingtheneedsofstrugglingadolescentlearners. AdministratorsalsoreceivedKCLMupdatenewslettersfromKDEfourtimesduringthecourseoftheyear.TargetedStudents

Thisprojectwasdesignedtoservelowachievingreadersintheninthgrade. Inthespringoftheireighth-gradeyear,studentsinmiddleschoolsthatfeedintothenineparticipatinghighschoolsweregiventhemiddleschoolformofthespringGroupReadingAssessmentandDiagnosticEvaluation(GRADE). StudentsthatscoredanNCEof40orlowerontheGRADEweredefinedaslowachievingreaders,andwereplacedintheeligibilitypool.

Studentsinalldayresourceclasseswerenoteligible;allotherstudentswereeligible. Theevaluationteamdirectedthefacultyatallfeedermiddlesschoolstoidentifystudentsinalldayresourceclasses,andinstructedtheschoolsnottogivethespringGRADEtothesestudents. Inaddition,evaluatorsdirectedthefacultyatthenineparticipatinghighschoolstoidentifystudentsthatwereplacedinalldayresourceclassesaftertheyenrolledinninth-gradethisfallandaskedthemtoprovideevaluatorswiththenamesofthesestudentsforremovalfromthestudy.Finally,middleschoolssenthomepassiveconsentformswithalleighth-gradestudentswhowouldbetestedforparticipationinthestudy. Thestudywasdescribed,andparentsweredirectedtocontacttheevaluatorsiftheychosetonotallowtheirstudenttoparticipateinthestudy. Noparentdeclinedpermissionfortheirchild’sparticipationinthestudy.

Twothousandtwohundredfourstudentswerelistedontheschoolregistersheetsasenrolledatthefeedermiddleschools. Evaluatorsreceivedanadditionaltwenty-fivestudent

Cantrell,Carter,&Rintamaa(2012)

8

9SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

GRADEsheetsthatwerenotontheenrollmentlists.Therewere637studentsthatmetthestudy’scriteriononthespringGRADEforeligibility.SelectionProcessforInterventionists

Interventionteacherswererecruitedandhiredbyindividualschools. Advertisementsfortheinterventionteacherpositionincludedthefollowingcriteria: experiencedclassroomteacher,respectedbyfacultyandadministration;familiaritywithand/orinterestininterdisciplinaryandproject-basedlearning;willingnesstolearnandapplynewskillsandknowledge;planning/reflectingskills;strongleadershipability;adaptabilityandproblemsolvingskills;presentationskills;collaborationskills;abilitytomediatebetweentheschoolandcommunityorganizations;andpersonalcommunicationskills.DesiredCharacteristicsoftheInterventionClassroom

Classesweretobenolargerthantwentyninth-gradestudents. Theinterventionclasswastomeetdailyforatleast45minutesfortheentireyear.

10SRCohortIIEvaluationReport:Kentucky

SRCohortIIEvaluationReport:Kentucky

Figure2:KCLMInterventionLogicModel Cantrell,Carter&Rintamaa,2012

10

SRCohortIIEvaluationReport:Kentucky

PlannedTrainingModel

Duringthesummerof2010,theinterventionteachersfromeachschoolweretoreceivefivedaysofinitialtrainingonthecorestrands. ThistrainingwastobedevelopedandprovidedbytheKentuckyDepartmentofEducationliteracystaff. Schooladministratorswererequiredtoattendonedayofsummertrainingtolearnaboutthegoalsoftheinterventionandtheexpectationsforsupportoftheinterventionistandstudents. Bytheendoftheinitialtraining,theinterventionistsweretocreateaninstructionalplanfordirectstrategyinstruction,andintegrationofstrategiesintostandards-basedunitsofstudy. Eachparticipantwastoleaveunderstandinghowtoteach,use,andassesswithintheKCLMframework,andwithaplantobeginimmediateimplementationasthe2010-2011schoolyearbegan.

Throughouttheyearofimplementation,theinterventionistsweretoreceiveon-going

trainingandsupportthroughface-to-facemeetingsandanonlineKentuckyVirtualSchools(KYVS)learningcommunity,whichwastoincludebookstudies,Webinars,adiscussionforum,andrepositoriesforsharingresources.KDEliteracystaffplannedsitevisitstotheschoolsinthefallandagaininthespringtoprovideon-sitecoachingandsupportforupto50additionalhours.

Administratorsweretoengageinprofessionaldevelopmentfocusingontheguidefrom

theNationalAssociationofSecondarySchoolPrincipals,CreatingaCultureofLiteracy:AGuideforMiddleandHighSchoolPrincipalsaswellasaKDEDVDresource,LiteracyLeadership:StoriesofSchoolwideSuccess.Thissharedleadershipnetworkwasintendedtohelpadministratorsandcoachesdevelopacloseworkingrelationshipcriticalinsupportinginstructionalimprovementefforts.

Table1.2

PlannedPDActivities

PDhrs. Activities Date Attendees*40 Trainingoncorestrandsofintervention;develop

instructionalplanandunitsofstudy8 Trainingongoalsforinterventionandon

providingsupporttointerventionists50+ OngoingPD: Onlinelearningcommunity,

includingbookstudies,webinars,discussionforum,sharingnetwork

July2010 I

July2010 A

2010 I

16 Onsitevisits/coaching 2010 I

*I=Interventionist;A=Administrator

Cantrell,Carter&Rintamaa,2012

11

SRCohortIIEvaluationReport:KentuckyPlannedClassroomInstructionalModel

TheKentuckyDepartmentofEducationcreatedaUnitPlanningTemplate(UPT)toassistinterventionteachersincreatingunitsofinstructionfortheinterventionclass. OnecomponentoftheUPTlistedthestrategiesandactivitiesthatcomprisetheKCLM.Teachersselectedstrategiestoteachineachunit.Figure3showsthestrategiesandactivitiesaslistedontheUPT.Teacherswerefreetoselectthestrategiestheytaughtbasedontheirassessmentofstudents’needs.Aspartoftheinterventionclass,studentsassignedtotheinterventionwilltaketheSAT-10online(reading)andTOWL-4(writing)diagnosticassessmentstwiceperyear.

Inaddition,KCLMisdesignedtoengagestudentsinreadingavarietyoflevel-appropriatetextsrelatedtoessentialquestion(s)foreachcontent-relatedunit,includingnonfiction,informationalandproceduraldocuments,andnarrativetexts.PlannedExperiencesforControlStudents

Expectationsforthecontrolstudentswere“businessasusual,”whereinstudentsinthecontrolgrouptookaregularelectivesuchasband,theaterarts,civics,orphysicaleducation.

Cantrell,Carter,&Rintamaa(2012)

12

SRCohortIIEvaluationReport:Kentucky

Figure3:UnitPlanningTemplate

Cantrell,Carter&Rintamaa,2012

13

SRCohortIIEvaluationReport:KentuckyKeyEvaluationDesignFeatures

Theevaluationisdesignedtomeasuretheimpactofthetargetedinterventiononstudentoutcomesteacherefficacy. Theimpactstudywasguidedbythefollowingresearchquestions:Theimpactresearchquestionsthatmotivatedthestudydesignandanalysisplanare:

• WhatistheimpactofKentucky’sCognitiveLiteracyModel(KCLM)onthereadingachievementoflowachievingreaders?

• WhatistheimpactofKCLMonthewritingachievementoflowachievingreaders?• WhatistheimpactofKCLMontheperceivedreadingstrategyuseoflowachieving

readers?• WhatistheimpactofKCLMonthemotivationandengagementoflowachievingreaders?

Studentoutcomemeasuresareasfollows:

• GroupReadingAssessmentandDiagnosticEvaluation(GRADE)• TheKentuckyStateWritingAssessment• MetacognitiveAwarenessofReadingStrategiesInventory(MARSI)• StudentMotivationSurvey

Inaddition,thefollowingimplementationquestionswereanswered:

• Whatisthestate-levelimplementationoftheprofessionaldevelopmenttrainingandsupportforinterventionteachersintheproject?

• WhichcomponentsofKCLMwereimplementedmostfrequentlybyteachersinclassrooms?

• WhatwasthequalityofKCLMimplementationinclassrooms?

EvaluationofImplementation

SummaryoftheDesignoftheImplementationStudy

Theresearchquestionsthatguidedtheimplementationstudyofthetargetedinterventionare:

• Whatisthestate-levelimplementationoftheprofessionaldevelopmenttrainingandsupportforinterventionteachersintheproject?

• WhichcomponentsofKCLMwereimplementedmostfrequentlybyteachersinclassrooms?

• WhatwasthequalityofKCLMimplementationinclassrooms?

Cantrell,Carter&Rintamaa,2012

14

15SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

ImplementationDataCollectionandAnalysis

Summertraining.Duringthesummerof2010,theselectedinterventionteachersfromeachschoolreceivedfivedaysofinitialtrainingonthecorestrands. ThistrainingwasdevelopedandprovidedbytheKentuckyDepartmentofEducationliteracyconsultants.Attendancerecordswerekeptateachtrainingsession,andindividualteacherattendancewascomputedattheendofthetrainingintermsofpercentageofdaysattended.Schooladministratorswillberequiredtoattendonedayofsummertrainingtolearnaboutthegoalsoftheinterventionandtheexpectationsforsupportoftheinterventionistandstudents. Participationwillbeassessedandscoredforadequacyandfidelitythroughattendancerecordsprovidedbythedevelopers.

Researchassistantsattendedeachtrainingsessionandtookdetailedfieldnotesinfiveminuteintervals. AcodelistwasdevelopedrelatedtoContentandDelivery.ContentcodesrelatedtothekeycomponentsofKCLM.Deliverycodesrelatedtovarioustrainingformatssuchaswholegroup,smallgroup,discussion,andlecture.Toestablishreliability,threeresearchassistantsindependentlycodedonedayoffieldnotes.Agreementwas90%forcontentand90%fordelivery.Afterthecodingtheresearchassistantsagreedthatsomeofthecodesshouldbefurtherbrokendownforamoreaccuratedescriptionofwhatwasgoingonduringthetimeinterval.Researchassistantsandinvestigatorsdiscussedaddingadditionalcodesforcontentanddelivery.Thefollowingcodeswereaddedtocontent:Collaboration,Planning-working,andPlanning-discussion.Thefollowingcodeswereaddedtodelivery:Wholegroup-lectureandWholegroup-discussion.Theresearchassistantscontinuedindependentlycodingthenextfourdaysoftheinterventionteachertraining.Researchassistantsmetagaintocheckagreement.Agreementachievedforcontentwas95%and100%agreementfordelivery.

Coachingandmentoring.Interventionistsreceivedon-goingtrainingandsupportthroughanonlinelearningcommunitythatincludedbookstudies,Webinars,adiscussionforum,andrepositoriesforsharingresources.KDEconsultantsmadesitevisitstotheschoolsinthefallandagaininthespringtoprovideon-sitecoachingandsupport. Participationwasassessedandscoredforadequacyandfidelitythroughattendancerecordsprovidedbythedevelopers.

Classroominstruction.Implementationfidelityforthetreatmentconditionwasestablishedthroughclassroomobservationsusingastandardizedobservationprotocolforinterventionclasses. Readinginterventionteacherswereobservedtwiceduringtheyear. Thestandardizedobservationprotocolforinterventionteachersincludedtwocomponents:achecklistofessentialKCLMcomponentsandaqualityrubricforassessingteachers’implementationquality.Inthefall2010,tworesearchassistantsmettocreatealistofmodelcomponentsbasedonthetrainingsfromKDE. ThislistwassenttoKDEforvalidation. KDEaddedafewcomponentstomakethelistcomplete. Toidentifythemodelfeaturesthatweremostcriticaltoprogramsuccess,evaluatorsaskedthedeveloperstorankeachsetoffeaturesforeachcomponentinorderofimportance. Toconfirmtheserankings,evaluatorssoughtfeedbackfromthreeexpertscholarswhoconfirmedandelaboratedontheshortenedlistforeachfeature.TheleadevaluatorandKCLMtrainerviewedtwovideosofKCLMinstructionandindependentlycompletedtheprotocol.Then,theydiscussedtheirscores,reachedconsensusonscoring,andmademinorrevisionsto

16SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

somewordingontheprotocol.Fortraininginusingtheprotocol,researchassistantsviewedonevideo,ratedtheinstructionusingtheprotocol,anddiscusseddisagreements. Toestablishinter-observeragreement,theresearchassistantsviewedthesecondvideoandindependentlycompletedaprotocol. AgreementwiththeleadevaluatoronthechecklistofkeyaspectsofKCLMcomponentsforthisprotocolwas74.4%.Thelargestareaofdisagreementwasrelatedtovocabularyinstructionunderthe“InstructionalStrategies”component. Theleadevaluatorprovidedadditionaltrainingaroundvocabularyinstructiontoclarifymisunderstandings.Agreementonthequalitydescriptorsforthesecondvideowas91.7%.

Itwasexpectedthatteacherswouldimplementsomeaspectofeachforthefourmodelcomponents(motivationandengagement,strategicprocesses,instructionalstrategies,andcommunication)duringeachclassperiodbutitwasnotexpectedthatteacherswouldimplementeveryaspectofeachcomponenteachclassperiod.Theprotocolyieldedinformationonwhichaspectsofeachcomponentwereimplementedduringobservationsandwhetherthequalityofimplementationwas(a)developing,(b)adequate,or(c)exemplary.Percentagesofobservationsthatincludedeachcomponentwerecomputed,andtheproportionofobservationsratedateachqualitylevelwasprovided.

KCLMImplementationResultsCharacteristicsofInterventionists

Nineinterventionistpositionswerefilledby11teachersduringtheyear. Interventionteacherswereselectedandhiredbyindividualschooldistrictsandmettheplannedcharacteristics,accordingtoschoolpersonnel. Atotalofnineinterventionistswerehiredinthesummerof2010. Duringthecourseoftheschoolyear,twointerventionistsleftandwerereplaced,foratotalof11interventionists.TheseinterventionistsimplementedtheKCLMtargetedinterventionswithinthenineschools.Oneinterventionistwasmaleand10werefemale.Allinterventionistswerewhite.Alloftheinterventionistshadamastersdegreeorhigher,andthree(27%)werecertifiedasreadingspecialists.Interventionistshadanaverageof12.9yearsofexperience.ImplementationofProfessionalDevelopmentModel

Interventionisttraining.Trainingwasprovidedtointerventionistsforfivedaysinthesummer,andthreedaysduringtheschoolyear. Table2.1showsthecontentamountanddeliveryoftheprofessionaldevelopmentmodel.

17SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

Table2.1Contentanddeliveryofprofessionaldevelopmenttraining(KCLM)

Numberof Minutestotal

Percentoftraining

Content Housekeeping 70 2.77Overviewofgrant 220 8.71Collaboration 275 10.89Readingstrategies&Strategicprocessing 640 25.30Communication 175 6.93Motivationandbehaviormanagement 225 8.91Foundationalreading–basicskills 60 2.38Assessment 360 14.26Planningwork 280 11.09Planningdiscussion 115 4.55Projectbasedlearning 110 4.36

Delivery

FormatWholegrouplecture 790 31.23Wholegroupdiscussion 1035 40.91Smallgroup 340 13.44Individual 365 14.43

Professionaldevelopmentinputs.Duringthesummertraining,sevenofthenineinterventionistsattendedallfivedays. Oneinterventionistattendedfourdaysandwasabsentonedayduetoillness. Thismaterialwascoveredwiththeabsenteeduringthefollowingday’strainingduringlunchandbreaks. Anotherinterventionistdidnotattendthesummertraining.Thatinterventionistwastrainedone-on-oneduringthreedaysattheinterventionist’sschool.Duringtheschoolyeartherewerethreeregulartrainingdates. Allinterventionistsattendedtwoofthesedates,andeightattendedthethird. TheabsentinterventionistmadeupthemissedtrainingdateviaSkypewithKDEstaff. InterventionistsalsoattendedtheKentuckyReadingAssociationconference. Allnineinterventionistsattendedallthreedaysoftheconference.Attendanceandparticipationwasadequateforallinterventionists.

Inadditiontosummertraining,interventionistsreceivedsitevisitsupportfromKDEstaff

throughouttheyear. Thenumberofvisitsrangedfromfourdaysto11daysperinterventionistatsixhoursperdaywithanaverageof5.56days. Whenextrasupportwasneeded,additionalvisitswerescheduled.Itemsdiscussedduringthevisitswerestrategyimplementation,project-basedlearning,datacollection,studentengagementandmotivation,assessmentproceduresandothervarioustopicsdependingontypeofsupportneededrelatedtotheKCLMmodel.

InterventionistsreceivedsupportthroughtheuseofaNING,Skype,emailandphonecalls.

Phonecalls,emailandSkypewereusedforongoingsupportandprofessionaldevelopment. The

18SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

NINGwasusedforbookstudies,discussionforums,bloggingandsharingresources.KDEstaffestimatethateachinterventionistwassupportedanestimatedthreetimesperweekviaphonecallsandemails,anestimatedonceperweekviaNING,andoncepermonthviaSkype.

Allparticipationintheprofessionaldevelopmenttrainingandsupportwasconsideredtobeadequateforallinterventionists.

Professionaldevelopmentforadministrators. Administratorswereprovidedwithonetrainingdayinthesummerof2010todiscusstherequirementsofthegrantandwaystheycouldsupporttheintervention. Administratorsfromallnineschoolsattended.

Duringtheschoolyear,KDEstaffmetwithschooladministratorstodiscussgrantrequirements,evidence-basedcomponentsofthetargetedintervention,schedulingissues,observationsofinterventionists,literacyleadership,literacyplanning,andmeetingtheneedsofadolescentlearners—especiallylowachievingstudents. Thenumberofmeetingsrangedfromtwotosixperschoolwithanaverageof3.11meetingsperschool. KDEstaffalsosharedaKCLMUpdatenewsletterwithalladministratorsfourtimesthroughouttheyear. Alladministratorparticipationwasconsideredtobeadequate.ImplementationofClassroomModel

Classsize,intensity,andduration. Classsizesvariedthroughouttheyearduetostudentattrition(i.e.,transferringschools,droppingout,etc.). Classesrangedfromasfewas12studentstoasmanyas20.Theinterventionclassmeteverydaythroughoutthecourseoftheyear,andrangedfrom45minutesto75minutesdaily.

ClassroomImplementationResults. TheobservationprotocolincludedratingsforbothpresenceofindicatorsforeachcomponentforKCLM,andqualityofeachcomponentofKCLM.Thenextfourtablesshowthenumberoftimesinterventionistswereobservedincorporatingindicatorsofmotivationandengagement,strategicprocessing,instructionalstrategies,andcommunicationskillsintheirlessons.Table2.2indicatesthemodelcomponentsmostwidelyimplementedbyteachersandthenumbersandpercentsoflessonsthatincludedthosecomponents.Othermodelcomponentswereobservedinfewerthan50%oflessons.

19SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

Table2.2

MostWidelyImplementedModelComponents

ModelComponent Round1N=18

% Round2 %N=18

Motivation&EngagementTeachermakesconnectionsbetweenlearningactivitiesandrealworldissuesTeachervariesinstructionalformat,i.e.groupwork,lecture,andpartnerworkStudentpriorknowledge,interestandbackgroundusedindeterminingcontent

StrategicProcessingTeacherexplainscognitivestrategiesforcomprehensionTeachermodelsusingcognitivestrategiesforcomprehensionTeacherencouragesandprovidesopportunitiesforstudentstopracticeusingcognitivestrategiesforcomprehension

InstructionalStrategiesTeacherexplicitlyincorporateshigherlevelquestionsTeacherusesinstructionaltoolstosupportstudentcomprehensionorvocabularyTeacherdescribes,explains,andprovidesanexampleofnewkeyterms

CommunicationSkillsTeacherusesduringtext-baseddiscussionstrategiesTeacherincludeswritingtolearnactivities

ForeachKCLMcomponent,interventionistsfocusedonsomecomponentsmorethanothers.Formotivationandengagement,interventionistsweremorelikelytomakeconnectionsbetweenthelessonandrealworldissues,tovaryinstructionalformat,andtotiethecontentintostudentexperiences. Interventionistswerenotaslikelytoincorporatetechnology,providestudentfeedback,facilitateproblemsolvingandprovidestudentswithchoices.Forstrategicprocessing,interventionistsweremorelikelytoexplain,modelandencouragestudentpracticeofcognitivestrategies. Interventionistswerenotaslikelytoaskstudenttoreflectoncognitiveprocessingandreadingcomprehensionorprovideexplicitinstructioninfoundationalreadingskills.Overall,interventionistswerenotaslikelytoincorporateinstructionalstrategiesintotheirlessons. Inthefirstroundofobservations,interventionistsdiduseinstructionaltoolstosupportstudentcomprehensionorvocabulary,andinthesecondroundtheywerelikelytodescribe,

13 72 9 50

9 50 9 50

11 61 12 67

14

78

9

50

12 67 10 56

15 83 13 72

7

39

10

56

12 67 6 33

8 44 11 61

11

61

6

33

15 83 9 50

Cantrell,Carter,&Rintamaa(2012)

20SRCohortIIEvaluationReport:Kentucky

explainandgiveexamplesofnewvocabularywords.Communicationskillswerealsounder-utilizedbyinterventionists. Therewassomeevidenceofteacherusesduringtext-baseddiscussionstrategies,andwritingtolearnactivities.

ClassroomobservationsalsoincludedaqualityscoringforeachKCLMcomponent. Table2.3showseachteacher’squalityscorebyKCLMcomponentforeachroundofobservation. Eachroundrepresentsanaveragebetweenscoresforeachclassperiodobserved.Table2.3showsthemeanscoresbyteacherforqualityofeachKCLMcomponentandtheoverallscoreforthemodel.Intermsofoverallqualityofimplementation,implementationofKCLMwasadequatein3ofthe9classrooms.

Table2.3

MeanScoresbyTeacherforQualityofKCLMcomponent

Classrooms

123456789

Note.1-1.4=Developing,1.5-2=Adequate,2.1-3=Exemplary

Experiencesforcontrolstudentsduringinterventionperiod. Studentswhowereselectedforthecontrolgroupreceivedaregularelectiveaspartoftheirfreshmanprogram.Awiderangeofelectivesweretakenincludingband,chorus,civics,andphysicaleducation.

Additionalreadingprograms.Intwoschools,additionalreadingassistanceprogramswere

providedtostudentswhoqualified. AtoneschoolstudentsfromboththeinterventionandcontrolgroupstookanunstructuredreadingclasswheretheteacherhadaccesstomaterialsfromStudyIslandandDiscoveryEducationaswellasRead180andSystem44.Inthesecondschool,studentsinbothinterventionandcontrolgroupstookaonesemesterclasscalledReadingRevisited,ahighlystructuredclassrelyingheavilyonvocabularyworkbookexercises.

Implicationsforimpactanalysis.Twofactorsrelatedtoimplementationshouldbe

consideredwheninterpretingtheimpactanalysistofollow.First,teacherswerelearningtoimplementtheinterventionastheywereimplementingit,andthisresultedinavariationin

MotivationEngagement

Mean

StrategicProcessingMean

InstructionalStrategiesMean

CommunicationSkillsMean

OverallscoreMean

1 1 1 1 11.5 1.5 1 1.5 1.382.25 2 2 2 2.061 1 1 1 12 2 2 1.5 1.882 2 2 1.5 1.881 1 1 1 11.5 1 1 1 1.131 1 1 1 1

Cantrell,Carter,&Rintamaa(2012)

20SRCohortIIEvaluationReport:Kentucky

implementationquality.Teacherswereadheringtothemajorcomponentsoftheintervention,buttheydidnotnecessarilyimplementthosecomponentsastheyweredesignedtobeimplemented.Therefore,impactfindingsshouldnotbedirectlyattributedtotheinterventionmodelasitisdesigned.Second,thatsomeschoolscontinuedtoimplementreadingprogramstargetedatlowachievingreaderspresentsaconfoundingfactorthatmayhaveinfluencedoutcomesforbothtreatmentandcontrolgroups.

StudyDesign

EvaluationofImpact

Samplingselectionprocess. Evaluatorsimplementedastratifiedrandomsamplingprocedureforstudentswithineachschoolusingfourdemographicvariables: specialeducationstatus,free/reducedlunchstatus,ethnicity,andgender. Withineachschoolstudentsweresortedbydemographicvariablescreatingsubgroupsofstudents,andstudentswithineachsubgroupwerethensortedbyassessmentscore. Usingarandomnumbergeneratortoassignthefirststudenttoeithertheinterventionorcontrolgroup,thestudentswerethenalternatelyassignedsequentiallytotheinterventionorcontrolgroup.

Evaluatorsprovidedalistofstudentsthatqualifiedfortheinterventiongrouptothenineparticipatinghighschoolsduringthesummerof2011. Thesestudentswerescheduledtobeintheinterventionclass,andstudentsandparentswerenotifiedwhentheclassschedulesweregiventoallstudentsatthebeginningofthefallsemester. Schoolswerenotprovidedthenamesofthestudentsinthecontrolgroup.

Astudentwasremovedfromthestudypost-randomassignmentifhe/shedroppedoutofschoolormoved/transferredtoaschoolnotparticipatingintheStrivingReadersprogram.Also,ifthestudentdidnottaketheposttestinthespringof9thgrade,theywereremoved.Finally,astudentwasineligiblepost-randomassignmentif,afterthestudentenrolledintheninth-grade,thehighschoolplacesthestudentinalldayresourceclasses. Theschoolswereinstructedtoinformtheevaluatorsattheendofthestudyifacontrolstudentwasplacedinalldayresourceclasses. Thusthecriterionthehighschoolusedtoassignastudenttoalldayresourcewouldnotbeaffectedbytheresultsoftherandomassignment,andwasappliedequallytointerventionandcontrolstudents. Therewere13interventionstudentsthatwereassignedtoalldayresourceclassespost-randomassignment,andareineligibletobeinthestudy. Thereweresixteencontrolstudentswhowereplacedinalldayresourceclassesafterrandomassignment.

Samplesize.Exhibit1showsthesamplesizeresultsforreadingachievementtest,GRADE,dividedbycondition. Outofthepopulationof2,229eighthgradestudents,637wereidentifiedaslowachievingreaders(NCEof40orlower). Threehundrednineteenstudentswereassignedtothetreatmentgroup,and318studentswereassignedtothecontrolgroup.Afterrandomassignment,13studentsinthetreatmentgroupand16studentsinthecontrolgroupwereassignedtoalldayresourceclassesafterenrollinginhighschoolsowereineligibleforthestudy. Additionally,22studentsinthetreatmentgroupdidnotenrollinaparticipating

Cantrell,Carter,&Rintamaa(2012)

22SRCohortIIEvaluationReport:Kentucky

highschool. Thenumberofstudentsinthecontrolgroupthatdidnotenrollinaparticipatinghighschoolisnotavailable.

Therewere306targeted,eligiblestudentsatbaselineintheinterventiongroupand302

studentsinthecontrolgroup. Attritionintheinterventiongroupattheendoftheschoolyeartotaled74students(36studentsmovedand38studentsdidnottakethespringtest),resultinginananalyticsampleof232interventionstudents. Attritioninthecontrolgrouptotaled49students(34studentseitherdidnotenrollintheparticipatinghighschoolormovedduringtheyearand15studentsdidnottakethespringtest),resultinginananalyticsampleof253controlstudents.

Cantrell,Carter,&Rintamaa(2012)

23SRCohortIIEvaluationReport:Kentucky

Exhibit2showsthesamplesizeresultsforthewritingachievement,bycondition. Thetargetednumberofeligiblestudentsatbaselineintheinterventiongroupwas306,with302studentsinthecontrolgroup. Attritionintheinterventiongroupduringtheschoolyeartotaled134students(36studentsmovedand98studentsdidnottakethespringwritingtest),resultinginananalyticsampleof172interventionstudents. Attritioninthecontrolgrouptotaled109students(34studentseitherdidnotenrollintheparticipatinghighschoolormovedduringtheyear,and75studentsdidnotcompletethespringsurvey),resultinginananalyticsampleof193controlstudent.

Cantrell,Carter,&Rintamaa(2012)

24SRCohortIIEvaluationReport:Kentucky

Exhibit3showsthesamplesizeresultsforstudentsurvey,bycondition. Thetargetednumberofeligiblestudentsatbaselineintheinterventiongroupwas306,with302studentsinthecontrolgroup. Attritionintheinterventiongroupduringtheschoolyeartotaled152students(36studentsmovedand116studentsdidnotcompletethespringsurvey),resultinginananalyticsampleof154interventionstudents. Attritioninthecontrolgrouptotaled129students(34studentseitherdidnotenrollintheparticipatinghighschoolormovedduringtheyear,and95studentsdidnotcompletethespringsurvey),resultinginananalyticsampleof173controlstudent

Cantrell,Carter,&Rintamaa(2012)

25SRCohortIIEvaluationReport:Kentucky

ImpactMeasuresandDataCollection

ThefollowingmeasureswereusedtoascertaintheimpactoftheKCLMinterventiononstudents’readingandwritingachievement,readingstrategyuse,andreadingmotivation.

GroupReadingandDiagnosticEvaluation(GRADE).TheGRADEisanationallynormedreadingassessment,anditincludesnormalizedscoresforoverallreadingachievement. NormalCurveEquivalentscores(NCEs)includingvocabularyandcomprehensionitemscloselyalignstudentoutcomeswiththegoalsoftheintervention.Inthespringoftheplanningyear,theGRADE(levelM)wasgiventoalleighth-gradestudentsinparticipatingfeederschoolsasapretest. Inthefollowingspring,theGRADE(levelH)wasgiventoallninth-gradestudentsasapostest.

KentuckyStateWritingAssessment. TheKentuckystatewritingassessmentprovidesaholisticscorebasedonanalyticcategorieswhichcloselyalignstudentoutcomeswiththegoalsoftheintervention. TheStateWritingAssessmentwasgiventoalleighth-gradestudentsinspringoftheplanningyear,andallninth-gradestudentsinspringofthefollowingyear. Thetestisdesignedsothatstudentsgetachoicebetweentwowritingtaskswhichincludethreepossiblemodesofwriting(inform,narrateforapurpose,orpersuade)andfourpossibleresponseformats(article,editorial,letterorspeech). Inadditiontoawritingtask,eachstudentisalsopresentedwithdraftversionsofthreepiecesofwriting. Fourmultiple-choicequestionsdealingwithediting/revisingareprovidedwitheachofthethreedraftsso,eachstudentalsorespondsto12multiple-choicequestions.

MetacognitiveAwarenessofReadingStrategiesInventory(MARSI).TheMARSI(Mokhtari&Reichard,2002)isastudentself-reportmeasuredesignedspecificallytoassessadolescents’perceiveduseofreadingstrategiesduringacademicreading.Forthisstudy,theMARSIwasadaptedtomeasureself-efficacywithstrategyuse.“Ican”wasaddedtothebeginningofeachstrategystatementinthesurvey.Thesurveywasgiventoallstudentsinfallandspringofninthgrade. Thesurveyitemsarepresentedonascaleof1to5,where1isequalto“notatallconfident”and5isequalto“completelyconfident.” Exampleitemswouldbe“IamabletohaveapurposeinmindwhileIread”or“IcantakenoteswhilereadingtohelpmeunderstandwhatIread.”

AdolescentMotivationSurvey. Inthespringofeighthgradeandinthespringofninthgrade,allstudentsinparticipatingschools(andeighth-gradefeederschools)completedasurveymeasuringseveraldimensionsofintrinsicreadingmotivation(challenge,curiosity,intrinsictaskvalue,attainment),extrinsicreadingmotivation(extrinsictaskvalue,compliance),readingrelatedself-beliefs(expectancy,difficulty)andleisurereading.Thesurveyitemsarepresentedonascaleof1to5andthevaluevariesdependingonthedimensionmeasured. Asampleitemforthechallengedimensionis“Ilikehard,challengingbooks”where1equals “notatalltrueand5equals“verytrue.” Asampleitemfromtheleisuredimensionis“Howmuchtimehaveyouspentreadingamagazinethisweek?”with1equaling“none,”and5equaling“morethanfivehours.” Reliabilityforsubscaleswere>.70withtheexceptionofleisure

Cantrell,Carter,&Rintamaa(2012)

26SRCohortIIEvaluationReport:Kentucky

(a=.585)andextrinsictaskvalue(a=.60).Thesescalesweretakenfromexistingmeasures(Eccles&Wigfield,1995;Hopper,2005;Wigfield&Guthrie,1997).SummaryofAnalyticApproach

HierarchicalLinearModels(HLMs)wereusedtoestimatetheimpactoftheKCLMonstudentachievement,motivation,andreadingstrategiesoutcomes.TheGRADENormalCurveEquivalents(NCEs)wereusedtoestimatetheimpactoftheKCLMinterventiononachievement.HolisticscoresfromthestatewritingassessmentwereusedtoestimatetheimpactoftheKCLMonwritingachievement.TheaverageoftheitemsontheMARSIwasusedtoestimatetheimpactonreadingstrategyuse,andtheaverageoftheitemsontheAdolescentMotivationSurveywasusedtoestimatetheimpactonmotivation.

Atwo-levelHLMmodel(studentsassignedtointerventionorcontrolgroupwithinschools)willbeusedtodeterminetheimpactofKCLM.Fourhypotheseswillbetested:

H1: TheKCLMinterventionhasnoimpactonstudentachievement.H2: TheKCLMinterventionhasnoimpactonstudentwritingachievement.H2: TheKCLMinterventionhasnoimpactonselfefficacyofstudentstrategyuse.H3: TheKCLMinterventionhasnoimpactonstudentmotivation.

Level-1HLM:StudentLevel.Atthestudentlevel,thespringoutcomevariable(reading

achievement,writingachievement,strategyuse,ormotivation)willbemodeledasafunctionoffalloutcomevariables(covariate),intervention/controlstatusandfourdemographicvariables:gender,ethnicity,free/reducedlunchstatus,andspecialeducation.

Level-2HLM:SchoolLevel.Thisanalysiswillbeperformedonninth-gradestudents’scoresfromninehighschools. InadditiontothebaseyearReadingKCCTscore,otherschoollevelvariablesthatwillbeincludedaretheschoolpercentofstudentsqualifyingforfreeorreducedlunchfees,schoolpercentofwhitestudentsintheschool,andschoolpercentofblackstudents,andthepercentofstudentswithdisabilities.DescriptionoftheFirstYearSample

Ninehighschoolsgeographicallydistributedacrossthestateparticipatedinthestudy.HighschooldemographicdatawascollectedfromtheKentuckyDepartmentofEducationwebsiteforthe2009-10academicyear. Theaveragenumberof10th-12thgradestudentsenrolledattheparticipatingschoolswas454,rangingfrom301to803students. TheaveragepercentofWhitestudentswas91.58%(53.9%,98.8%),andtheaveragepercentBlackstudentswas5.76%(0%,37.7%). Theaveragepercentofstudentsreceivingfree/reducedlunchwas62.25%(40.7%,83.5%),andtheaveragenumberofstudentsenrolledinspecialeducationclasseswas12.73%,rangingfrom7.2%to16.5%.

Studentdemographicdatawascollectedfromthemiddleschoolsforeveryenrolledeighth-gradestudent.Schoolswerecontactedforstudentdemographicinformationifthere

Cantrell,Carter,&Rintamaa(2012)

27SRCohortIIEvaluationReport:Kentucky

werestudentsthatwerenotontheselistsbutcompletedtheeighth-grade.Table2.4showsdemographiccharacteristicsofstudentsintheinterventionandcontrolgroups.StudentsintheStrivingReadersclassesweretypicallyWhitemales,receivingfree/reducedlunchservicesandarenotassignedtospecialeducationclasses. Studentsthatreceivedtheinterventionwereverysimilarindemographicsascomparedtothecontrolgroup,withthepossibleexceptionofgender,whereaslightlyhigherpercentofmalesreceivedtheintervention.

Table2.4

InterventionandControlStudentDemographics(andProportions)

Gender Ethnicity Lunch Special EducationGroup Male Female White Minor-

ityReg

PayFree/Red

NotIn

SpecialEd.

Interv

136

96

205

27

40

192

161

71

(.59) (.41) (.88) (.12) (.17) (.83) (.69) (.31)

Contl 140 113 219 34 45 208 179 74 (.55) (.45) (.87) (.13) (.18) (.82) (.71) (.29)

Total

276

109

424

61

85

400

340

145

(.57) (.43) (.88) (.12) (.38) (.62) (.73) (.16)

ImpactsonStudents

Table2.5belowshowstheresultsonstudentreadingachievement,writingachievement,readingstrategyuseselfefficacyandreadingmotivationforinterventionandcontrolstudentsafteroneyearofintervention. Theunadjustedmeansandstandarddeviationsforeachmeasureisdisplayed,andthemeansadjustedfortheHLMresultsaredisplayed. Theestimatedimpactoftheintervention,theeffectsize,andthesignificancelevelareshown.

Cantrell,Carter,&Rintamaa(2012)

28SRCohortIIEvaluationReport:Kentucky

Table2.5ImpactoftheTargetInterventiononStudentReadingAchievement,WritingAchievement,StrategyUseSelfEfficacy,andMotivation

Unadjusted HLM-adjusted

Means Means Control Tx Control Tx Estimated

ImpactEffectSize

p

ReadingAchievementSpringNCE 36.7 35.6 37.2 36.4 -0.79 -0.059 .439 (13.49) (13.48)

No.ofstudents 253 232

WritingAchievement

No.ofstudents 193 172

StrategyUseSelfEfficacySpring score 3.3 3.5 3.4 3.5 0.154 0.250 .012*

(0.62) (0.61)No.ofstudents 173 154

MotivationSpringscore 2.9 3.0 2.9 3.0 .128 0.217 .015*

(0.59) (0.62)

No.ofstudents 173 154

Note. Standarddeviationsarepresentedinparenthesis. Effectsizecalculatedastheimpactdividedbythecontrolgroupstandarddeviation.*Designatesstatisticalsignificanceatthe.05levelofsignificance.

Thereisnosignificanteffectoftheinterventionatthe.05levelonthereadingand

writingachievementafterthefirstyearoftheprogram. However,asignificanteffectoftheinterventionisshownforstrategyuseselfefficacyandreadingmotivation.

Springscore 826.3 827.2 826.9 827.6 0.62 0.066 .481 (9.41) (11.78)

Cantrell,Carter,&Rintamaa(2012)

29SRCohortIIEvaluationReport:Kentucky

DiscussionandConclusions

TheimpactresultsfromoneyearofstudydonotrevealsignificanteffectsoftheKCLMinterventiononstudents’readingorwritingachievement,butthereweresignificantimpactsonstudents’self-efficacyforstrategyuseandstudents’readingmotivation.Developmentofdemonstrableimprovementsinreadingperformancemayrequiregreaterlengthsoftimetogaincomfortwithflexiblestrategyuseandtoreapbenefitsofincreasedmotivation.Whileitseemsthestudentsinthisstudyreportedincreasedconfidencewithusingreadingstrategies,theymaynothaveinternalizedandpracticedstrategyusetoasufficientenoughextenttoachievepurposefulflexibleuseunderawiderangeofconditions.Nevertheless,theimpactsoftheinterventiononstudents’strategyuseandmotivationarenoteworthygiventheemphasisplacedonthesedimensionsoflearninginrecommendationsforimprovingadolescents’literacyachievement(Biancarosa&Snow,2006;Kamil,Borman,Dole,Kral,Salinger,&Torgesen,2008).

Althoughtheplannedprofessionaldevelopmentmodelwasimplementedathighlevels,implementationoftheclassroommodelwaslower.ElementsoftheKCLMmodelwereevidentineachclassroomobservation,buttheteachersinthisprojectimplementedsomecomponentsoftheinterventiontoagreaterextentthanotherimportantcomponents.Also,themajorityofteachersimplementedtheinterventionatdevelopinglevelsofquality.Higherlevelsofimplementationmayhaveresultedinhigherlevelsofachievementforstudents.Itisprobablethatteacherswouldhavebeenabletoachievehigherlevelsofimplementationinfutureyearswithongoingsupporthadtheprojectextendedforthefullplannedprojectduration.

Cantrell,Carter,&Rintamaa(2012)

31SRCohortIIEvaluationReport:Kentucky

References

Allington,R.(2007).Effectiveteachers,effectiveinstruction.InBeers,K.,Probst,R.E.,&Rief,L.(Eds.),Adolescentliteracy:Turningpracticeintopromise(pp.273-288).Portsmouth,NH:Heinemann.

Almasi,J.(2003).Teachingstrategicprocessesinreading.NewYork:TheGuilfordPress.

Anderson,V.,&Hidi,S.(1988/1989).Teachingstudentstosummarize.EducationalLeadership,46,26-28.Retrievedfromhttp://www.ascd.org/publications/educational-leadership.aspx

Armbruster,B.B.,Anderson,T.H.,&Ostertag,J.(1987).Improvingcontent-areareadingusing

instructionalgraphics.ReadingResearchQuarterly,26(4),393-416.Retrievedfromhttp://www.reading.org/general/publications/journals/rrq.aspx

Bandura,A.(1993).Perceivedself-efficacyincognitivedevelopmentandfunctioning.

EducationalPsychologist,28(2),117-148.doi:10.1207/s15326985ep2802_3

Bandura,A.(1997).Theexerciseofcontrol.NewYork,NY:Longman.

Biancarosa,G.,&Snow,C.E. (2006). Readingnext–Avisionforactionandresearchinmiddleandhighschoolliteracy: AreportfromtheCarnegieCorporationofNewYork.Washington,DC: AllianceforExcellentEducation.

Blank,W.(1997).Authenticinstruction.InW.E.Blank&S.Harwell(Eds.),Promisingpractices

forconnectinghighschooltotherealworld(pp.15-21).Tampa,FL:UniversityofSouthFlorida.(ERICDocumentReproductionServiceNo.ED407586)

Bloome,D.(1986).Readingasasocialprocessinamiddleschoolclassroom.InD.Bloome(Ed.),

Literacyandschooling(pp.123-149).Ablex:Norwood,NJ.Bransford,J.,Brown,A.,&Cocking,R.(1999).Howpeoplelearn:Brain,mind,experience,and

school.Washington,DC:NationalAcademyPress.Brewster,C.andFager,J.(2000).IncreasingStudentEngagementandMotivation:FromTime-

on-TasktoHomework.NorthwestRegionalEducationalLaboratory.RetrievedJuly9,2009,fromtheWorldWideWeb:http://www.nwrel.org/request/oct00/textonly.html.

Brookbank,D.,Grover,S.,Kullberg,K.,&Strawser,C.(1999).Improvingstudentachievement

throughorganizationofstudentlearning.Chicago:Master'sActionResearchProject,SaintXavierUniversityandIRI/Skylight.

Caine,R.N.,&Caine,G.(1994).Makingconnections:Teachingandthehumanbrain.Menlo

Cantrell,Carter,&Rintamaa(2012)

30SRCohortIIEvaluationReport:Kentucky

Park,CA:Addison-WesleyCampbell,D.T.&Stanley,J.C.(1963).Experimentalandquasi-experimentaldesignsfor

research.Chicago:Rand-McNally.Carney,P.(2007).KentuckyReadingFirstEvaluation.UniversityofKentucky.Cartwright,K.(2008).Literacyprocesses:Cognitiveflexibilityinlearningandteaching.New

York:GuilfordPress.Chen,Z.(1999).Children'sanalogicalproblemsolving:Theeffectsofsuperficial,structural,and

proceduralsimilarities.JournalofExperimentalChildPsychology,62(3),410-431.doi:10.1006/jecp.1996.0037

Cook,T.D.,&Campbell,D.T.(1979).Quasi-experimentation:Designandanalysisfor

fieldsettings.Boston:HoughtonMifflin.Creswell,J.W.&PlanoClark,V.L.(2007).Designingandconductingmixedmethodsresearch.

ThousandOaks,CA:Sage.Creswell,J.W.(1998).Qualitativeinquiryandresearchdesign:Choosingamongfive

traditions.ThousandOaks,CA:Sage.Deshler,D.,Palincsar,A.,Biancarosa,G.,&Nair,M.(2007).Informedchoicesforstruggling

adolescentreaders.Newark,Delaware:InternationalReadingAssociation.Dev,P.C.(1997).Intrinsicmotivationandacademicachievement.Whatdoestheirrelationship

implyfortheclassroomteacher?RemedialandSpecialEducation,19(1),12-19.doi:10.1177/074193259701800104

Durkin,D.(1978-79).Whatclassroomobservationsrevealaboutreadingcomprehension

instruction.ReadingResearchQuarterly,15,481-533.doi:10.1598/RRQ.14.4.2Dyson,A.H.(1992).WhistleforWillie,lostpuppies,andcartoondogs:Thesociocultural

dimensionsofyoungchildren'scomposing.JournalofReadingBehavior,24(4),433-462.Eccles,J.S.,&Wigfield,A.(1995).Inthemindoftheactor:Thestructureofadolescents’

achievementtaskvaluesandexpectancy-relatedbeliefs. PersonalityandSocialPsychologyBulletin,21(3),215-225.doi:10.1177/0146167295213003

Fink,R.(2006).WhyJaneandJohncouldn’tread—andhowtheylearned:Anewlookatstriving

readers.Newark,DE:InternationalReadingAssociation.Fisher,D.,Frey,N.,&Rothenberg,C.(2008).Content-areaconversations:Howtoplan

Cantrell,Carter,&Rintamaa(2012)

32SRCohortIIEvaluationReport:Kentucky

discussion-basedlessonsfordiverselanguagelearners.Alexandria,VA: AssociationforSupervisionandCurriculumDevelopment

Flavell,J.H.(1979).Metacognitionandcognitivemonitoring:Anewareaofcognitive-

developmentalinquiry.AmericanPsychologist,34,906-911.doi:10.1037/0003-066X.34.10.906

Fugate,M.&Waterman,K.(2004).Groupreadingassessmentanddiagnosticevaluation

reliabilityregardingeducationaldecisionmaking.JournalofEducationalPsychology,96(2),187-201.

Gambrell,L.B.,Palmer,B.M.,Codling,R.M.,&Mazzoni,S.A.(1996).Assessingmotivationto

read.TheReadingTeacher,49,518-533.doi:10.1598/RT.49.7.2Graham,S.,&Perin,D.(2007).Writingnext:Effectivestrategiestoimprovewritingof

adolescentsinmiddleandhighschools.AreporttoCarnegieCorporationofNewYork.Washington,DC:AllianceforExcellentEducation.

Guthrie,J.T.,&Wigfield,A.(2000).Engagementandmotivationinreading.InM.L.Kamil,P.B.

Mosenthal,P.D.Pearson,&R.Barr(Eds.),Handbookofreadingresearch:VolumeIII(pp.403-422).NewYork:Erlbaum.

Hopper,R.(2005).Whatareteenagersreading?Adolescentfictionreadinghabitsandreading

choices.Literacy39(3),113-120.doi:10.1111/j.1467-9345.2005.00409.xcKamil,M,L.,Borman,G.D.,Dole,J.,Kral,C.C.,Salinger,T.S.,&Torgesen,J.(2008). Improving

adolescentliteracy:Effectiveclassroomandinterventionpractices. Washington,D.C.:U.S.DepartmentofEducation,InstituteofEducationSciences,NationalCenterforEducationEvaluationandRegionalAssistance.

Kane,S.(2007).Literacyandlearninginthecontentareas.Scotsdale,AZ:HolcombHathaway.Kushman,J.W.,Sieber,C.,&Heariold-Kinney,P.(2000).Thisisn'ttheplaceforme:School

dropout.InD.Capuzzi&D.R.Gross(Eds.),Youthatrisk:Apreventionresourceforcounselors,teachers,andparents(3rded.,pp.471-507).Alexandria,VA:AmericanCounselingAssociation.

Langer,J.A.(2001).Guidelinesforteachingmiddleandhighschoolstudentstoreadandwrite

well.Albany,NY:NationalResearchCenteronEnglishLearningandAchievement,StateUniversityofNewYorkatAlbany.

Lapp,D.&Fisher,D.(2009).Introduction.InLapp,D.&Fisher,D.(Eds.),Essentialreadingsin

comprehension.Newark,DE:InternationalReadingAssociation.

Cantrell,Carter,&Rintamaa(2012)

33SRCohortIIEvaluationReport:Kentucky

Lee,V.E.,&Smith,J.B.(1996).Collectiveresponsibilityforlearninganditseffectsongainsinachievementforearlysecondarystudents.AmericanJournalofEducation,104(2),103-147.doi:10.1086/444122

Lenz,B.K.,&Hughes,C.A.(1990).Awordidentificationstrategyforadolescentswithlearning

disabilities.JournalofLearningDisabilities,23(3),149-158.doi:10.1177/002221949002300304

Marzano,R.(1998).ATheory-BasedMeta-AnalysisofResearchonInstruction.Mid-Continent

RegionalEducationalLab.,Aurora,CO.Marzano,R.(2004).Buildingbackgroundknowledgeforacademicachievement:Researchon

whatworksinschools.Alexandria,VA:AssociationforSupervisionandCurriculumDevelopment.

Marzano,R.J.,Pickering,D.J.,&Pollock,J.E.(2001).Classroominstructionthatworks:

Research-basedstrategiesforincreasingstudentachievement.Alexandria,VA:AssociationforSupervisionandCurriculumDevelopment.

Mastropieri,M.A.,Scruggs,T.E.,Hamilton,S.L.,Wolfe,S.,Whedon,C.,&Canevaro,A.(1996).

Promotingthinkingskillsofstudentswithlearningdisabilities:Effectsonrecallandcomprehensionofexpositoryprose.Exceptionality,6(1),1-11.doi:10.1207/s15327035ex0601_1

Matsumura,L.,Garnier,H.,Pascal,J.,andValdes,R.(2002).Measuringinstructionalqualityin

accountabilitysystems:Classroomassignmentsandstudentachievement.EducationalAssessment,1532-6977,8(3),207–229.doi:10.1207/S15326977EA0803_01

Midgley,C.,Kaplan,A.,Middleton,M.,Maehr,M.L.,Urdan,T.,Anderman,L.H.,Anderson,E.,

&Roeser,R.(1998). Thedevelopmentandvalidationofscalesassessingstudents‘achievementgoalorientations.ContemporaryEducationalPsychology,23,113-131.doi:10.1006/ceps.1998.0965

Moats,L.C.(1998).Teachingdecoding.AmericanEducator,Spring/Summer,42-49.Retrieved

fromhttp://www.aft.org/newspubs/periodicals/ae/Mokhtari,K.&Reichard,C.A.(2002).Assessingstudents‘metacognitiveawarenessof

readingstrategies.JournalofEducationalPsychology,94(2),249-259.doi:10.1037/0022-0663.94.2.249

Moore,D.W.,&Readence,J.E.(1984).Aquantitativeandqualitativereviewofgraphic

organizerresearch.JournalofEducationalResearch,78(1),11-17.Retrievedfromhttp://www.tandf.co.uk/journals/titles/00220671.asp

Cantrell,Carter,&Rintamaa(2012)

34SRCohortIIEvaluationReport:Kentucky

NationalCouncilofTeachersofMathematics.(2000).Principlesandstandardsforschoolmathematics.Reston,VA.

Patrick,H.,Ryan,A.M.,&Kaplan,A.(2007).Earlyadolescents‘perceptionsoftheclassroom

socialenvironment,motivationalbeliefs,andengagement.JournalofEducationalPsychology,99,83-98.doi:10.1037/0022-0663.99.1.83

Pearson,P.D.,Roehler,L.R.,Dole,J.A.,&Duffy,G.G.(1992).Developingexpertiseinreading

comprehension.InJ.Samuels.,&A.Farstrup,(Eds.),Whatresearchhastosayaboutreadinginstruction.Newark,Delaware:InternationalReadingAssociation.

Pitcher,S.M.,Albright,L.K.,DeLaney,C.J.,Walker,N.T.,Seunarinesingh,K.,Mogge,S.,

Headley,K.N.,Ridgeway,V.G.,Peck,S.,Hunt,R.,&Dunston,P.J.(2007).Assessingadolescents‘motivationtoread.JournalofAdolescentandAdultLiteracy,50,378-396.doi:10.1598/JAAL.50.5.5

Pressley,M.,Brown,R.,El-Dinary,P.B.,&Afflerbach,P.(1995).Thecomprehensioninstruction

thatstudentsneed:Instructionfosteringconstructivelyresponsivereading.LearningDisabilitiesResearchandPractice,10,215–224.Retrievedfromhttp://mc.manuscriptcentral.com/ldrp

Raudenbush,S.,Spybrook,J.,Liu,X.,&Congdon,R.(2004)Optimaldesignforlongitudinaland

multilevelresearch:DocumentationfortheOptimalDesignsoftware.AnnArbor:UniversityofMichigan.

Redfield,D.,&Rousseau,A.(1981).Ameta-analysisofexperimentalresearchonteacher

questioningbehavior.ReviewofEducationResearch,51,237-246.doi:10.3102/00346543051002237

Ross,J.A.(1988).Controllingvariables:Ameta-analysisoftrainingstudies.Reviewof

EducationalResearch,58(4),405-437.doi:10.2307/1170280Rozzelle,J.,Searce,C.(2009).PowerToolsforAdolescentLiteracy:StrategiesforLearning.

Retrievedfromhttp://go.solution-tree.com/literacy/

Seidman,I.E.(1998).Interviewingasqualitativeresearch.NewYork:TeachersCollegePress.

Silver,H.,Strong,R.,Perini,M.(2007).TheHiddenSkillsofAcademicLiteracy.Alexandria:ThoughtfulEducationPress.

Smith,M.W.,&Wilhelm,J.D.(2006).Goingwiththeflow.Portsmouth,NH:Heinemann.Snider,VE.(1989).Readingcomprehensionperformanceofadolescentswithlearning

disabilities.LearningDisabilityQuarterly,12,87-96.doi:10.2307/1510724

Cantrell,Carter,&Rintamaa(2012)

35SRCohortIIEvaluationReport:Kentucky

StanfordAchievementTestSeries,TenthEdition(SAT-10).PearsonforEducation.

http://pearsonassess.com/HAIWEB/Cultures/en-us/Productdetail.htm?Pid=SAT10C.RetrievedJuly28,2009.

Stone,C.L.(1983).Ameta-analysisofadvancedorganizerstudies.JournalofExperimental

Education,51(7),194-199.Retrievedfromhttp://www.tandf.co.uk/journals/titles/00220973.asp

Strong,R.,Silver,H.,andPerini,M.(2001).Teachingwhatmattersmost:Standardsand

strategiesforraisingstudentachievement.Alexandria:AssociationforSupervisionandCurriculumDevelopment.

Strong,R.,Silver,H.,Perini,M.,Tuculescu,G.(2002).ReadingforAcademicSuccess.Thousand

Oaks:Corwin.TestofWrittenLanguage-Intermediate,4thEdition(TOWL-4).PearsonPsychCorp.Retrieved

fromtheWorldWideWebJuly28,2009:http://psychcorp.pearsonassessments.com/HAIWEB/Cultures/en-us/Productdetail.htm?Pid=PAa19045.

NorthCentralRegionalEducationalLaboratory.(n.d.)Thethinkingcurriculum.OakBrook,IL:

Author.[On-line].AvailableInternet:http://www.ncre.org/sdrs/areas/rpl_esys/thinking.htm

Tomlinson,C.(2001).Howtodifferentiateinstructioninmixed-abilityclassrooms.Alexandria:

ASCD.White,T.G.,Sowell,J.,&Yanagihara,A.(1989).Teachingelementarystudentstouseword-part

clues.TheReadingTeacher,42,302-309.Retrievedfromhttp://www.reading.org/general/publications/journals/rt.aspx

Wigfield,A.,&Guthrie,J.T.(1997).Relationsofchildren’smotivationforreadingtotheamount

andbreadthoftheirreading.JournalofEducationalPsychology,89(3),420-432.doi:10.1037/0022-0663.89.3.420

Wiggins,G.,&McTighe,J.(1998).Understandingbydesign.Alexandria,VA:Associationfor

SupervisionandCurriculumDevelopment.Williams,K.(2001).Groupreadingassessmentanddiagnosticevaluationtechnical

manual.CirclePines,MN:AmericanGuidanceServices.Woods,E.G.(1995).Reducingthedropoutrate.InSchoolImprovementResearchSeries(SIRS):

Researchyoucanuse(Close-upNo.17).Portland,OR:NorthwestRegionalEducationalLaboratory.

Cantrell,Carter,&Rintamaa(2012)

36SRCohortIIEvaluationReport:Kentucky

AppendixA

SummaryofAnalyticApproach

HierarchicalLinearModels(HLMs)wereusedtoestimatetheimpactoftheinterventiononstudentachievementinreadingandwriting,andmotivation,andself-efficacyoutcomes.TheGRADENormalCurveEquivalents(NCEs)wereusedtoestimatetheimpactoftheinterventiononreadingachievement,andthestandardwritingscoreswereusedtomeasuretheimpactonwriting. Theaveragesurveyscoreswereusedtoestimatetheimpactonself-efficacyinreadingstrategyuseandimpactonmotivation.

Atwo-levelHLMmodel(studentsassignedtointerventionorcontrolgroupwithinschools)wasusedtodeterminetheimpactofthetargetedintervention. Atthestudentlevel,thespringoutcomevariable(readingachievement,writingachievement,self-efficacyinstrategyuse,ormotivation)wasmodeledasafunctionoffalloutcomevariables,intervention/controlstatus,andfourdemographicvariables:gender,ethnicity,free/reducedlunchstatus,andspecialeducation.Level-1Model:StudentOutcomes(achievement,readingstrategies,ormotivation)

M

Yij=� 0j+

where

� 1j(Y*ij)+� 2j(Tij)+

� mj mij+ ijm=3

Yij isthespringstudentoutcome(post-test)scoreforstudentiatschoolj;� 0jisthemeanstudentoutcome(post-test)scoreforcontrolstudentsat schoolj;Y*ijisthefallstudentoutcome(pre-test)scoreforstudenticenteredat schoolj;� 1jistheaveragestudentoutcome(pre-test)slopeforstudentsatschoolj;Tij=1ifstudentiisassignedtoLSCinterventionatschoolj,and0ifcontrol;� 2jisthemeandifferenceofstudentoutcomepre-postgainbetweeninterventionand

controlstudentsatschoolj; mij areadditionalcovariatesrepresentingdemographiccharacteristicsofstudentiat

schoolj(gender,ethnicity,free/reducedlunch,andspecialeducationstatus);� mjarecoefficientscorrespondingtostudentdemographiccovariates(gender,

ethnicity,free/reducedlunch,specialeducationstatus),and ij istherandomeffectrepresentingthedifferencebetweenstudentij’sscoreandthe

predictedmeanscoreforschoolj. Theseresidualeffectsareassumednormallydistributedwithmean0andvariancea 2.

Level-2Model: StudentAchievement–SchoolLevel

Thisanalysiswasperformedondatafrom9thgradestudentscollectedforoneyear. Thecovariatesinthismodelpertaintotheconcurrentyearthestudentwasintheinterventionor

Cantrell,Carter,&Rintamaa(2012)

37SRCohortIIEvaluationReport:Kentucky

controlgroupwiththeexceptionoftheReadingKentuckyCoreContentTest(KCCT)score,forwhichthescoreforthebaseyear,spring,2010,wasused. InadditiontothebaseyearReadingKCCTscore,otherschoollevelcovariatesincludedenrollment,percentofwhitestudentsintheschool,percentofAfricanAmericanstudents,percentofstudentsqualifyingforfreeorreducedlunchfeesandpercentofstudentswithdisabilities..

Q

� 0j=

� 1j=� 2j=� mj

=

y 00+

y 10y 20y m0

y oqWqj

+q

µ 0j

where

y 00 isthemeanstudentoutcome(post-test)scoreof9thgradecontrolstudents

inKentuckyStrivingReadersmiddleschoolsWqj areschoollevelcovariatesincludingbaseyearReadingKCCT(spring,2010),and

averageschoolpercentfree/reducedlunch,percentwhitestudents,percentblackstudents,andpercentdisability;

y oqarecoefficientscorrespondingtoschool-levelcovariates;µ 0j istheuniqueeffectofschooljonmeanstudentoutcome,holdingWqjconstant(or

conditioningonWqj)-thiseffectisassumednormallydistributedwithmean0andvarianceT 2;

y 10isthefallstudentoutcome(pre-test)slope;y 20istheoveralltargetinterventiontreatmenteffectonspringstudentoutcome

(post-test)scores;y m0 isthefixedmthstudentcovariateeffect(gender,ethnicity,free/reducedlunch,special

educationstatus) onthespringoutcomevariable.

SelectionofCovariates. TherandomassignmentprocedureincludedallstudentdemographicvariablesintheHLMmodel,sowereincludedregardlessofsignificance.Interactioneffectswerenotconsidered.

Cantrell,Carter,&Rintamaa(2012)

38SRCohortIIEvaluationReport:Kentucky

AppendixB

StudyMeasures

StudentSurvey

Itemsforthestudentsurveywereadaptedfromthefollowingpre-existinginventories:Eccles,J.S.,&Wigfield,A.(1995).Inthemindoftheactor:Thestructureofadolescents’

achievementtaskvaluesandexpectancy-relatedbeliefs.PersonalityandSocialPsychologyBulletin,21(3),215-225.doi:10.1177/0146167295213003

Hopper,R.(2005).Whatareteenagersreading?Adolescentfictionreadinghabitsandreadingchoices.Literacy39(3),113-120.doi:10.1111/j.1467-9345.2005.00409.xc

Mokhtari,K.&Reichard,C.A.(2002).Assessingstudents‘metacognitiveawarenessofreadingstrategies.JournalofEducationalPsychology,94(2),249-259.doi:10.1037/0022-0663.94.2.249

Wigfield,A.,&Guthrie,J.T.(1997).Relationsofchildren’smotivationforreadingtotheamountandbreadthoftheirreading.JournalofEducationalPsychology,89(3),420-432.doi:10.1037/0022-0663.89.3.420

Cantrell,Carter,&Rintamaa(2012)

39SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

41SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

40SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

42SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

43SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

44SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

45SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

46SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

47SRCohortIIEvaluationReport:Kentucky

49SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

TeacherObservationProtocol

TeacherFirstName:

TeacherLastName:_

School:

Date(includingdayoftheweek):_

Time:_

Observer:_

Notes:

48SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

Cantrell,Carter,&Rintamaa(2012)

51SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

50SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

51SRCohortIIEvaluationReport:Kentucky

Cantrell,Carter,&Rintamaa(2012)

52SRCohortIIEvaluationReport:Kentucky

Recommended