0antrell,Carter&Rintamaa,2012
C
2012
SusanC. Cantrell,Janis
C. Carter,and
MargaretRintamaa
CollaborativeCenterfor
LiteracyDevelopment
UniversityofKentucky
March,2012
StrivingReadersCohortIIEvaluation Report:
Kentucky
1SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter&Rintamaa,2012
Acknowledgments
Theauthorsgratefullyacknowledgethecontributionsofthemanyindividualswhofacilitatedthisevaluation. ThisevaluationcouldnothavebeensuccessfulwithoutthesupportofCindyParker,RobinHebert,MaryRuddandMonicaOsbornefromtheKentuckyDepartmentofEducation(KDE)andadministratorsandstaffinthenineStrivingReadersschools. WewanttothanktheStrivingReadersinterventionistswhoworkdiligentlywithstudentsandteacherseachdayandtowhomweturnregularlyforinformationtofurtherthiswork. Furthermore,wegratefullyacknowledgethemanystudentswhoparticipatedinthisstudy.
Additionalmembersoftheevaluationteamwhodeservecreditfortheircontributions
tothisreportincludethefollowingindividualstowhomweareimmenselygrateful:
DataAnalystDeepshikhaSigdel
ResearchAssistantsLauraDudneyAmandaGoodwinJessicaGreweMelissaMurphyJenniePahl
StaffSupportElisha ComerJudyJohnson
Finally,weextendourgratitudetothestaffoftheCollaborativeCenterforLiteracyDevelopment(CCLD)fortheirongoingsupportofthiswork,AbtAssociatesInc.forthetechnicalassistancetheyprovidedduringthestudy,andtheU.S.DepartmentofEducationforfundingthisproject.
2SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter&Rintamaa,2012
TableofContents
ExecutiveSummaryofFindings:ImplementationandImpact……………………………………. 4KCLMTargetedIntervention…………………….……………………………………………………………….. 5
Implementation….……………………………………………………………………………………………….. 5Impacts.…………………………………………………………………………………………………………….... 5
IntroductionandStudyBackground…………………………………………………………………………….. 5DescriptionoftheInterventionModel….…………………………………………………………………… 5FrameworkoftheSupplementalLiteracyInterventionClass..………………………………….. 6ComponentsofKCLM.……………………………………………………………………………………………… 7TargetedStudents…………………………………………………………………………………………………….. 8SelectionProcessforInterventionists……………………………………………………………………….. 9DesiredCharacteristicsoftheInterventionClassroom……………………………………………… 9LogicModelforKentuckyCognitiveLiteracyModelIntervention………………………………. 10PlannedTrainingModel…………………………………………………………………………………………….. 11PlannedClassroomInstructionalModel…………………………………………………………………….. 12PlannedExperiencesforControlStudents…………………………………………………………………. 12UnitPlanningTemplate…………………………………………………………………………………………….. 13KeyEvaluationDesignFeatures…………………………………………………………………………………. 14
EvaluationoftheImplementation…………………………………………….….……………………………… 14SummaryoftheDesignoftheImplementationStudy.…..……………………………………….…. 14ImplementationDataCollectionandAnalysis..…………………………………………………………. 15
SummerTraining…………………………………………………………………………………………………. 15CoachMentoring…………………………………………………..……..……………………………………… 15ClassroomInstruction…………………....………………………..……………………………………….... 15
KCLMImplementationResults……………………………………..…………………………………………….. 16CharacteristicsofInterventionists……………………………………………………………………..……… 16ImplementationofProfessionalDevelopmentModel…………………….………………………… 16
Interventioniststraining….………………………………………………………………………………….. 16Professionaldevelopmentinputs………………………………………………………………………… 17Professionaldevelopmentforadministrators……………………………………………………… 18
ImplementationofClassroomModel……………………………………………………………………….. 18Classsize,intensity,andduration……………………………………………………………………….. 18Classroomimplementationresults……………………………………………………………………… 18Experiencesforcontrolstudentsduringinterventionperiod………………………………. 20Additionalreadingprograms………………………………………………………………………………. 20Implicationsforimpactanalysis…………………………………………………………………………… 20
EvaluationofImpact……………………………………….…………………………………………………………… 21StudyDesign……………………………………………………………………………………….…………………….. 21
3SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter&Rintamaa,2012
Samplingselectionprocess…...……………………………………………………….……………………. 21Samplesize……………….……………………………………………………………………………………….. 21ConsortDiagramReadingAchievement…………………………………………………..………… 22ConsortDiagramWritingAchievement……………………………………………………………… 23ConsortDiagramStudentEfficacyandMotivation….…………………………………………… 24MetacognitiveAwarenessofReadingStrategiesInventory(MARSI).…………….…….. 25AdolescentMotivationSurvey………………...…………………………………………………….……. 25
SummaryofAnalyticApproach………………...………………………………………………………………. 26Level-1HLM:StudentLevel………………..……………………………………………………………….. 26Level-2HLM:SchoolLevel………………………………………………………………………..………… 26
DescriptionoftheFirstYearSample………….……………………………………………………………… 26ImpactsonStudents………………..………………………………………………………………………………… 27
DiscussionandConclusions……………………………………………………………..…………………………… 29References………………………….………………………………………………………………………………………. 30
APPENDICESA:SummaryofAnalyticProcedures………………………………………………………………………….. 36B:StudyMeasures…………………………………………………………………………………………………….. 38
StudentSurvey…………………………………………………………………………………………………….. 39TeacherObservationProtocol……………………………………………………………………………… 48
EXHIBITS
1:ConsortDiagramReadingAchievement…………………………………………………..……………. 222:ConsortDiagramWritingAchievement…………………………………………………………………. 233:ConsortDiagramStudentEfficacyandMotivation….…………………………………………….. 24
FIGURES
1:ComponentsofKCLM……………………………………………………………………………………………. 72:KCLMInterventionLogicModel…………………………………………………………………………………… 103:UnitPlanningTemplate…………………………………………………………………………………………. 13
TABLES
1.1FrameworkoftheSupplementalLiteracyInterventionClass…..…………………………… 61.2PlannedPDActivities…………………………………………………………………....……………………. 112.1 Contentanddeliveryofprofessionaldevelopmenttraining(KCLM)……….…………… 172.2 MostWidelyImplementedModelComponents……………………………………….……….. 192.3 MeanScoresbyTeacherforQualityofKCLMcomponent....................................... 202.4 InterventionandControlStudentDemographics(andProportions)....................... 272.5 ImpactoftheTargetInterventiononStudentReadingAchievement,WritingAchievement,
StrategyUseSelfEfficacy,andMotivation…….……………………………… 28
4SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter&Rintamaa,2012
StrivingReadersCohortII: Kentucky
ExecutiveSummaryofFindings:ImplementationandImpact
ThisStrivingReadersevaluationexaminedtheimpactofatargetedinterventionforstrugglingadolescentreadersinparticipatingschools. Thisstudywasconductedinninehighschoolsinnineschooldistrictsservinglargepercentagesofat-riskstudentsinKentucky. ThetargetedinterventionforstrugglingreaderswastheKentuckyCognitiveLiteracyModel(KCLM)developedbytheKentuckyDepartmentofEducation.
TheeffectivenessoftheKCLMinterventionwasdeterminedthrougharandomized
controlfieldtrialutilizingatreatmentandcontrolgroupdesign.TheKCLMwasasupplementtotheregularcurriculumwhereinstudentsinthetargetedinterventionparticipatedinareadingclassinplaceofanelectiveaspartoftheirregularschoolday. Thecontrol-groupconditionwas“businessasusual,”whereinstudentsinthecontrolgrouptakearegularelectivesuchasband,theaterarts,civics,orphysicaleducation. ThisstudyexaminedtheimplementationofKCLManditsimpactonstrugglingninth-gradestudents’readingandwritingachievement,self-efficacywithreadingstrategies,andmotivationforreading.
InthisStrivingReadersproject,eachschoolemployedaninterventionteacherwhowas
responsibleforteachingthetargetedinterventiontostrugglingreaders.Theimpactresearchquestionsthatmotivatedthestudydesignandanalysisplanare:
• WhatistheimpactofKentucky’sCognitiveLiteracyModel(KCLM)onthereading
achievementoflow-achievingreaders?• WhatistheimpactofKCLMonthewritingachievementoflowachievingreaders?• WhatistheimpactofKCLMontheperceivedreadingstrategyuseoflowachieving
readers?• WhatistheimpactofKCLMonthemotivationandengagementoflowachieving
readers?Inaddition,thefollowingimplementationquestionswillbeanswered:
• What is the state-level implementation of theprofessional development training and
supportforinterventionteachersintheproject?• WhichcomponentsofKCLMwereimplementedmostfrequentlybyteachersin
classrooms?• WhatwasthequalityofKCLMimplementationinclassrooms?
5SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter&Rintamaa,2012
KCLMTargetedIntervention
ThisstudyexaminedboththeimplementationandimpactoftheKCLMoverthecourseofthefirstyearoftheStrivingReadersproject.
Implementation. Duringtheyearoftheproject,KCLMteachersparticipatedintrainingandon-sitesupport,andtrainingwasprovidedforschooladministrators. Duringtheyear,KCLMteacherswereprovided11daysprofessionaldevelopmenttrainingandupto66hourssupportfromvisitsbyKDEliteracystaff. Schooladministratorswereprovidedonedayoftrainingandatleasttwosupportmeetingsregardingtheinterventionfortheyear. Overall,participationintheprofessionaldevelopmenttheKCLMteachersandadministratorswashigh,with100%ofteachersandadministratorsparticipatingfullyinthetrainingeitherthroughtheformaltrainingdatesorthroughmakeuptraining. Classroomimplementationfidelitywasmeasuredthroughclassroomobservations. ObservationsindicatedthatteachersimplementedsomecomponentsoftheKCLMmodelmorereadilythanothercomponents.Theinterventionwasimplementedwithadequatequalityin3of9classrooms.
Impacts. Thisstudyusedhierarchicallinearmodeling(HLM)analysestomeasuretheimpactsoftheKCLMonninth-grade(highschool)students’readingachievement,writingachievement,perceivedstrategyuse,andmotivation.InthisStrivingReadersstudy,therewerenoimpactsonstudents’readingachievementasmeasuredbytheGroupReadingAssessmentandDiagnosticEvaluationandnoimpactsonwritingachievementasmeasuredbytheKentuckyStateWritingAssessment. Studentsurveyresultsindicatedsignificanteffectsoftheinterventiononparticipatingstudents’self-efficacyforstrategyuseandonstudents’readingmotivation.
IntroductionandStudyBackground
DescriptionoftheInterventionModel
ThetargetedinterventionfortheKentuckyprojectwastheKCLM,developedbytheKentuckyDepartmentofEducation(KDE). ThepurposeoftheKCLMwastoassiststudentswhoweresignificantlybehindgradelevelinreadingbyprovidingthemwiththesupportstobesuccessfulinlearningacrossthecurriculum. Table1.1illustratesthestrandsoftheintervention.
6SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter&Rintamaa,2012
Table1.1
FrameworkoftheSupplementalLiteracyInterventionClass
Motivationand
Engagement• Thematicinstructionwithproject/inquiry-basedlearningproduces
studentswhoaremorefullyengagedandmotivatedtolearn.• Literacyisasocialaccomplishment(Bloome,1986;Dyson,1992).
Strategic
Processing• Strategy—adeliberatecognitiveprocessofselecting,enacting,
monitoringandregulatingbehavior. Includescomprehensionstrategy
instructionandfoundationalreadingskills.• Skill—amentalactivitythatcanbeappliedtospecificlearningsituations.• Metacognition—keytostrategicprocessingbecauseitenablesstudents
tomonitorprogresstowardachievingtheirgoals(Flavell,1979).Instructional
Strategiesfor
ContentLearning
• RobertMarzano’scharacteristicsofeffectivevocabularyinstruction• Marzano,etal.strategiesforlearningsuchascues,questions,and
advanceorganizers;non-linguisticrepresentations;identifyingsimilaritiesanddifferences;summarizingandnotetaking
Communication
Skills• Includesreading,writinganddiscussionoutcomesthataddressavariety
ofapproachestoessentialquestionsandtexts.• Writinghelpsreadersclarifymeaningandprovidesopportunitiesfor
authenticengagementandcommunication.• Exchangingideas,especiallythroughextendeddiscussionofmeaning
andinterpretationoftext,isessentialtoalearningcommunity.
Itisimportanttonotethatwhilethesearecategorizedintospecificstrands,theyalsoareembeddedacrossstrandstointegrateatotalliteracyexperienceforstudents.ThefourcomponentsoftheKCLMframeworkweretiedtogetherbycontent-relatedthemessuchassuccess,theenvironment,andproblemsolving.
7SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
Figure1showsthemajorcoursecomponentsandspecificdimensionsofeachcomponent.
SupplementalInterventionModelComponentsMotivation&Engagement
StrategicProcesses InstructionalStrategies
Communication
Theme-basedInstructionConnectionsbetweenlearningactivitiesandrealworldissuesVariedinstructionalformatStudentpriorknowledge,interestandbackgroundusedindeterminingcontentTechnologyisusedtofacilitatelearningGoalsettingandprivatefeedbackFocusonproblem-solvingprocessesAutonomyforlearningandmeaningfulchoices
Explicitcomprehensionstrategyinstruction(modeling,explanation,practice,andreflection)Explicitinstructioninfoundationalreadingskills
Instructionaltoolsforcomprehensionandvocabularylearning.Vocabularyprocessingthroughvisual,auditory,physicaland/oremotionalexperiences;opportunitiestousetheirownwordsornon-linguisticrepresentationstodefinenewwords;teacherexplanationsandexamplesofnew,keytermsAfocusonsummarizingandidentifyingsimilaritiesanddifferencesHigherlevelquestioning
Pre-,during-,andpost-textbaseddiscussionstrategiesWritingtolearnactivitiesExplicitinstructioninwritingstrategiesExplicitinstructioninfoundational writingskills
Figure1:ComponentsofKCLM.
8SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter&Rintamaa,2012
SRCohortIIEvaluationReport:Kentucky
IntheKentuckyproject,ninth-gradestudentswhoscoredtwogradelevelsormorebelowgradelevelinreadingreceivedaminimumof225minutesperweekandamaximumof375minutesperweekofsupplementalreadinginstructioninatargetedinterventionclasstaughtbyaninterventionteacher. Studentswereplacedinthiscourseinadditiontotheirregularreading/languageartsclassesforanentireschoolyear.
Overthecourseoftheproject,theprofessionaldevelopmentmodelforthetargetedinterventionincludedsummerandfollow-uptrainingsandon-sitesupportfromKDEliteracystaff.Tolearnhowtoimplementthetargetedintervention,teachersparticipatedinasummerworkshop,whichwasledbyKDEliteracystaff. Duringtheschoolyear,thetrainersledtheteachersinfollow-upworkshops. Acrosstheproject,KCLMteachersreceived11daysofworkshoptraininginthetargetedinterventionintotal. Tosupporttheirongoinglearninganddevelopment,teachersalsoparticipatedinsitevisitsandregulardistancesupportbyKDEliteracystaff.
Theprofessionaldevelopmentmodelincludedtrainingandsupportforadministrators,aswell. SchooladministratorsattendedaonedaymeetinginthesummertolearnabouttheKCLMmodelandthewaysinwhichtheschoolsshouldsupporttheintervention. Additionally,KDEliteracystaffparticipatedinon-sitemeetingswithadministratorsuptosixtimesduringtheschoolyear.Topicsatthosemeetingsincludedgrantrequirements,evidence-basedcomponentsofthetargetedintervention,schedulingissues,observationsofinterventionists,literacyleadership,literacyplanning,andmeetingtheneedsofstrugglingadolescentlearners. AdministratorsalsoreceivedKCLMupdatenewslettersfromKDEfourtimesduringthecourseoftheyear.TargetedStudents
Thisprojectwasdesignedtoservelowachievingreadersintheninthgrade. Inthespringoftheireighth-gradeyear,studentsinmiddleschoolsthatfeedintothenineparticipatinghighschoolsweregiventhemiddleschoolformofthespringGroupReadingAssessmentandDiagnosticEvaluation(GRADE). StudentsthatscoredanNCEof40orlowerontheGRADEweredefinedaslowachievingreaders,andwereplacedintheeligibilitypool.
Studentsinalldayresourceclasseswerenoteligible;allotherstudentswereeligible. Theevaluationteamdirectedthefacultyatallfeedermiddlesschoolstoidentifystudentsinalldayresourceclasses,andinstructedtheschoolsnottogivethespringGRADEtothesestudents. Inaddition,evaluatorsdirectedthefacultyatthenineparticipatinghighschoolstoidentifystudentsthatwereplacedinalldayresourceclassesaftertheyenrolledinninth-gradethisfallandaskedthemtoprovideevaluatorswiththenamesofthesestudentsforremovalfromthestudy.Finally,middleschoolssenthomepassiveconsentformswithalleighth-gradestudentswhowouldbetestedforparticipationinthestudy. Thestudywasdescribed,andparentsweredirectedtocontacttheevaluatorsiftheychosetonotallowtheirstudenttoparticipateinthestudy. Noparentdeclinedpermissionfortheirchild’sparticipationinthestudy.
Twothousandtwohundredfourstudentswerelistedontheschoolregistersheetsasenrolledatthefeedermiddleschools. Evaluatorsreceivedanadditionaltwenty-fivestudent
Cantrell,Carter,&Rintamaa(2012)
8
9SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
GRADEsheetsthatwerenotontheenrollmentlists.Therewere637studentsthatmetthestudy’scriteriononthespringGRADEforeligibility.SelectionProcessforInterventionists
Interventionteacherswererecruitedandhiredbyindividualschools. Advertisementsfortheinterventionteacherpositionincludedthefollowingcriteria: experiencedclassroomteacher,respectedbyfacultyandadministration;familiaritywithand/orinterestininterdisciplinaryandproject-basedlearning;willingnesstolearnandapplynewskillsandknowledge;planning/reflectingskills;strongleadershipability;adaptabilityandproblemsolvingskills;presentationskills;collaborationskills;abilitytomediatebetweentheschoolandcommunityorganizations;andpersonalcommunicationskills.DesiredCharacteristicsoftheInterventionClassroom
Classesweretobenolargerthantwentyninth-gradestudents. Theinterventionclasswastomeetdailyforatleast45minutesfortheentireyear.
10SRCohortIIEvaluationReport:Kentucky
SRCohortIIEvaluationReport:Kentucky
Figure2:KCLMInterventionLogicModel Cantrell,Carter&Rintamaa,2012
10
SRCohortIIEvaluationReport:Kentucky
PlannedTrainingModel
Duringthesummerof2010,theinterventionteachersfromeachschoolweretoreceivefivedaysofinitialtrainingonthecorestrands. ThistrainingwastobedevelopedandprovidedbytheKentuckyDepartmentofEducationliteracystaff. Schooladministratorswererequiredtoattendonedayofsummertrainingtolearnaboutthegoalsoftheinterventionandtheexpectationsforsupportoftheinterventionistandstudents. Bytheendoftheinitialtraining,theinterventionistsweretocreateaninstructionalplanfordirectstrategyinstruction,andintegrationofstrategiesintostandards-basedunitsofstudy. Eachparticipantwastoleaveunderstandinghowtoteach,use,andassesswithintheKCLMframework,andwithaplantobeginimmediateimplementationasthe2010-2011schoolyearbegan.
Throughouttheyearofimplementation,theinterventionistsweretoreceiveon-going
trainingandsupportthroughface-to-facemeetingsandanonlineKentuckyVirtualSchools(KYVS)learningcommunity,whichwastoincludebookstudies,Webinars,adiscussionforum,andrepositoriesforsharingresources.KDEliteracystaffplannedsitevisitstotheschoolsinthefallandagaininthespringtoprovideon-sitecoachingandsupportforupto50additionalhours.
Administratorsweretoengageinprofessionaldevelopmentfocusingontheguidefrom
theNationalAssociationofSecondarySchoolPrincipals,CreatingaCultureofLiteracy:AGuideforMiddleandHighSchoolPrincipalsaswellasaKDEDVDresource,LiteracyLeadership:StoriesofSchoolwideSuccess.Thissharedleadershipnetworkwasintendedtohelpadministratorsandcoachesdevelopacloseworkingrelationshipcriticalinsupportinginstructionalimprovementefforts.
Table1.2
PlannedPDActivities
PDhrs. Activities Date Attendees*40 Trainingoncorestrandsofintervention;develop
instructionalplanandunitsofstudy8 Trainingongoalsforinterventionandon
providingsupporttointerventionists50+ OngoingPD: Onlinelearningcommunity,
includingbookstudies,webinars,discussionforum,sharingnetwork
July2010 I
July2010 A
2010 I
16 Onsitevisits/coaching 2010 I
*I=Interventionist;A=Administrator
Cantrell,Carter&Rintamaa,2012
11
SRCohortIIEvaluationReport:KentuckyPlannedClassroomInstructionalModel
TheKentuckyDepartmentofEducationcreatedaUnitPlanningTemplate(UPT)toassistinterventionteachersincreatingunitsofinstructionfortheinterventionclass. OnecomponentoftheUPTlistedthestrategiesandactivitiesthatcomprisetheKCLM.Teachersselectedstrategiestoteachineachunit.Figure3showsthestrategiesandactivitiesaslistedontheUPT.Teacherswerefreetoselectthestrategiestheytaughtbasedontheirassessmentofstudents’needs.Aspartoftheinterventionclass,studentsassignedtotheinterventionwilltaketheSAT-10online(reading)andTOWL-4(writing)diagnosticassessmentstwiceperyear.
Inaddition,KCLMisdesignedtoengagestudentsinreadingavarietyoflevel-appropriatetextsrelatedtoessentialquestion(s)foreachcontent-relatedunit,includingnonfiction,informationalandproceduraldocuments,andnarrativetexts.PlannedExperiencesforControlStudents
Expectationsforthecontrolstudentswere“businessasusual,”whereinstudentsinthecontrolgrouptookaregularelectivesuchasband,theaterarts,civics,orphysicaleducation.
Cantrell,Carter,&Rintamaa(2012)
12
SRCohortIIEvaluationReport:Kentucky
Figure3:UnitPlanningTemplate
Cantrell,Carter&Rintamaa,2012
13
SRCohortIIEvaluationReport:KentuckyKeyEvaluationDesignFeatures
Theevaluationisdesignedtomeasuretheimpactofthetargetedinterventiononstudentoutcomesteacherefficacy. Theimpactstudywasguidedbythefollowingresearchquestions:Theimpactresearchquestionsthatmotivatedthestudydesignandanalysisplanare:
• WhatistheimpactofKentucky’sCognitiveLiteracyModel(KCLM)onthereadingachievementoflowachievingreaders?
• WhatistheimpactofKCLMonthewritingachievementoflowachievingreaders?• WhatistheimpactofKCLMontheperceivedreadingstrategyuseoflowachieving
readers?• WhatistheimpactofKCLMonthemotivationandengagementoflowachievingreaders?
Studentoutcomemeasuresareasfollows:
• GroupReadingAssessmentandDiagnosticEvaluation(GRADE)• TheKentuckyStateWritingAssessment• MetacognitiveAwarenessofReadingStrategiesInventory(MARSI)• StudentMotivationSurvey
Inaddition,thefollowingimplementationquestionswereanswered:
• Whatisthestate-levelimplementationoftheprofessionaldevelopmenttrainingandsupportforinterventionteachersintheproject?
• WhichcomponentsofKCLMwereimplementedmostfrequentlybyteachersinclassrooms?
• WhatwasthequalityofKCLMimplementationinclassrooms?
EvaluationofImplementation
SummaryoftheDesignoftheImplementationStudy
Theresearchquestionsthatguidedtheimplementationstudyofthetargetedinterventionare:
• Whatisthestate-levelimplementationoftheprofessionaldevelopmenttrainingandsupportforinterventionteachersintheproject?
• WhichcomponentsofKCLMwereimplementedmostfrequentlybyteachersinclassrooms?
• WhatwasthequalityofKCLMimplementationinclassrooms?
Cantrell,Carter&Rintamaa,2012
14
15SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
ImplementationDataCollectionandAnalysis
Summertraining.Duringthesummerof2010,theselectedinterventionteachersfromeachschoolreceivedfivedaysofinitialtrainingonthecorestrands. ThistrainingwasdevelopedandprovidedbytheKentuckyDepartmentofEducationliteracyconsultants.Attendancerecordswerekeptateachtrainingsession,andindividualteacherattendancewascomputedattheendofthetrainingintermsofpercentageofdaysattended.Schooladministratorswillberequiredtoattendonedayofsummertrainingtolearnaboutthegoalsoftheinterventionandtheexpectationsforsupportoftheinterventionistandstudents. Participationwillbeassessedandscoredforadequacyandfidelitythroughattendancerecordsprovidedbythedevelopers.
Researchassistantsattendedeachtrainingsessionandtookdetailedfieldnotesinfiveminuteintervals. AcodelistwasdevelopedrelatedtoContentandDelivery.ContentcodesrelatedtothekeycomponentsofKCLM.Deliverycodesrelatedtovarioustrainingformatssuchaswholegroup,smallgroup,discussion,andlecture.Toestablishreliability,threeresearchassistantsindependentlycodedonedayoffieldnotes.Agreementwas90%forcontentand90%fordelivery.Afterthecodingtheresearchassistantsagreedthatsomeofthecodesshouldbefurtherbrokendownforamoreaccuratedescriptionofwhatwasgoingonduringthetimeinterval.Researchassistantsandinvestigatorsdiscussedaddingadditionalcodesforcontentanddelivery.Thefollowingcodeswereaddedtocontent:Collaboration,Planning-working,andPlanning-discussion.Thefollowingcodeswereaddedtodelivery:Wholegroup-lectureandWholegroup-discussion.Theresearchassistantscontinuedindependentlycodingthenextfourdaysoftheinterventionteachertraining.Researchassistantsmetagaintocheckagreement.Agreementachievedforcontentwas95%and100%agreementfordelivery.
Coachingandmentoring.Interventionistsreceivedon-goingtrainingandsupportthroughanonlinelearningcommunitythatincludedbookstudies,Webinars,adiscussionforum,andrepositoriesforsharingresources.KDEconsultantsmadesitevisitstotheschoolsinthefallandagaininthespringtoprovideon-sitecoachingandsupport. Participationwasassessedandscoredforadequacyandfidelitythroughattendancerecordsprovidedbythedevelopers.
Classroominstruction.Implementationfidelityforthetreatmentconditionwasestablishedthroughclassroomobservationsusingastandardizedobservationprotocolforinterventionclasses. Readinginterventionteacherswereobservedtwiceduringtheyear. Thestandardizedobservationprotocolforinterventionteachersincludedtwocomponents:achecklistofessentialKCLMcomponentsandaqualityrubricforassessingteachers’implementationquality.Inthefall2010,tworesearchassistantsmettocreatealistofmodelcomponentsbasedonthetrainingsfromKDE. ThislistwassenttoKDEforvalidation. KDEaddedafewcomponentstomakethelistcomplete. Toidentifythemodelfeaturesthatweremostcriticaltoprogramsuccess,evaluatorsaskedthedeveloperstorankeachsetoffeaturesforeachcomponentinorderofimportance. Toconfirmtheserankings,evaluatorssoughtfeedbackfromthreeexpertscholarswhoconfirmedandelaboratedontheshortenedlistforeachfeature.TheleadevaluatorandKCLMtrainerviewedtwovideosofKCLMinstructionandindependentlycompletedtheprotocol.Then,theydiscussedtheirscores,reachedconsensusonscoring,andmademinorrevisionsto
16SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
somewordingontheprotocol.Fortraininginusingtheprotocol,researchassistantsviewedonevideo,ratedtheinstructionusingtheprotocol,anddiscusseddisagreements. Toestablishinter-observeragreement,theresearchassistantsviewedthesecondvideoandindependentlycompletedaprotocol. AgreementwiththeleadevaluatoronthechecklistofkeyaspectsofKCLMcomponentsforthisprotocolwas74.4%.Thelargestareaofdisagreementwasrelatedtovocabularyinstructionunderthe“InstructionalStrategies”component. Theleadevaluatorprovidedadditionaltrainingaroundvocabularyinstructiontoclarifymisunderstandings.Agreementonthequalitydescriptorsforthesecondvideowas91.7%.
Itwasexpectedthatteacherswouldimplementsomeaspectofeachforthefourmodelcomponents(motivationandengagement,strategicprocesses,instructionalstrategies,andcommunication)duringeachclassperiodbutitwasnotexpectedthatteacherswouldimplementeveryaspectofeachcomponenteachclassperiod.Theprotocolyieldedinformationonwhichaspectsofeachcomponentwereimplementedduringobservationsandwhetherthequalityofimplementationwas(a)developing,(b)adequate,or(c)exemplary.Percentagesofobservationsthatincludedeachcomponentwerecomputed,andtheproportionofobservationsratedateachqualitylevelwasprovided.
KCLMImplementationResultsCharacteristicsofInterventionists
Nineinterventionistpositionswerefilledby11teachersduringtheyear. Interventionteacherswereselectedandhiredbyindividualschooldistrictsandmettheplannedcharacteristics,accordingtoschoolpersonnel. Atotalofnineinterventionistswerehiredinthesummerof2010. Duringthecourseoftheschoolyear,twointerventionistsleftandwerereplaced,foratotalof11interventionists.TheseinterventionistsimplementedtheKCLMtargetedinterventionswithinthenineschools.Oneinterventionistwasmaleand10werefemale.Allinterventionistswerewhite.Alloftheinterventionistshadamastersdegreeorhigher,andthree(27%)werecertifiedasreadingspecialists.Interventionistshadanaverageof12.9yearsofexperience.ImplementationofProfessionalDevelopmentModel
Interventionisttraining.Trainingwasprovidedtointerventionistsforfivedaysinthesummer,andthreedaysduringtheschoolyear. Table2.1showsthecontentamountanddeliveryoftheprofessionaldevelopmentmodel.
17SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
Table2.1Contentanddeliveryofprofessionaldevelopmenttraining(KCLM)
Numberof Minutestotal
Percentoftraining
Content Housekeeping 70 2.77Overviewofgrant 220 8.71Collaboration 275 10.89Readingstrategies&Strategicprocessing 640 25.30Communication 175 6.93Motivationandbehaviormanagement 225 8.91Foundationalreading–basicskills 60 2.38Assessment 360 14.26Planningwork 280 11.09Planningdiscussion 115 4.55Projectbasedlearning 110 4.36
Delivery
FormatWholegrouplecture 790 31.23Wholegroupdiscussion 1035 40.91Smallgroup 340 13.44Individual 365 14.43
Professionaldevelopmentinputs.Duringthesummertraining,sevenofthenineinterventionistsattendedallfivedays. Oneinterventionistattendedfourdaysandwasabsentonedayduetoillness. Thismaterialwascoveredwiththeabsenteeduringthefollowingday’strainingduringlunchandbreaks. Anotherinterventionistdidnotattendthesummertraining.Thatinterventionistwastrainedone-on-oneduringthreedaysattheinterventionist’sschool.Duringtheschoolyeartherewerethreeregulartrainingdates. Allinterventionistsattendedtwoofthesedates,andeightattendedthethird. TheabsentinterventionistmadeupthemissedtrainingdateviaSkypewithKDEstaff. InterventionistsalsoattendedtheKentuckyReadingAssociationconference. Allnineinterventionistsattendedallthreedaysoftheconference.Attendanceandparticipationwasadequateforallinterventionists.
Inadditiontosummertraining,interventionistsreceivedsitevisitsupportfromKDEstaff
throughouttheyear. Thenumberofvisitsrangedfromfourdaysto11daysperinterventionistatsixhoursperdaywithanaverageof5.56days. Whenextrasupportwasneeded,additionalvisitswerescheduled.Itemsdiscussedduringthevisitswerestrategyimplementation,project-basedlearning,datacollection,studentengagementandmotivation,assessmentproceduresandothervarioustopicsdependingontypeofsupportneededrelatedtotheKCLMmodel.
InterventionistsreceivedsupportthroughtheuseofaNING,Skype,emailandphonecalls.
Phonecalls,emailandSkypewereusedforongoingsupportandprofessionaldevelopment. The
18SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
NINGwasusedforbookstudies,discussionforums,bloggingandsharingresources.KDEstaffestimatethateachinterventionistwassupportedanestimatedthreetimesperweekviaphonecallsandemails,anestimatedonceperweekviaNING,andoncepermonthviaSkype.
Allparticipationintheprofessionaldevelopmenttrainingandsupportwasconsideredtobeadequateforallinterventionists.
Professionaldevelopmentforadministrators. Administratorswereprovidedwithonetrainingdayinthesummerof2010todiscusstherequirementsofthegrantandwaystheycouldsupporttheintervention. Administratorsfromallnineschoolsattended.
Duringtheschoolyear,KDEstaffmetwithschooladministratorstodiscussgrantrequirements,evidence-basedcomponentsofthetargetedintervention,schedulingissues,observationsofinterventionists,literacyleadership,literacyplanning,andmeetingtheneedsofadolescentlearners—especiallylowachievingstudents. Thenumberofmeetingsrangedfromtwotosixperschoolwithanaverageof3.11meetingsperschool. KDEstaffalsosharedaKCLMUpdatenewsletterwithalladministratorsfourtimesthroughouttheyear. Alladministratorparticipationwasconsideredtobeadequate.ImplementationofClassroomModel
Classsize,intensity,andduration. Classsizesvariedthroughouttheyearduetostudentattrition(i.e.,transferringschools,droppingout,etc.). Classesrangedfromasfewas12studentstoasmanyas20.Theinterventionclassmeteverydaythroughoutthecourseoftheyear,andrangedfrom45minutesto75minutesdaily.
ClassroomImplementationResults. TheobservationprotocolincludedratingsforbothpresenceofindicatorsforeachcomponentforKCLM,andqualityofeachcomponentofKCLM.Thenextfourtablesshowthenumberoftimesinterventionistswereobservedincorporatingindicatorsofmotivationandengagement,strategicprocessing,instructionalstrategies,andcommunicationskillsintheirlessons.Table2.2indicatesthemodelcomponentsmostwidelyimplementedbyteachersandthenumbersandpercentsoflessonsthatincludedthosecomponents.Othermodelcomponentswereobservedinfewerthan50%oflessons.
19SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
Table2.2
MostWidelyImplementedModelComponents
ModelComponent Round1N=18
% Round2 %N=18
Motivation&EngagementTeachermakesconnectionsbetweenlearningactivitiesandrealworldissuesTeachervariesinstructionalformat,i.e.groupwork,lecture,andpartnerworkStudentpriorknowledge,interestandbackgroundusedindeterminingcontent
StrategicProcessingTeacherexplainscognitivestrategiesforcomprehensionTeachermodelsusingcognitivestrategiesforcomprehensionTeacherencouragesandprovidesopportunitiesforstudentstopracticeusingcognitivestrategiesforcomprehension
InstructionalStrategiesTeacherexplicitlyincorporateshigherlevelquestionsTeacherusesinstructionaltoolstosupportstudentcomprehensionorvocabularyTeacherdescribes,explains,andprovidesanexampleofnewkeyterms
CommunicationSkillsTeacherusesduringtext-baseddiscussionstrategiesTeacherincludeswritingtolearnactivities
ForeachKCLMcomponent,interventionistsfocusedonsomecomponentsmorethanothers.Formotivationandengagement,interventionistsweremorelikelytomakeconnectionsbetweenthelessonandrealworldissues,tovaryinstructionalformat,andtotiethecontentintostudentexperiences. Interventionistswerenotaslikelytoincorporatetechnology,providestudentfeedback,facilitateproblemsolvingandprovidestudentswithchoices.Forstrategicprocessing,interventionistsweremorelikelytoexplain,modelandencouragestudentpracticeofcognitivestrategies. Interventionistswerenotaslikelytoaskstudenttoreflectoncognitiveprocessingandreadingcomprehensionorprovideexplicitinstructioninfoundationalreadingskills.Overall,interventionistswerenotaslikelytoincorporateinstructionalstrategiesintotheirlessons. Inthefirstroundofobservations,interventionistsdiduseinstructionaltoolstosupportstudentcomprehensionorvocabulary,andinthesecondroundtheywerelikelytodescribe,
13 72 9 50
9 50 9 50
11 61 12 67
14
78
9
50
12 67 10 56
15 83 13 72
7
39
10
56
12 67 6 33
8 44 11 61
11
61
6
33
15 83 9 50
Cantrell,Carter,&Rintamaa(2012)
20SRCohortIIEvaluationReport:Kentucky
explainandgiveexamplesofnewvocabularywords.Communicationskillswerealsounder-utilizedbyinterventionists. Therewassomeevidenceofteacherusesduringtext-baseddiscussionstrategies,andwritingtolearnactivities.
ClassroomobservationsalsoincludedaqualityscoringforeachKCLMcomponent. Table2.3showseachteacher’squalityscorebyKCLMcomponentforeachroundofobservation. Eachroundrepresentsanaveragebetweenscoresforeachclassperiodobserved.Table2.3showsthemeanscoresbyteacherforqualityofeachKCLMcomponentandtheoverallscoreforthemodel.Intermsofoverallqualityofimplementation,implementationofKCLMwasadequatein3ofthe9classrooms.
Table2.3
MeanScoresbyTeacherforQualityofKCLMcomponent
Classrooms
123456789
Note.1-1.4=Developing,1.5-2=Adequate,2.1-3=Exemplary
Experiencesforcontrolstudentsduringinterventionperiod. Studentswhowereselectedforthecontrolgroupreceivedaregularelectiveaspartoftheirfreshmanprogram.Awiderangeofelectivesweretakenincludingband,chorus,civics,andphysicaleducation.
Additionalreadingprograms.Intwoschools,additionalreadingassistanceprogramswere
providedtostudentswhoqualified. AtoneschoolstudentsfromboththeinterventionandcontrolgroupstookanunstructuredreadingclasswheretheteacherhadaccesstomaterialsfromStudyIslandandDiscoveryEducationaswellasRead180andSystem44.Inthesecondschool,studentsinbothinterventionandcontrolgroupstookaonesemesterclasscalledReadingRevisited,ahighlystructuredclassrelyingheavilyonvocabularyworkbookexercises.
Implicationsforimpactanalysis.Twofactorsrelatedtoimplementationshouldbe
consideredwheninterpretingtheimpactanalysistofollow.First,teacherswerelearningtoimplementtheinterventionastheywereimplementingit,andthisresultedinavariationin
MotivationEngagement
Mean
StrategicProcessingMean
InstructionalStrategiesMean
CommunicationSkillsMean
OverallscoreMean
1 1 1 1 11.5 1.5 1 1.5 1.382.25 2 2 2 2.061 1 1 1 12 2 2 1.5 1.882 2 2 1.5 1.881 1 1 1 11.5 1 1 1 1.131 1 1 1 1
Cantrell,Carter,&Rintamaa(2012)
20SRCohortIIEvaluationReport:Kentucky
implementationquality.Teacherswereadheringtothemajorcomponentsoftheintervention,buttheydidnotnecessarilyimplementthosecomponentsastheyweredesignedtobeimplemented.Therefore,impactfindingsshouldnotbedirectlyattributedtotheinterventionmodelasitisdesigned.Second,thatsomeschoolscontinuedtoimplementreadingprogramstargetedatlowachievingreaderspresentsaconfoundingfactorthatmayhaveinfluencedoutcomesforbothtreatmentandcontrolgroups.
StudyDesign
EvaluationofImpact
Samplingselectionprocess. Evaluatorsimplementedastratifiedrandomsamplingprocedureforstudentswithineachschoolusingfourdemographicvariables: specialeducationstatus,free/reducedlunchstatus,ethnicity,andgender. Withineachschoolstudentsweresortedbydemographicvariablescreatingsubgroupsofstudents,andstudentswithineachsubgroupwerethensortedbyassessmentscore. Usingarandomnumbergeneratortoassignthefirststudenttoeithertheinterventionorcontrolgroup,thestudentswerethenalternatelyassignedsequentiallytotheinterventionorcontrolgroup.
Evaluatorsprovidedalistofstudentsthatqualifiedfortheinterventiongrouptothenineparticipatinghighschoolsduringthesummerof2011. Thesestudentswerescheduledtobeintheinterventionclass,andstudentsandparentswerenotifiedwhentheclassschedulesweregiventoallstudentsatthebeginningofthefallsemester. Schoolswerenotprovidedthenamesofthestudentsinthecontrolgroup.
Astudentwasremovedfromthestudypost-randomassignmentifhe/shedroppedoutofschoolormoved/transferredtoaschoolnotparticipatingintheStrivingReadersprogram.Also,ifthestudentdidnottaketheposttestinthespringof9thgrade,theywereremoved.Finally,astudentwasineligiblepost-randomassignmentif,afterthestudentenrolledintheninth-grade,thehighschoolplacesthestudentinalldayresourceclasses. Theschoolswereinstructedtoinformtheevaluatorsattheendofthestudyifacontrolstudentwasplacedinalldayresourceclasses. Thusthecriterionthehighschoolusedtoassignastudenttoalldayresourcewouldnotbeaffectedbytheresultsoftherandomassignment,andwasappliedequallytointerventionandcontrolstudents. Therewere13interventionstudentsthatwereassignedtoalldayresourceclassespost-randomassignment,andareineligibletobeinthestudy. Thereweresixteencontrolstudentswhowereplacedinalldayresourceclassesafterrandomassignment.
Samplesize.Exhibit1showsthesamplesizeresultsforreadingachievementtest,GRADE,dividedbycondition. Outofthepopulationof2,229eighthgradestudents,637wereidentifiedaslowachievingreaders(NCEof40orlower). Threehundrednineteenstudentswereassignedtothetreatmentgroup,and318studentswereassignedtothecontrolgroup.Afterrandomassignment,13studentsinthetreatmentgroupand16studentsinthecontrolgroupwereassignedtoalldayresourceclassesafterenrollinginhighschoolsowereineligibleforthestudy. Additionally,22studentsinthetreatmentgroupdidnotenrollinaparticipating
Cantrell,Carter,&Rintamaa(2012)
22SRCohortIIEvaluationReport:Kentucky
highschool. Thenumberofstudentsinthecontrolgroupthatdidnotenrollinaparticipatinghighschoolisnotavailable.
Therewere306targeted,eligiblestudentsatbaselineintheinterventiongroupand302
studentsinthecontrolgroup. Attritionintheinterventiongroupattheendoftheschoolyeartotaled74students(36studentsmovedand38studentsdidnottakethespringtest),resultinginananalyticsampleof232interventionstudents. Attritioninthecontrolgrouptotaled49students(34studentseitherdidnotenrollintheparticipatinghighschoolormovedduringtheyearand15studentsdidnottakethespringtest),resultinginananalyticsampleof253controlstudents.
Cantrell,Carter,&Rintamaa(2012)
23SRCohortIIEvaluationReport:Kentucky
Exhibit2showsthesamplesizeresultsforthewritingachievement,bycondition. Thetargetednumberofeligiblestudentsatbaselineintheinterventiongroupwas306,with302studentsinthecontrolgroup. Attritionintheinterventiongroupduringtheschoolyeartotaled134students(36studentsmovedand98studentsdidnottakethespringwritingtest),resultinginananalyticsampleof172interventionstudents. Attritioninthecontrolgrouptotaled109students(34studentseitherdidnotenrollintheparticipatinghighschoolormovedduringtheyear,and75studentsdidnotcompletethespringsurvey),resultinginananalyticsampleof193controlstudent.
Cantrell,Carter,&Rintamaa(2012)
24SRCohortIIEvaluationReport:Kentucky
Exhibit3showsthesamplesizeresultsforstudentsurvey,bycondition. Thetargetednumberofeligiblestudentsatbaselineintheinterventiongroupwas306,with302studentsinthecontrolgroup. Attritionintheinterventiongroupduringtheschoolyeartotaled152students(36studentsmovedand116studentsdidnotcompletethespringsurvey),resultinginananalyticsampleof154interventionstudents. Attritioninthecontrolgrouptotaled129students(34studentseitherdidnotenrollintheparticipatinghighschoolormovedduringtheyear,and95studentsdidnotcompletethespringsurvey),resultinginananalyticsampleof173controlstudent
Cantrell,Carter,&Rintamaa(2012)
25SRCohortIIEvaluationReport:Kentucky
ImpactMeasuresandDataCollection
ThefollowingmeasureswereusedtoascertaintheimpactoftheKCLMinterventiononstudents’readingandwritingachievement,readingstrategyuse,andreadingmotivation.
GroupReadingandDiagnosticEvaluation(GRADE).TheGRADEisanationallynormedreadingassessment,anditincludesnormalizedscoresforoverallreadingachievement. NormalCurveEquivalentscores(NCEs)includingvocabularyandcomprehensionitemscloselyalignstudentoutcomeswiththegoalsoftheintervention.Inthespringoftheplanningyear,theGRADE(levelM)wasgiventoalleighth-gradestudentsinparticipatingfeederschoolsasapretest. Inthefollowingspring,theGRADE(levelH)wasgiventoallninth-gradestudentsasapostest.
KentuckyStateWritingAssessment. TheKentuckystatewritingassessmentprovidesaholisticscorebasedonanalyticcategorieswhichcloselyalignstudentoutcomeswiththegoalsoftheintervention. TheStateWritingAssessmentwasgiventoalleighth-gradestudentsinspringoftheplanningyear,andallninth-gradestudentsinspringofthefollowingyear. Thetestisdesignedsothatstudentsgetachoicebetweentwowritingtaskswhichincludethreepossiblemodesofwriting(inform,narrateforapurpose,orpersuade)andfourpossibleresponseformats(article,editorial,letterorspeech). Inadditiontoawritingtask,eachstudentisalsopresentedwithdraftversionsofthreepiecesofwriting. Fourmultiple-choicequestionsdealingwithediting/revisingareprovidedwitheachofthethreedraftsso,eachstudentalsorespondsto12multiple-choicequestions.
MetacognitiveAwarenessofReadingStrategiesInventory(MARSI).TheMARSI(Mokhtari&Reichard,2002)isastudentself-reportmeasuredesignedspecificallytoassessadolescents’perceiveduseofreadingstrategiesduringacademicreading.Forthisstudy,theMARSIwasadaptedtomeasureself-efficacywithstrategyuse.“Ican”wasaddedtothebeginningofeachstrategystatementinthesurvey.Thesurveywasgiventoallstudentsinfallandspringofninthgrade. Thesurveyitemsarepresentedonascaleof1to5,where1isequalto“notatallconfident”and5isequalto“completelyconfident.” Exampleitemswouldbe“IamabletohaveapurposeinmindwhileIread”or“IcantakenoteswhilereadingtohelpmeunderstandwhatIread.”
AdolescentMotivationSurvey. Inthespringofeighthgradeandinthespringofninthgrade,allstudentsinparticipatingschools(andeighth-gradefeederschools)completedasurveymeasuringseveraldimensionsofintrinsicreadingmotivation(challenge,curiosity,intrinsictaskvalue,attainment),extrinsicreadingmotivation(extrinsictaskvalue,compliance),readingrelatedself-beliefs(expectancy,difficulty)andleisurereading.Thesurveyitemsarepresentedonascaleof1to5andthevaluevariesdependingonthedimensionmeasured. Asampleitemforthechallengedimensionis“Ilikehard,challengingbooks”where1equals “notatalltrueand5equals“verytrue.” Asampleitemfromtheleisuredimensionis“Howmuchtimehaveyouspentreadingamagazinethisweek?”with1equaling“none,”and5equaling“morethanfivehours.” Reliabilityforsubscaleswere>.70withtheexceptionofleisure
Cantrell,Carter,&Rintamaa(2012)
26SRCohortIIEvaluationReport:Kentucky
(a=.585)andextrinsictaskvalue(a=.60).Thesescalesweretakenfromexistingmeasures(Eccles&Wigfield,1995;Hopper,2005;Wigfield&Guthrie,1997).SummaryofAnalyticApproach
HierarchicalLinearModels(HLMs)wereusedtoestimatetheimpactoftheKCLMonstudentachievement,motivation,andreadingstrategiesoutcomes.TheGRADENormalCurveEquivalents(NCEs)wereusedtoestimatetheimpactoftheKCLMinterventiononachievement.HolisticscoresfromthestatewritingassessmentwereusedtoestimatetheimpactoftheKCLMonwritingachievement.TheaverageoftheitemsontheMARSIwasusedtoestimatetheimpactonreadingstrategyuse,andtheaverageoftheitemsontheAdolescentMotivationSurveywasusedtoestimatetheimpactonmotivation.
Atwo-levelHLMmodel(studentsassignedtointerventionorcontrolgroupwithinschools)willbeusedtodeterminetheimpactofKCLM.Fourhypotheseswillbetested:
H1: TheKCLMinterventionhasnoimpactonstudentachievement.H2: TheKCLMinterventionhasnoimpactonstudentwritingachievement.H2: TheKCLMinterventionhasnoimpactonselfefficacyofstudentstrategyuse.H3: TheKCLMinterventionhasnoimpactonstudentmotivation.
Level-1HLM:StudentLevel.Atthestudentlevel,thespringoutcomevariable(reading
achievement,writingachievement,strategyuse,ormotivation)willbemodeledasafunctionoffalloutcomevariables(covariate),intervention/controlstatusandfourdemographicvariables:gender,ethnicity,free/reducedlunchstatus,andspecialeducation.
Level-2HLM:SchoolLevel.Thisanalysiswillbeperformedonninth-gradestudents’scoresfromninehighschools. InadditiontothebaseyearReadingKCCTscore,otherschoollevelvariablesthatwillbeincludedaretheschoolpercentofstudentsqualifyingforfreeorreducedlunchfees,schoolpercentofwhitestudentsintheschool,andschoolpercentofblackstudents,andthepercentofstudentswithdisabilities.DescriptionoftheFirstYearSample
Ninehighschoolsgeographicallydistributedacrossthestateparticipatedinthestudy.HighschooldemographicdatawascollectedfromtheKentuckyDepartmentofEducationwebsiteforthe2009-10academicyear. Theaveragenumberof10th-12thgradestudentsenrolledattheparticipatingschoolswas454,rangingfrom301to803students. TheaveragepercentofWhitestudentswas91.58%(53.9%,98.8%),andtheaveragepercentBlackstudentswas5.76%(0%,37.7%). Theaveragepercentofstudentsreceivingfree/reducedlunchwas62.25%(40.7%,83.5%),andtheaveragenumberofstudentsenrolledinspecialeducationclasseswas12.73%,rangingfrom7.2%to16.5%.
Studentdemographicdatawascollectedfromthemiddleschoolsforeveryenrolledeighth-gradestudent.Schoolswerecontactedforstudentdemographicinformationifthere
Cantrell,Carter,&Rintamaa(2012)
27SRCohortIIEvaluationReport:Kentucky
werestudentsthatwerenotontheselistsbutcompletedtheeighth-grade.Table2.4showsdemographiccharacteristicsofstudentsintheinterventionandcontrolgroups.StudentsintheStrivingReadersclassesweretypicallyWhitemales,receivingfree/reducedlunchservicesandarenotassignedtospecialeducationclasses. Studentsthatreceivedtheinterventionwereverysimilarindemographicsascomparedtothecontrolgroup,withthepossibleexceptionofgender,whereaslightlyhigherpercentofmalesreceivedtheintervention.
Table2.4
InterventionandControlStudentDemographics(andProportions)
Gender Ethnicity Lunch Special EducationGroup Male Female White Minor-
ityReg
PayFree/Red
NotIn
SpecialEd.
Interv
136
96
205
27
40
192
161
71
(.59) (.41) (.88) (.12) (.17) (.83) (.69) (.31)
Contl 140 113 219 34 45 208 179 74 (.55) (.45) (.87) (.13) (.18) (.82) (.71) (.29)
Total
276
109
424
61
85
400
340
145
(.57) (.43) (.88) (.12) (.38) (.62) (.73) (.16)
ImpactsonStudents
Table2.5belowshowstheresultsonstudentreadingachievement,writingachievement,readingstrategyuseselfefficacyandreadingmotivationforinterventionandcontrolstudentsafteroneyearofintervention. Theunadjustedmeansandstandarddeviationsforeachmeasureisdisplayed,andthemeansadjustedfortheHLMresultsaredisplayed. Theestimatedimpactoftheintervention,theeffectsize,andthesignificancelevelareshown.
Cantrell,Carter,&Rintamaa(2012)
28SRCohortIIEvaluationReport:Kentucky
Table2.5ImpactoftheTargetInterventiononStudentReadingAchievement,WritingAchievement,StrategyUseSelfEfficacy,andMotivation
Unadjusted HLM-adjusted
Means Means Control Tx Control Tx Estimated
ImpactEffectSize
p
ReadingAchievementSpringNCE 36.7 35.6 37.2 36.4 -0.79 -0.059 .439 (13.49) (13.48)
No.ofstudents 253 232
WritingAchievement
No.ofstudents 193 172
StrategyUseSelfEfficacySpring score 3.3 3.5 3.4 3.5 0.154 0.250 .012*
(0.62) (0.61)No.ofstudents 173 154
MotivationSpringscore 2.9 3.0 2.9 3.0 .128 0.217 .015*
(0.59) (0.62)
No.ofstudents 173 154
Note. Standarddeviationsarepresentedinparenthesis. Effectsizecalculatedastheimpactdividedbythecontrolgroupstandarddeviation.*Designatesstatisticalsignificanceatthe.05levelofsignificance.
Thereisnosignificanteffectoftheinterventionatthe.05levelonthereadingand
writingachievementafterthefirstyearoftheprogram. However,asignificanteffectoftheinterventionisshownforstrategyuseselfefficacyandreadingmotivation.
Springscore 826.3 827.2 826.9 827.6 0.62 0.066 .481 (9.41) (11.78)
Cantrell,Carter,&Rintamaa(2012)
29SRCohortIIEvaluationReport:Kentucky
DiscussionandConclusions
TheimpactresultsfromoneyearofstudydonotrevealsignificanteffectsoftheKCLMinterventiononstudents’readingorwritingachievement,butthereweresignificantimpactsonstudents’self-efficacyforstrategyuseandstudents’readingmotivation.Developmentofdemonstrableimprovementsinreadingperformancemayrequiregreaterlengthsoftimetogaincomfortwithflexiblestrategyuseandtoreapbenefitsofincreasedmotivation.Whileitseemsthestudentsinthisstudyreportedincreasedconfidencewithusingreadingstrategies,theymaynothaveinternalizedandpracticedstrategyusetoasufficientenoughextenttoachievepurposefulflexibleuseunderawiderangeofconditions.Nevertheless,theimpactsoftheinterventiononstudents’strategyuseandmotivationarenoteworthygiventheemphasisplacedonthesedimensionsoflearninginrecommendationsforimprovingadolescents’literacyachievement(Biancarosa&Snow,2006;Kamil,Borman,Dole,Kral,Salinger,&Torgesen,2008).
Althoughtheplannedprofessionaldevelopmentmodelwasimplementedathighlevels,implementationoftheclassroommodelwaslower.ElementsoftheKCLMmodelwereevidentineachclassroomobservation,buttheteachersinthisprojectimplementedsomecomponentsoftheinterventiontoagreaterextentthanotherimportantcomponents.Also,themajorityofteachersimplementedtheinterventionatdevelopinglevelsofquality.Higherlevelsofimplementationmayhaveresultedinhigherlevelsofachievementforstudents.Itisprobablethatteacherswouldhavebeenabletoachievehigherlevelsofimplementationinfutureyearswithongoingsupporthadtheprojectextendedforthefullplannedprojectduration.
Cantrell,Carter,&Rintamaa(2012)
31SRCohortIIEvaluationReport:Kentucky
References
Allington,R.(2007).Effectiveteachers,effectiveinstruction.InBeers,K.,Probst,R.E.,&Rief,L.(Eds.),Adolescentliteracy:Turningpracticeintopromise(pp.273-288).Portsmouth,NH:Heinemann.
Almasi,J.(2003).Teachingstrategicprocessesinreading.NewYork:TheGuilfordPress.
Anderson,V.,&Hidi,S.(1988/1989).Teachingstudentstosummarize.EducationalLeadership,46,26-28.Retrievedfromhttp://www.ascd.org/publications/educational-leadership.aspx
Armbruster,B.B.,Anderson,T.H.,&Ostertag,J.(1987).Improvingcontent-areareadingusing
instructionalgraphics.ReadingResearchQuarterly,26(4),393-416.Retrievedfromhttp://www.reading.org/general/publications/journals/rrq.aspx
Bandura,A.(1993).Perceivedself-efficacyincognitivedevelopmentandfunctioning.
EducationalPsychologist,28(2),117-148.doi:10.1207/s15326985ep2802_3
Bandura,A.(1997).Theexerciseofcontrol.NewYork,NY:Longman.
Biancarosa,G.,&Snow,C.E. (2006). Readingnext–Avisionforactionandresearchinmiddleandhighschoolliteracy: AreportfromtheCarnegieCorporationofNewYork.Washington,DC: AllianceforExcellentEducation.
Blank,W.(1997).Authenticinstruction.InW.E.Blank&S.Harwell(Eds.),Promisingpractices
forconnectinghighschooltotherealworld(pp.15-21).Tampa,FL:UniversityofSouthFlorida.(ERICDocumentReproductionServiceNo.ED407586)
Bloome,D.(1986).Readingasasocialprocessinamiddleschoolclassroom.InD.Bloome(Ed.),
Literacyandschooling(pp.123-149).Ablex:Norwood,NJ.Bransford,J.,Brown,A.,&Cocking,R.(1999).Howpeoplelearn:Brain,mind,experience,and
school.Washington,DC:NationalAcademyPress.Brewster,C.andFager,J.(2000).IncreasingStudentEngagementandMotivation:FromTime-
on-TasktoHomework.NorthwestRegionalEducationalLaboratory.RetrievedJuly9,2009,fromtheWorldWideWeb:http://www.nwrel.org/request/oct00/textonly.html.
Brookbank,D.,Grover,S.,Kullberg,K.,&Strawser,C.(1999).Improvingstudentachievement
throughorganizationofstudentlearning.Chicago:Master'sActionResearchProject,SaintXavierUniversityandIRI/Skylight.
Caine,R.N.,&Caine,G.(1994).Makingconnections:Teachingandthehumanbrain.Menlo
Cantrell,Carter,&Rintamaa(2012)
30SRCohortIIEvaluationReport:Kentucky
Park,CA:Addison-WesleyCampbell,D.T.&Stanley,J.C.(1963).Experimentalandquasi-experimentaldesignsfor
research.Chicago:Rand-McNally.Carney,P.(2007).KentuckyReadingFirstEvaluation.UniversityofKentucky.Cartwright,K.(2008).Literacyprocesses:Cognitiveflexibilityinlearningandteaching.New
York:GuilfordPress.Chen,Z.(1999).Children'sanalogicalproblemsolving:Theeffectsofsuperficial,structural,and
proceduralsimilarities.JournalofExperimentalChildPsychology,62(3),410-431.doi:10.1006/jecp.1996.0037
Cook,T.D.,&Campbell,D.T.(1979).Quasi-experimentation:Designandanalysisfor
fieldsettings.Boston:HoughtonMifflin.Creswell,J.W.&PlanoClark,V.L.(2007).Designingandconductingmixedmethodsresearch.
ThousandOaks,CA:Sage.Creswell,J.W.(1998).Qualitativeinquiryandresearchdesign:Choosingamongfive
traditions.ThousandOaks,CA:Sage.Deshler,D.,Palincsar,A.,Biancarosa,G.,&Nair,M.(2007).Informedchoicesforstruggling
adolescentreaders.Newark,Delaware:InternationalReadingAssociation.Dev,P.C.(1997).Intrinsicmotivationandacademicachievement.Whatdoestheirrelationship
implyfortheclassroomteacher?RemedialandSpecialEducation,19(1),12-19.doi:10.1177/074193259701800104
Durkin,D.(1978-79).Whatclassroomobservationsrevealaboutreadingcomprehension
instruction.ReadingResearchQuarterly,15,481-533.doi:10.1598/RRQ.14.4.2Dyson,A.H.(1992).WhistleforWillie,lostpuppies,andcartoondogs:Thesociocultural
dimensionsofyoungchildren'scomposing.JournalofReadingBehavior,24(4),433-462.Eccles,J.S.,&Wigfield,A.(1995).Inthemindoftheactor:Thestructureofadolescents’
achievementtaskvaluesandexpectancy-relatedbeliefs. PersonalityandSocialPsychologyBulletin,21(3),215-225.doi:10.1177/0146167295213003
Fink,R.(2006).WhyJaneandJohncouldn’tread—andhowtheylearned:Anewlookatstriving
readers.Newark,DE:InternationalReadingAssociation.Fisher,D.,Frey,N.,&Rothenberg,C.(2008).Content-areaconversations:Howtoplan
Cantrell,Carter,&Rintamaa(2012)
32SRCohortIIEvaluationReport:Kentucky
discussion-basedlessonsfordiverselanguagelearners.Alexandria,VA: AssociationforSupervisionandCurriculumDevelopment
Flavell,J.H.(1979).Metacognitionandcognitivemonitoring:Anewareaofcognitive-
developmentalinquiry.AmericanPsychologist,34,906-911.doi:10.1037/0003-066X.34.10.906
Fugate,M.&Waterman,K.(2004).Groupreadingassessmentanddiagnosticevaluation
reliabilityregardingeducationaldecisionmaking.JournalofEducationalPsychology,96(2),187-201.
Gambrell,L.B.,Palmer,B.M.,Codling,R.M.,&Mazzoni,S.A.(1996).Assessingmotivationto
read.TheReadingTeacher,49,518-533.doi:10.1598/RT.49.7.2Graham,S.,&Perin,D.(2007).Writingnext:Effectivestrategiestoimprovewritingof
adolescentsinmiddleandhighschools.AreporttoCarnegieCorporationofNewYork.Washington,DC:AllianceforExcellentEducation.
Guthrie,J.T.,&Wigfield,A.(2000).Engagementandmotivationinreading.InM.L.Kamil,P.B.
Mosenthal,P.D.Pearson,&R.Barr(Eds.),Handbookofreadingresearch:VolumeIII(pp.403-422).NewYork:Erlbaum.
Hopper,R.(2005).Whatareteenagersreading?Adolescentfictionreadinghabitsandreading
choices.Literacy39(3),113-120.doi:10.1111/j.1467-9345.2005.00409.xcKamil,M,L.,Borman,G.D.,Dole,J.,Kral,C.C.,Salinger,T.S.,&Torgesen,J.(2008). Improving
adolescentliteracy:Effectiveclassroomandinterventionpractices. Washington,D.C.:U.S.DepartmentofEducation,InstituteofEducationSciences,NationalCenterforEducationEvaluationandRegionalAssistance.
Kane,S.(2007).Literacyandlearninginthecontentareas.Scotsdale,AZ:HolcombHathaway.Kushman,J.W.,Sieber,C.,&Heariold-Kinney,P.(2000).Thisisn'ttheplaceforme:School
dropout.InD.Capuzzi&D.R.Gross(Eds.),Youthatrisk:Apreventionresourceforcounselors,teachers,andparents(3rded.,pp.471-507).Alexandria,VA:AmericanCounselingAssociation.
Langer,J.A.(2001).Guidelinesforteachingmiddleandhighschoolstudentstoreadandwrite
well.Albany,NY:NationalResearchCenteronEnglishLearningandAchievement,StateUniversityofNewYorkatAlbany.
Lapp,D.&Fisher,D.(2009).Introduction.InLapp,D.&Fisher,D.(Eds.),Essentialreadingsin
comprehension.Newark,DE:InternationalReadingAssociation.
Cantrell,Carter,&Rintamaa(2012)
33SRCohortIIEvaluationReport:Kentucky
Lee,V.E.,&Smith,J.B.(1996).Collectiveresponsibilityforlearninganditseffectsongainsinachievementforearlysecondarystudents.AmericanJournalofEducation,104(2),103-147.doi:10.1086/444122
Lenz,B.K.,&Hughes,C.A.(1990).Awordidentificationstrategyforadolescentswithlearning
disabilities.JournalofLearningDisabilities,23(3),149-158.doi:10.1177/002221949002300304
Marzano,R.(1998).ATheory-BasedMeta-AnalysisofResearchonInstruction.Mid-Continent
RegionalEducationalLab.,Aurora,CO.Marzano,R.(2004).Buildingbackgroundknowledgeforacademicachievement:Researchon
whatworksinschools.Alexandria,VA:AssociationforSupervisionandCurriculumDevelopment.
Marzano,R.J.,Pickering,D.J.,&Pollock,J.E.(2001).Classroominstructionthatworks:
Research-basedstrategiesforincreasingstudentachievement.Alexandria,VA:AssociationforSupervisionandCurriculumDevelopment.
Mastropieri,M.A.,Scruggs,T.E.,Hamilton,S.L.,Wolfe,S.,Whedon,C.,&Canevaro,A.(1996).
Promotingthinkingskillsofstudentswithlearningdisabilities:Effectsonrecallandcomprehensionofexpositoryprose.Exceptionality,6(1),1-11.doi:10.1207/s15327035ex0601_1
Matsumura,L.,Garnier,H.,Pascal,J.,andValdes,R.(2002).Measuringinstructionalqualityin
accountabilitysystems:Classroomassignmentsandstudentachievement.EducationalAssessment,1532-6977,8(3),207–229.doi:10.1207/S15326977EA0803_01
Midgley,C.,Kaplan,A.,Middleton,M.,Maehr,M.L.,Urdan,T.,Anderman,L.H.,Anderson,E.,
&Roeser,R.(1998). Thedevelopmentandvalidationofscalesassessingstudents‘achievementgoalorientations.ContemporaryEducationalPsychology,23,113-131.doi:10.1006/ceps.1998.0965
Moats,L.C.(1998).Teachingdecoding.AmericanEducator,Spring/Summer,42-49.Retrieved
fromhttp://www.aft.org/newspubs/periodicals/ae/Mokhtari,K.&Reichard,C.A.(2002).Assessingstudents‘metacognitiveawarenessof
readingstrategies.JournalofEducationalPsychology,94(2),249-259.doi:10.1037/0022-0663.94.2.249
Moore,D.W.,&Readence,J.E.(1984).Aquantitativeandqualitativereviewofgraphic
organizerresearch.JournalofEducationalResearch,78(1),11-17.Retrievedfromhttp://www.tandf.co.uk/journals/titles/00220671.asp
Cantrell,Carter,&Rintamaa(2012)
34SRCohortIIEvaluationReport:Kentucky
NationalCouncilofTeachersofMathematics.(2000).Principlesandstandardsforschoolmathematics.Reston,VA.
Patrick,H.,Ryan,A.M.,&Kaplan,A.(2007).Earlyadolescents‘perceptionsoftheclassroom
socialenvironment,motivationalbeliefs,andengagement.JournalofEducationalPsychology,99,83-98.doi:10.1037/0022-0663.99.1.83
Pearson,P.D.,Roehler,L.R.,Dole,J.A.,&Duffy,G.G.(1992).Developingexpertiseinreading
comprehension.InJ.Samuels.,&A.Farstrup,(Eds.),Whatresearchhastosayaboutreadinginstruction.Newark,Delaware:InternationalReadingAssociation.
Pitcher,S.M.,Albright,L.K.,DeLaney,C.J.,Walker,N.T.,Seunarinesingh,K.,Mogge,S.,
Headley,K.N.,Ridgeway,V.G.,Peck,S.,Hunt,R.,&Dunston,P.J.(2007).Assessingadolescents‘motivationtoread.JournalofAdolescentandAdultLiteracy,50,378-396.doi:10.1598/JAAL.50.5.5
Pressley,M.,Brown,R.,El-Dinary,P.B.,&Afflerbach,P.(1995).Thecomprehensioninstruction
thatstudentsneed:Instructionfosteringconstructivelyresponsivereading.LearningDisabilitiesResearchandPractice,10,215–224.Retrievedfromhttp://mc.manuscriptcentral.com/ldrp
Raudenbush,S.,Spybrook,J.,Liu,X.,&Congdon,R.(2004)Optimaldesignforlongitudinaland
multilevelresearch:DocumentationfortheOptimalDesignsoftware.AnnArbor:UniversityofMichigan.
Redfield,D.,&Rousseau,A.(1981).Ameta-analysisofexperimentalresearchonteacher
questioningbehavior.ReviewofEducationResearch,51,237-246.doi:10.3102/00346543051002237
Ross,J.A.(1988).Controllingvariables:Ameta-analysisoftrainingstudies.Reviewof
EducationalResearch,58(4),405-437.doi:10.2307/1170280Rozzelle,J.,Searce,C.(2009).PowerToolsforAdolescentLiteracy:StrategiesforLearning.
Retrievedfromhttp://go.solution-tree.com/literacy/
Seidman,I.E.(1998).Interviewingasqualitativeresearch.NewYork:TeachersCollegePress.
Silver,H.,Strong,R.,Perini,M.(2007).TheHiddenSkillsofAcademicLiteracy.Alexandria:ThoughtfulEducationPress.
Smith,M.W.,&Wilhelm,J.D.(2006).Goingwiththeflow.Portsmouth,NH:Heinemann.Snider,VE.(1989).Readingcomprehensionperformanceofadolescentswithlearning
disabilities.LearningDisabilityQuarterly,12,87-96.doi:10.2307/1510724
Cantrell,Carter,&Rintamaa(2012)
35SRCohortIIEvaluationReport:Kentucky
StanfordAchievementTestSeries,TenthEdition(SAT-10).PearsonforEducation.
http://pearsonassess.com/HAIWEB/Cultures/en-us/Productdetail.htm?Pid=SAT10C.RetrievedJuly28,2009.
Stone,C.L.(1983).Ameta-analysisofadvancedorganizerstudies.JournalofExperimental
Education,51(7),194-199.Retrievedfromhttp://www.tandf.co.uk/journals/titles/00220973.asp
Strong,R.,Silver,H.,andPerini,M.(2001).Teachingwhatmattersmost:Standardsand
strategiesforraisingstudentachievement.Alexandria:AssociationforSupervisionandCurriculumDevelopment.
Strong,R.,Silver,H.,Perini,M.,Tuculescu,G.(2002).ReadingforAcademicSuccess.Thousand
Oaks:Corwin.TestofWrittenLanguage-Intermediate,4thEdition(TOWL-4).PearsonPsychCorp.Retrieved
fromtheWorldWideWebJuly28,2009:http://psychcorp.pearsonassessments.com/HAIWEB/Cultures/en-us/Productdetail.htm?Pid=PAa19045.
NorthCentralRegionalEducationalLaboratory.(n.d.)Thethinkingcurriculum.OakBrook,IL:
Author.[On-line].AvailableInternet:http://www.ncre.org/sdrs/areas/rpl_esys/thinking.htm
Tomlinson,C.(2001).Howtodifferentiateinstructioninmixed-abilityclassrooms.Alexandria:
ASCD.White,T.G.,Sowell,J.,&Yanagihara,A.(1989).Teachingelementarystudentstouseword-part
clues.TheReadingTeacher,42,302-309.Retrievedfromhttp://www.reading.org/general/publications/journals/rt.aspx
Wigfield,A.,&Guthrie,J.T.(1997).Relationsofchildren’smotivationforreadingtotheamount
andbreadthoftheirreading.JournalofEducationalPsychology,89(3),420-432.doi:10.1037/0022-0663.89.3.420
Wiggins,G.,&McTighe,J.(1998).Understandingbydesign.Alexandria,VA:Associationfor
SupervisionandCurriculumDevelopment.Williams,K.(2001).Groupreadingassessmentanddiagnosticevaluationtechnical
manual.CirclePines,MN:AmericanGuidanceServices.Woods,E.G.(1995).Reducingthedropoutrate.InSchoolImprovementResearchSeries(SIRS):
Researchyoucanuse(Close-upNo.17).Portland,OR:NorthwestRegionalEducationalLaboratory.
Cantrell,Carter,&Rintamaa(2012)
36SRCohortIIEvaluationReport:Kentucky
AppendixA
SummaryofAnalyticApproach
HierarchicalLinearModels(HLMs)wereusedtoestimatetheimpactoftheinterventiononstudentachievementinreadingandwriting,andmotivation,andself-efficacyoutcomes.TheGRADENormalCurveEquivalents(NCEs)wereusedtoestimatetheimpactoftheinterventiononreadingachievement,andthestandardwritingscoreswereusedtomeasuretheimpactonwriting. Theaveragesurveyscoreswereusedtoestimatetheimpactonself-efficacyinreadingstrategyuseandimpactonmotivation.
Atwo-levelHLMmodel(studentsassignedtointerventionorcontrolgroupwithinschools)wasusedtodeterminetheimpactofthetargetedintervention. Atthestudentlevel,thespringoutcomevariable(readingachievement,writingachievement,self-efficacyinstrategyuse,ormotivation)wasmodeledasafunctionoffalloutcomevariables,intervention/controlstatus,andfourdemographicvariables:gender,ethnicity,free/reducedlunchstatus,andspecialeducation.Level-1Model:StudentOutcomes(achievement,readingstrategies,ormotivation)
M
Yij=� 0j+
where
� 1j(Y*ij)+� 2j(Tij)+
� mj mij+ ijm=3
Yij isthespringstudentoutcome(post-test)scoreforstudentiatschoolj;� 0jisthemeanstudentoutcome(post-test)scoreforcontrolstudentsat schoolj;Y*ijisthefallstudentoutcome(pre-test)scoreforstudenticenteredat schoolj;� 1jistheaveragestudentoutcome(pre-test)slopeforstudentsatschoolj;Tij=1ifstudentiisassignedtoLSCinterventionatschoolj,and0ifcontrol;� 2jisthemeandifferenceofstudentoutcomepre-postgainbetweeninterventionand
controlstudentsatschoolj; mij areadditionalcovariatesrepresentingdemographiccharacteristicsofstudentiat
schoolj(gender,ethnicity,free/reducedlunch,andspecialeducationstatus);� mjarecoefficientscorrespondingtostudentdemographiccovariates(gender,
ethnicity,free/reducedlunch,specialeducationstatus),and ij istherandomeffectrepresentingthedifferencebetweenstudentij’sscoreandthe
predictedmeanscoreforschoolj. Theseresidualeffectsareassumednormallydistributedwithmean0andvariancea 2.
Level-2Model: StudentAchievement–SchoolLevel
Thisanalysiswasperformedondatafrom9thgradestudentscollectedforoneyear. Thecovariatesinthismodelpertaintotheconcurrentyearthestudentwasintheinterventionor
Cantrell,Carter,&Rintamaa(2012)
37SRCohortIIEvaluationReport:Kentucky
controlgroupwiththeexceptionoftheReadingKentuckyCoreContentTest(KCCT)score,forwhichthescoreforthebaseyear,spring,2010,wasused. InadditiontothebaseyearReadingKCCTscore,otherschoollevelcovariatesincludedenrollment,percentofwhitestudentsintheschool,percentofAfricanAmericanstudents,percentofstudentsqualifyingforfreeorreducedlunchfeesandpercentofstudentswithdisabilities..
Q
� 0j=
� 1j=� 2j=� mj
=
y 00+
y 10y 20y m0
y oqWqj
+q
µ 0j
where
y 00 isthemeanstudentoutcome(post-test)scoreof9thgradecontrolstudents
inKentuckyStrivingReadersmiddleschoolsWqj areschoollevelcovariatesincludingbaseyearReadingKCCT(spring,2010),and
averageschoolpercentfree/reducedlunch,percentwhitestudents,percentblackstudents,andpercentdisability;
y oqarecoefficientscorrespondingtoschool-levelcovariates;µ 0j istheuniqueeffectofschooljonmeanstudentoutcome,holdingWqjconstant(or
conditioningonWqj)-thiseffectisassumednormallydistributedwithmean0andvarianceT 2;
y 10isthefallstudentoutcome(pre-test)slope;y 20istheoveralltargetinterventiontreatmenteffectonspringstudentoutcome
(post-test)scores;y m0 isthefixedmthstudentcovariateeffect(gender,ethnicity,free/reducedlunch,special
educationstatus) onthespringoutcomevariable.
SelectionofCovariates. TherandomassignmentprocedureincludedallstudentdemographicvariablesintheHLMmodel,sowereincludedregardlessofsignificance.Interactioneffectswerenotconsidered.
Cantrell,Carter,&Rintamaa(2012)
38SRCohortIIEvaluationReport:Kentucky
AppendixB
StudyMeasures
StudentSurvey
Itemsforthestudentsurveywereadaptedfromthefollowingpre-existinginventories:Eccles,J.S.,&Wigfield,A.(1995).Inthemindoftheactor:Thestructureofadolescents’
achievementtaskvaluesandexpectancy-relatedbeliefs.PersonalityandSocialPsychologyBulletin,21(3),215-225.doi:10.1177/0146167295213003
Hopper,R.(2005).Whatareteenagersreading?Adolescentfictionreadinghabitsandreadingchoices.Literacy39(3),113-120.doi:10.1111/j.1467-9345.2005.00409.xc
Mokhtari,K.&Reichard,C.A.(2002).Assessingstudents‘metacognitiveawarenessofreadingstrategies.JournalofEducationalPsychology,94(2),249-259.doi:10.1037/0022-0663.94.2.249
Wigfield,A.,&Guthrie,J.T.(1997).Relationsofchildren’smotivationforreadingtotheamountandbreadthoftheirreading.JournalofEducationalPsychology,89(3),420-432.doi:10.1037/0022-0663.89.3.420
Cantrell,Carter,&Rintamaa(2012)
39SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
41SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
40SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
42SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
43SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
44SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
45SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
46SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
47SRCohortIIEvaluationReport:Kentucky
49SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
TeacherObservationProtocol
TeacherFirstName:
TeacherLastName:_
School:
Date(includingdayoftheweek):_
Time:_
Observer:_
Notes:
48SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
Cantrell,Carter,&Rintamaa(2012)
51SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
50SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
51SRCohortIIEvaluationReport:Kentucky
Cantrell,Carter,&Rintamaa(2012)
52SRCohortIIEvaluationReport:Kentucky