Principal Investigator’s Report

Page 1 of 40

ChicagoArtsPartnershipsinEducation’s

PortfolioDevelopmentProject

PrincipalInvestigator’sReport

By

Dr.LawrenceScripp,PrincipalInvestigator

With

SarahSutherland,ResearchAssociateJoshGilbert,ResearchAssistant

CenterforMusicandtheArtsinEducation,Inc.

SubmittedtotheFederalDepartmentofEducation’sArtsEducationModelDevelopmentandDissemination(AEMDD)Program

May8,2015

CAPE’sPortfolioDevelopmentProject(PDP)PrincipalInvestigator’sReportScripp,CMAIE,Inc.

Page 2 of 40

TableofContents

1.IntroductiontotheCAPEPortfolioDevelopmentProjectPrincipalInvestigator’sReport

3

2.DataAnalysisMethodology:EstablishingtheBasisforMeaningfulControl-TreatmentSchoolArts/ArtsIntegrationLearningandAcademicPerformanceComparisons

5

3.Five-PhaseAnalysisofStudentISATAcademicTestData 7

4.InvestigatingThreeMeasuresofArtsandArtsIntegrationLearning 12

5.TheExaminationofTreatmentSchoolTeacherPDandPerformanceVariablesandTheirLinkstoStudentArtsandAcademicLearningOutcomesDuringtheFinalYearoftheProject

20

6.LinkingtheChainofEvidenceI:DirectPairwiseCorrelationsBetweenTeacherPDandStudentAcademicPerformanceOutcomes

23

7.LinkingtheChainofEvidenceII:DirectPairwiseCorrelationsBetweenTeacherPDandStudentArtsLearningPerformanceOutcomes

26

8.LinkingtheChainofEvidenceIII:DirectPairwiseCorrelationsBetweenStudentArtsLearningandAcademicPerformanceOutcomes

29

9.DeterminingtheStrongestLinks:StepwiseRegressionTestingforMostSignificantTeacher,Student,andFamilyDemographicPredictorsofAcademicAchievement

30

10.Conclusions 38


Page 3 of 40

1. IntroductiontotheCAPEPortfolioDevelopmentProjectPrincipalInvestigator’sReport

TheCAPEPortfolioDevelopment(PDP)Projectwasdesignedasanimportantexperimentintheprofessionaldevelopmentofgrades4-6visualartsandmusicartsspecialistsinChicagoPublicSchools(CPS).WhileithasbeenshowninpreviousAEMDDprojectsthatCPSartsclusterschoolsingeneral—andartsintegrationteachingartistresidenciesinparticular—enhanceacademicperformance(Scripp&Paradis,2014;seepairresults.orgfordetails),thisprojecthypothesizesthatincorporatinghighqualityartsplusartsintegrationportfoliosintoartsspecialistteachingandassessmentpracticeswillfurtheroptimizetheimpactofartslearningonacademicachievement.Thus,theresearchandevaluationquestioninvestigatedinthisprojectis

Towhatextentdidthedevelopmentofartsandartsintegrationclassroomportfoliosystems—guidedbyveteranCAPEteachingartistsinvisualandmusicalarts—enhancebothartslearningandtheimpactofartslearningonacademicperformanceinhighminority,loweconomicstatusschools?

Theinvestigationofthisquestionwillbebasedontheanalysisofthemultiplefactorsthattogetherwillrepresentapossible‘chainofevidence’neededtoidentifycausallinksbetweenhighqualityteacherprofessionaldevelopmentandstudentlearningoutcomes.TherearefourmaindatalinksinthissequentialchainasdepictedintheTable1:

Table1:MultivariateOutcomes“ChainofEvidence”AnalyticFramework

I.ArtsTeacherPreparationPDOutcomeVariables

à

II.ArtsTeacherPerformanceOutcomeVariables

à

III.StudentArtsLearningOutcomeVariables

à

IV.StudentAcademicPerformanceOutcomeVariables

IA.ArtsTeacherPDAttendance

IIA.ArtsTeacherQuantityofStudentPortfolioWork

IIIA.StudentQualityofPortfolioWorkRatings

IVA.StudentFinalYearCombinedAcademicPerformanceTestScore

IB.ArtsTeacherPDReflection/Self-AssessmentSurvey

IIB.ArtsTeacherClassroomObservationRatings

IIB.StudentPortfolioConferencesPerformanceAssessmentRatings

IVB.StudentBaselinetoFinalYearCombinedAcademicPerformanceTestScore

IC.ArtsTeacherSelf-Esteem/ConfidencefromPDExitSurvey

IIC.ArtsTeacherPortfolioConferencePerformanceAssessment

IIIC.StudentPerformanceAssessmentInterviewRatings

ID.ArtsTeacherCombinedPDOutcomeVariable

Fromlefttoright,thesefourcolumnsrepresentacomplexsequenceofinterrelatedfactorsthatmayormaynotultimatelyinfluencestudentacademicachievement.Takenasawhole,thismodelrepresentsthevariouslinksinachainofevidencethatcouldpredict:


Page 4 of 40

• InwhatwaysteacherPDfactors(column1)couldinfluenceteacherperformance

outcomes(column2),studentartslearningoutcomes(column3),and/orstudentacademicoutcomes(column4).

• Inwhatwaysteacherperformanceoutcomes(column2)couldinfluencestudentartslearningoutcomes(column3)and/orstudentacademicoutcomes(column4).

• Inwhatwaysstudentartslearningoutcomes(column3)couldinfluencestudentacademicoutcomes(column4).

Anotherpossibilityisthateachcolumnoffactorsmayonlyaffecttheadjacentcolumn,suggestingachainoffactorsthatonlypredictsqualityinthenextstepoftheteacher-studentlearningsequence:teacherPDfactors(column1)couldinfluenceteacherperformanceoutcomes(column2),teacherperformanceoutcomescouldinfluencestudentartslearningoutcomes(column3),studentartslearningoutcomes(column3)couldinfluencestudentacademiclearningoutcomes(column4).Thenagain,resultsmayalsoprovethatsometeacherPDfactors(column1)predictstudentperformanceinthearts(column3)and/orstudentacademicoutcomes(column4).Findingsfromthisreportwillprovidestatisticallysignificantevidencethat,overthethreeyearsofprojectimplementation,teacherPDoutcomesinfluencedstudentartsandartsintegrationoutcomes,andacademiclearningoutcomessubstantially.First,studentsofartsspecialists—highlyratedfortheirartsplusartsintegrationportfoliopracticesincollaborationwithteachingartistsintreatment1schools—graduallyoutpacedstudentacademicandartslearningoutcomesincontrolschoolsovertime.Whilethemeasureofacademicimprovementwasincrementalfromyeartoyear,theoverallpositivepatternofacademicimprovementisunmistakablebytheendoftheproject.Furthermore,althoughseveralfactorsinfluencedstudentlearning,stepwiseregressiontechniquesrevealedthatPDPteacherparticipationinprofessionaldevelopmentandpositiveassessmentoftheirPDexperiencesinparticularpredictedstudentIllinoisStudentAchievementTest(ISAT)scoreswhencomparingbaselinetofinalyearresults.Bythefinalyearoftheproject,itwasthequalityofartsplusartsintegrationstudentportfolioworkalongwithteacherpositiveattitudesaboutPDPpracticesthataremoredeeplylinkedwithacademicachievementcomparedtothananyotherstudentlearningordemographicfactorotherthan“studentfamilyincome”inthetreatmentschools.Figure13presentedattheendofthisreportdelineatesallsignificantrelationshipsamongthevariableslistedinTable1justdiscussed2.ThefollowingsectionsofthereportdetailthemethodsbywhichconclusionsaboutcausallinksbetweenthevariousteacherandstudentlearningoutcomevariablesweredrawnbothinTable1andinFigure13.

***

1N.B.Theword“treatment”or“control”willnotbecapitalizedexceptwhenreferringtitledasheadingorwhenreferringtoaspecificvariable.2Thereaderisstronglyadvisedtorefertothisfigurethroughoutthereportwhenmultivariateanalysisisbeingdiscussedindetail.


Page 5 of 40

2.DataAnalysisMethodology:EstablishingtheBasisforMeaningfulControl-TreatmentSchoolArts/ArtsIntegrationLearningandAcademicPerformanceComparisonsThefirststepinvalidatingtheanalyticmethodsistounderstandtowhatextentthecontrolandtreatmentschoolgradelevellongitudinalcohortdataareequivalent,proportional,andcanbefairlycompared.Thefollowingdatadisplaysprovideameasureofequivalencybetweenthetreatmentandcontrolschoolstudentpopulationrandomsampledcohorts.Accommodatingdisproportionatestudentcohortpopulationsincontrolandtreatmentlongitudinaldatacomparisons

Bydesign,thenumberofcontrolandtreatmentschoolstudentcohortswasexactlyequivalent.However,bythesecondyearoftheprojectitwasclearthattwocontrolschoolswouldnolongerparticipateintheproject,thusmakingthedatasetdisproportionate.Facedwiththeprospectofasymmetricaldatasets,theinvestigatorsdecidedthat,because(a)thereducedcontrolschoolsamplestillhadsufficientstatisticalpowerfordeterminingitsrelationshiptothevariablessharedbetweenthetwodatasets,and(b)thetreatmentschoolsamplewouldneedtoremainlargeinordertoanalyze“within-group”comparisonswithrespecttodataonlycollectedinthePDPschools,therefore(c)thatanalyzingadisproportionalnumberofstudentsineachcohort—thoughnotideal—wasthebeststrategyfordeterminingfactorsinthetreatmentschooldatathatcouldaccountfordifferencesbetweenthecontrol-treatmentschoolcomparisons.

Figure1:DisproportionalNumberofControlandTreatmentSchoolsStudents

Figure1showsthatthereare78fewercontrolschoolstudentsthantreatmentschoolstudentsinthelongitudinalcohort.

ComparableDemographicFactorsAlthoughCAPEwasnotabletomaintainequalnumbersoftreatmentandcontrolschoolstudentsthroughouttheexperiment,theprofilesoffouroutoffivestudentdemographicfactorsinbothlongitudinalcohortsdisplayedinTable2werefunctionallyequivalent.

PDP Comparison of Research Cohort

0

50

100

150

200

N(R

esea

rch

Coh

ort)

107

185

Control TreatmentResearch Cohort


Page 6 of 40

Table2:Controlvs.TreatmentSchoolDemographicDataPercentages

Control TreatmentGender 54.5%Female

45.5%Male57.9%Female42.1%Male

Free/ReducedLunch(familyincome)

89.8%Free/Reduced10.2%No

92.8%Free/Reduced7.2%No

IEPServices 85.9%No14.1%Yes

86.3%No13.7%Yes

ELLStatus 96.0%No4.0%Yes

95.2%No4.8%Yes

Ethnicity 45.5%Black,Non-Hispanic52.5%Hispanic2.0%Other

58.1%Black,Non-Hispanic40.7%Hispanic1.2%Other

Thefifthdemographicfactor,ethnicity,thoughnotequivalent,revealsthatbothcontrolandtreatmentschoolshavecomparablepercentagesofminoritypopulationstudents,thoughcontrolschoolcohortscontainslightlymoreblackstudentsandtreatmentschoolcohortscontaincomparablymoreHispanicstudents.Becausetherearevirtuallynowhitestudentsineithercohort,thisprojectbringsaparticularfocusontotheeffectofartsandartsintegrationportfoliosonminoritystudentsinChicago.Itwasthejudgmentoftheresearchersthatdespitetheunequalnumberoftotalstudentsineachcohort,theunusuallyhighdegreeofequallydistributeddemographicfactorsbetweenthetwolongitudinalgroupsprovidedthebasisforafaircomparison.AccountingforpriorlevelsofstudentacademicachievementBecauseinitialacademicperformancesignificantlypredictsfutureacademicperformance,thelongitudinalsampleswererandomlyselectedfromthreelevelsofbaselineacademicdatacollectedbeforethePDPprojectbegan.

Sortingthelongitudinalcohortsaccordingtoacademicstatuspriortothebeginningoftheprojectalsoprovidedaprecisemetricfordeterminingadegreeofequivalencybetweenthesamplestudentcohorts.Inordertoachievebalancedrandomlyselectedstudentcohorts,allstudentswereclassifiedasHigh(H),Average(A),orLow(L)academicachieversbeforethebeginningofthePDPprogram.Abalancedtertiledistributionwithinthenormaldistributionplotofthe2010-2011ISATCombinedAverageScoreswasusedtodeterminethecategoricalboundariesforeachofthethreeHALcohorts:

H x≥215 A 195<x<215 L x≤195

Thesecutoffsresultedinanidenticaldistributionofthecombinedtreatmentandcontrolgroupstudentschosenforthestudy:

H 86 A 80 L 86


Page 7 of 40

Byvirtueofthisprocess,thefinalaveragedacademicscoresfortheeachleveloftheHALcohortinbothcontrolandtreatmentschoolsproducedvirtuallyindistinguishable2010-2011ISATCombinedAveragedScores.

Table3:ComparisonofbaselineISATscoresaccordingtoPre-designatedControl-HALcohorts

PreProjectDesignation

Completedataset Control Treatment

High(H) 230.65 230.68 230.62Average(A) 205.67 205.60 205.70Low(L) 176.40 176.63 176.27

SummaryPoint1:ThoughthePDPcontrolandtreatmentstudentcohortswereasymmetricalinnumberduetothewithdrawalofcontrolschoolsfromtheproject,thestudentdemographicfactorswerecommensuratewithregardtogender,familyincome,ethnicity,ELLstatus,IEPservices,andintermsofthedistributionofHigh,Average,andLow(HAL)academicallyratedstudentsrandomlyselectedatthebeginningoftheproject.

***

3.Five-PhaseAnalysisofStudentISATAcademicTestDataFiveanalyticframeworksfocusedonacademicachievementduringthethreeyearsofprojectimplementationdeterminedthatthePDPtreatmentschoolsgraduallyoutperformedthecontrolschoolsandbythefinalyearoftheprojectthispatternofimprovementbecamestatisticallysignificant.3a.Phase1OverallControl-Treatment(C-T)StudentISATTestScoreComparisonsComparingControlandTreatmentschoolcohortacademicperformanceservesasafirststepinmeasuringtheefficacyofPDPproject.AsshowninFigure2below,MathandReadingtestscoresfortheIllinoisStandardAchievementTest(ISAT)revealthat,spanningtheyearsoftheprojectimplementation(baselinetothirdyearofimplementation),bothcontrolandtreatmentschoolstudentcohortsimprovedincrementallyeachyear.Fromtheviewpointofeachannualreport,thetreatmentschoolsscoreswereneversignificantlyhigherthanthecontrolschools.

Figure2:Control-TreatmentSchoolCohortISATTestScoreComparisonsfromBaselinetoFinalYearofthePDPprogram

PDP ISAT Combined Average Scores by Research Cohort

100

150

200

250

300

Y

205.92220.44 227.26

240.96

203.22218.02

228.08242.99


YMean(2010-2011 ISAT COMBINED AVG SS)Mean(2011-2012 ISAT COMBINED AVG SS)Mean(2012-2013 ISAT COMBINED AVG SS)Mean(2013-2014 ISAT COMBINED AVG SS)


Page 8 of 40

3b.Phase2C-TSchool“Gainscore”AnalysisofISATScoresThegainscoreanalysis,however,providesevidenceforthesignificantdifferencebetweenthetwostudentcohorts.Lookingmorecloselyatthepatternoftestscoreresultslongitudinally,Table4revealsthatthetreatmentschoolcohortmeanscorestartsoutbelowthemeanscoreofthecontrolschools,3yetastheprogramproceeded,thetreatmentcohortmeanscoresincrementallymetandthensurpassedthemeanscoresofthecontrolschoolsbythethirdyearofPDP.InTable4weseetheyear-by-yeardatapreviouslydisplayedinbarchartformat,withaddedinformationregardingthegainscoresincolumn5.

Table4:Year-by-YearC-TSchoolMeanScoreDifferencesinStudentISATScores

Control Treatment MeanDifference TreatmentSchoolGainscore

tProb

Baseline2010-2011ISATCombinedAverageMeans

205.92

203.22

-2.7083

—

Prob>|t|=0.4128

2011-2012ISATCombinedAverageMeans

220.44

218.02

-2.4226

+0.2857

Prob>|t|=0.4306


227.26

228.08

0.8224

+3.3998

Prob>|t|=0.7682


240.96

242.99

2.0252

+1.2028

Prob>|t|=0.4909

t=positivetrend;*=significant(pvalue<.05);**=verysignificant(pvalue<.01)

WhilenoneofthecontiguousyearmeanscoresaresignificantlydifferentinTable4,thefifthcolumndataanalyzedinTable5belowshowsthattheaveragedifferenceinthechangeingainscoresbetweentheISATscoresfrom2010-2011(baseline)tothe3rdyearofimplementationin2013-2014isstatisticallysignificant.

Table5:AveragedIndividualStudentGainScoresBetweenBaselineandFinalYearProgramImplementation

Control Treatment MeanDifference

tProb

DifferencebetweenBaseline2010-2011toFinalYear2013-2014ISATCombinedAverageMeansDelta

35.30 39.68 4.376 Prob>|t|=0.0408*


3N.B.AlthoughtheHALcohortswerematchedbydividingthestudentpopulationsscoresintothreeequalparts,theoverallControl-Treatment(C-T)baselineISATscoresforthetwocohortswerenotidentical.


Page 9 of 40

SummaryPoint2:Therearepositive,statisticallysignificantdifferencesinC-TISATtestgainscoresthatindicatePDPTreatmentSchoolsasawholeoutperformedControlSchoolcohortswhencomparingbaselineandfinalyeardata.3c.Phase3ISATC-TSchoolGainscoreComparisonsAccordingtoDemographicFactorsThegradualemergenceofstatisticallysignificantdifferencesbetweenthecontrolandtreatmentschoolsisheightenedfurtherbylookingintothepatternofgainscoresamongthestudentdemographicfactors.TheexaminationofbaselinetofinalyearISATscoresinFigure3showsthattreatmentschoolsoutperformthecontrolschoolsfromtheviewpointofGender,Ethnicity,Free/ReducedLunch(familyincome)andpreviousHAL(academichistory)classification,suggestingthatthegainsintheTreatmentSchoolsapplytovirtuallythewholespectrumofstudents(all“bluebar”treatmentschoolsarehigherthanthe“redbar”controlschoolaveragegainscores).Figure3:ISATBaseline-FinalYearComparisonsbyGender,Ethnicity,FamilyIncome,HAL

LevelsofPriorAcademicAchievement

Itshouldbenoted,however,thatISATgainscoresfortwosmallsampledemographiccohorts—EnglishLanguageLearners(ELLstatus)andIndividualEducationPlan(IEPserved)students—favortheControlSchools.Perhapsbecauselanguageorlearning

PDP Difference of 2010-2011 to 2013-2014 ISATCombined Average Score by Gender by Research Cohort

0

5

10

15

20

25

30

35

40

45

50

0-20

11 to

201

3-20

14 IS

AT

CO

MB

INE

D A

VG

S

33.23

40.0637.79 39.15

Control Treatment Control TreatmentFemale Male

Research Cohort within Gender

Research Cohort Control Treatment

PDP Difference of 2010-2011 to 2013-2014 ISATCombined Average Scores by Ethnicity by Research Cohort

0

10

20

30

40

50

0-20

11 to

201

3-20

14 IS

AT

CO

MB

INE

D A

VG

S

36.8240.22

34.0638.41

36.25

47.25

Control Treatment Control Treatment Control TreatmentBlack, Non-Hispanic Hispanic Other

Research Cohort within Ethnicity


PDP Difference of 2010-2011 to 2013-2014 ISAT CombinedAverage Score by Free/Reduced Lunch by Research Cohort

0

10

20

30

40

50

0-20

11 to

201

3-20

14 IS

AT C

OM

BINE

D AV

G S

34.4339.33

42.78 44.30

Control Treatment Control TreatmentFree/Reduced No

Research Cohort within Free/Reduced Lunch


PDP Difference of 2010-2011 to 2013-2014 ISAT CombinedAverage Scores by HAL Designation by Research Cohort

0

10

20

30

40

50

0-20

11 to

201

3-20

14 IS

AT C

OM

BINE

D AV

G S

26.47

34.0836.24 38.18

44.91 46.31

Control Treatment Control Treatment Control TreatmentH A L

Research Cohort within HAL Designation



Page 10 of 40

challengedstudentsmaynothavehadequalaccesstotheportfolioprocess,itappearsthatasmallnumberoflanguageorlearningchallengedlearnersintreatmentschoolsdonotbenefitfromthePDPprograminthesamewayallotherstudentdemographicpopulationsdo.Yet,becauseofthesmallsamplesizeofthesedemographiccategories(rightcolumnsinbothdatadisplays),conclusivemeasuresofstatisticalsignificancecannotbedetermined.

Figure4:C-TISATComparisonsAccordingtoELLandIEPStudentClassifications

3d.Phase4IndividualSchoolOutlierAnalysis

ThepatternofgainscoresbytheseparatedcontrolandtreatmentschoolsprovidesadditionalevidenceindicatingthatthePDPprojecthasarelativelyuniformpositiveeffectontheTreatmentSchools.Figure5belowrevealsthatfourschoolsshowdistinctlydifferentdegreesofgainscorechangesovertime.Itappearsthatthecontrolschools,labeledby(C),havetwooutlier“lowincreaseschools”—ChaseandJahn—andtheTreatmentschools,labeledby(T),havetwooutlier“highincreaseschools”—HoyneandTalcott.TheindividualschooloutliersindicateunusualimprovementintwooftheTreatmentSchools,andtheunusuallackofimprovementoftwooftheControlSchools.

Figure5:SeparateSchoolControl(C)–Treatment(T)ISATGainScoreComparison

PDP Difference of 2010-2011 to 2013-2014 ISAT CombinedAverage Scores by ELL Status by Research Cohort

0

10

20

30

40

50

60

0-20

11 to

201

3-20

14 IS

AT C

OM

BIN

ED A

VG S

34.0439.57

61.13

41.79

Control Treatment Control TreatmentNo Yes

Research Cohort within ELL Status

n=4 n=8


PDP Difference of 2010-2011 to 2013-2014 ISATCombined Average Score by IEP Status by Research Cohort

0

10

20

30

40

50

0-20

11 to

201

3-20

14 IS

AT C

OM

BIN

ED A

VG S

33.26

39.24

46.7742.24

Control Treatment Control TreatmentNo Yes

Research Cohort within IEP Status

n=14 n=23


PDP Difference of 2010-2011 to 2013-2014 ISAT Combined Average Scores by Schools

0

10

20

30

40

50

60

0-20

11 to

2013

-201

4 ISA

T CO

MBIN

ED A

VG S

29.15

36.83

28.06

38.7142.21

39.50

52.19

39.6535.18 32.80

39.2233.34

36.7533.68

56.33

Chas

e (C)

Haley

(C)

Jahn

(C)

Laviz

zo (C

)

Sabin

(C)

Fort

Dear

born

(T)

Hoyn

e (T)

Kipli

ng (T

)

Lafay

ette (

T)

Lafay

ette/C

hopin

(T)

New

Sulliv

an (T

)

Pere

z (T)

Pirie

(T)

Rave

nswo

od (T

)

Talco

tt (T)

Schools


Page 11 of 40

NoteinFigure6thatthe“lowincrease”controlschoolsstartedwithhigheraveragedISATscoresandfinishedaroundtheotherschools’scores(Chase212to240,Jahn215to244),whilethe“highincrease”treatmentschoolsstartedwithloweraverageISATscoresandincreasedthegreaterdistancetofinishatorabovemosttheotherschools’scores(SeeC-TFigure6:OutlierSchoolISATProfiles(Hoyne194to247;Talcott192to244).4

Figure6:OutlierSchoolISATProfiles

3e.Phase5ISATMeets/Exceeds(MEX)CPSDistrictBenchmarkAnalysis

RevealingpatternsofC-TschooldifferencessimilartotheISATscoreanalysis,Table6demonstratesincrementalincreasesinthepercentageofMEXsofthetreatmentoverthecontrolschoolcohortsbythesecondyearofPDPprojectimplementation.5

Table6:AnnualDifferencesinPercentStudentsWhoMeetorExceed(MEX)

CPSISATBenchmarks

Academicwarning/Below Meets/Exceeds

Baseline2010-2011ISATMEX

C:30/92=32.6%T:50/160=31.2%

C:62/92=67.4%T:110/160=68.8%

2011-2012ISATMEX

C:30/95=31.6%T:49/154=31.8%

C:65/95=68.4%T:105/154=68.2%

2012-2013ISATMEX

C:67/96=69.8%T:99/161=61.5%

C:29/96=30.2%T:62/161=38.5%

2013-2014ISATMEX

C:55/91=60.4%T:84/151=55.6%

C:36/91=39.6%T:67/151=44.4%

4SeeAppendixFigureI.1forthematchedpairsanalysisofthedifferencebetween2010-2011to2013-2014ISATcombinedaveragescoresbyeachlongitudinalcohort.5NotethattheoverallloweringofthepercentageofMEXstudentsbetweenyear1andyear2ofPDPisduetochangesincalibrationoftheMEXbenchmarksbytheCPS.

PDP ISAT Combined Average Scores by Schools (All Years)

0

50

100

150

200

250

300

Y

212225234240

195216219

233215217228

244

196

228215

234209217

237252

199209222

242

194217226

247223

241246259

195209

223231214

227236247

198209222

237213

226239246

203220220

240

199207223232

192213

227244

Chase (C) Haley (C) Jahn (C) Lavizzo (C) Sabin (C) Fort Dearborn (T) Hoyne (T) Kipling (T) Lafayette (T) Lafayette/Chopin (T) New Sullivan (T) Perez (T) Pirie (T) Ravenswood (T) Talcott (T)Schools



0

50

100

150

200

250

300

Y

212225234240

195216219

233215217228

244

196

228215

234209217

237252

199209222

242

194217226

247223

241246259

195209

223231214

227236247

198209222

237213

226239246

203220220

240

199207223232

192213

227244




0

50

100

150

200

250

300

Y

212225234240

195216219

233215217228

244

196

228215

234209217

237252

199209222

242

194217226

247223

241246259

195209

223231214

227236247

198209222

237213

226239246

203220220

240

199207223232

192213

227244




0

50

100

150

200

250

300

Y

212225234240

195216219

233215217228

244

196

228215

234209217

237252

199209222

242

194217226

247223

241246259

195209

223231214

227236247

198209222

237213

226239246

203220220

240

199207223232

192213

227244




Page 12 of 40

AnomalousIEPStudentISATPerformanceinbothPDPControlandTreatmentSchools Figure7revealsthat,althoughIEPservicesinallschoolsdidsignificantlynarrowtheISATperformancegapbetweenIEPandNon-IEPstudentsfromthebaselinetothethirdyearofproject,thesesamestudents,ontheaverage,stillperformatastaggeringrateof26pointsbehindthosestudentswithoutIEPs.6

Figure7:ISATGainscoreDifferencesbetweenIEPandNon-IEPStudentsinTreatmentSchools

BecauseofthestatisticallysignificantdifferenceintheISATmeanscores(seeAppendixA:I.2andI.3),therestofthisreportwillnotincludetheIEPstudentstogiveusamoreaccuratepictureoftheoveralleffectsoftheprogram.

SummaryPoint3:Analysesofstudentdemographicfactors,C-ToutlierschoolISATprofiles,andschooldistrictISATbenchmarkdataprovideadditionalevidenceforthegradualyetsignificanteffectofthePDPprojectonacademicperformance.TheongoinginvestigationoftheeffectofPDPonstudentlearninginthisreportwillbeconductedwithoutincludingdatafromtherelativelysmallnumberofIEPoutlierstudentsintheoverallanalysis.

***

4.InvestigatingThreeMeasuresofArtsandArtsIntegrationLearning

Inthefinalyearoftheproject,theCMAIEresearchersadministeredthreeinstrumentsdesignedtomeasuretheimpactofartsintegrationportfoliodevelopmentonindividualstudentartslearning:

(1) TheArtsPlusArtsIntegrationPerformanceAssessmentInterview(PAI)administeredtobothcontrolandtreatmentschoolcohorts

6SeeAppendixFigures1.2and1.3forstatisticallysignificantdifferencesbetweenIEPandNon-IEPstudents.

PDP Difference of 2010-2011 to 2013-2014ISAT Combined Average Score by IEP Status

0

5

10

15

20

25

30

35

40

45

50

0-20

11 to

201

3-20

14 IS

AT C

OM

BIN

ED A

VG S

37.00

43.97

No YesIEP Status

PDP 2013-2014 ISAT CombinedAverage Score by IEP Status

0

50

100

150

200

250

300

Mean

(201

3-20

14 IS

AT C

OMBI

NED

AVG

SS)

245.93219.59

No YesIEP Status


Page 13 of 40

(2) ThequantitativeandqualitativeassessmentofindividualstudentPortfolioArtifactAnalysis(PAA)worksamplesinthetreatmentschools

(3) TheArtsIntegrationPortfolioConference(AIPC)PerformanceAssessmentProtocoldesignedtoelicitstudentandteacherreflectiveunderstandingofthePDPlearningoutcomesbasedondiscussionandinterpretationofindividualstudentAIPworksamples

Thesetoolsweredesignedtoengagestudentsandteacherstoreflectseparatelyontheirteachingandlearningexperiencesrelatedtoartsintegrationunits.Theprimarypurposeofthesetoolswastoprovideanauthenticassessmentvehicleforstudents’levelofunderstandingofartsandartsintegrationlearninginthecontextofdescribing,discussing,anddemonstratingaspectsoftheirownandtheirpeers’work.ThePAIandtheAIPCprovidedopportunitiesforratingone-on-onediscussionbetweenthestudentandinterviewerthatrevealedconceptualunderstanding,artisticprocess,contentmeaning,personalresponse,aestheticcriticism,andmetacognition.AsecondarypurposeoftheAIPCwastogiveteachersanopportunitytoarticulatetheirviewsonthemissionandgoalsofthePDPprojectandthentoreflectontheirobservationsofstudentperformanceintheAIPCinrelationtotheirpreviousstatements.Thestudentportfolioworkwasevaluatedtodeterminetheapplicationoftheirknowledgeandunderstandinginvariousartistic,musical,andwritingprojectsthroughouttheschoolyear.QuantitativeassessmentofindividualstudentportfolioworkproductsestablishedabaselinemeasureofteachersupportforPDPteachingpractices.Thequalitativeassessmentofstudentportfolioartifactsbroughtforthevidenceofstudentinterpretiveunderstandingofindividual,collaborative,andpeerartsintegrationlearningprocesses,products,andculminatingeventsdocumentedintheirportfolios.ThevalidityoftheanalysisofAIPCandPAIresponseswasensuredbythepresenceofstudentworkchosenbytheteacherandstudentsfortheconferencetorepresenttheirbestexamplesofstudentlearningprocessandproducts.Thereliabilityoftheanalysiswasensuredbyadefinedprotocol(seeAppendixB:1.1)conductedbyanoutsidefacilitator,videodocumentationandwrittentranscriptionofeachentiresession,andanoutsidescoringteamtrainedtorateeachchild’sandteacher’slevelofresponseaccordingtoacommonscoringrubric.ThePDPStudent“LevelofComplexity”ScoringSystemSharedAcrossTheThreeInstrumentsThecomparablestudentratingsystemdeployedbytheCMAIEteamenabledtheresearcherstodeterminecategoricaldifferencesinthe“sophisticationofresponse”acrossdiverseperformancetasksandworkproductsspecifictoeachunitoftheprogram.Basedon“skilltheory”frameworksdevisedbyKurtFischer7,theresponseratingsreflect

7A theory of cognitive development: The control and construction of hierarchies of skills. Fischer, Kurt W. Psychological Review, Vol. 87(6), Nov 1980, 477-531


Page 14 of 40

categoricallydifferentlevelsofcognitivecomplexityintherealmofartisticandintegrativethinking.ThescoringsystemforinterviewtranscriptsinPAI,PAA,andAIPCinstrumentsisbasedonacommon5-Levelqualitativescale:Table7:PDPCommonStudent“LevelofResponse”RatingScaleforThreePDPIndividual

StudentLearningAssessmentInstruments

LevelNR(NoNumericalScore):NoRelevantResponse

Irrelevantorindiscernibleresponse;silenceLevel1:SingleDimensionalResponses

Concrete,un-detailedresponse.Genericstatements,singularperspective.Unspecific,unfocused,diffused.Noelaboration,nodetail,nopersonalspecificsorproceduralrelationships.Listsundifferentiatedelements.Level2:MultipleSingleDimensions

Concreteconnections,someoccasionaldetail,someelaboration,oremergingspecificity.Somecoordinationofelements,likeaclearlyorderedprocedure.Specificpersonalinsight.Level3:CoordinationofDimensions

Detaileddescriptiverelationships.Oftenprovideselaborativedetailedstatements.Evidenceofhigher-orderrelationalthinking,includingelementsofinter-personalinsightandpurpose,artisticaesthetic,and/orhistoricalreferences.Level4:SystemicUnderstanding

Substantialdetailandspecificity.Causalstatements.Compareandcontrastrelationships.Criticalperspective,highlycomplex,multiplerelationships.

CodingdescriptionsandresponseexemplarsdisplayedinAppendixBdemonstratehowthePDPportfolioscoringsystemworksandprovidesanarrayofportfolioworksamplesthatshowhowtheassessmentofartisticqualityandreflectiveunderstandingofstudentworkwasmadepossiblethroughtheportfolioassessmentprocessesdevelopedinPDPprojectclassrooms.Theresultsofthestudentartsandartslearningoutcomesandtheirrelationtostandardizedmeasuresofacademicachievementnowfollow.

***Arts/ArtsIntegrationOutcomesMeasure1:Control-TreatmentSchoolStudentArtsPlusArtsIntegrationPerformanceAssessmentInterview(PAI)AdministeredDuringtheFinalyearoftheproject

TheindividualstudentPerformanceAssessmentInterview(PAI)ratings(seeAppendixB:1.1)revealimportantdifferencesinthelevelsofunderstandingofartsandartslearningprocessesbetweenthecontrolandthePDPtreatmentschools.


Page 15 of 40

DifferencesinmeanscoresdisplayedinFigure9indicatethatthetreatmentschoolstudents’understandingofartsmakingprocessesandartsintegrationlearningpracticesaresignificantlyhigherthanthoseofthecontrolschoolstudents.

Figure9:ComparisonofC-TPAIMeanScoreDifferencesbytheFinalYearoftheProject

Table8:DeterminationofStatisticalSignificantDifferencesofC-TPAIScoreComparisons Control Treatment Mean

DifferencetProb

PAIAverageScore

2.08509 2.25944 0.174351 Prob>|t|=0.0027**


FurtherdemographicanalysesrevealthatthePDPtreatmentstudentsoutperformthecontrolstudentsregardlessofGenderandEthnicity:

Figure10:DifferencesinPAIscoresDistributedEquallyAccordingtoStudentGender

PDP PAI Average Scores by Research Cohort

0

0.5

1

1.5

2

2.5

3

3.5

4

Mean

(PAI

AVG

)

2.092.26


PDP PAI Average Scores byGender by Research Cohort

0

0.5

1

1.5

2

2.5

3

3.5

4

Mea

n(PA

I AVG

)

2.122.30

2.02 2.20

Control Treatment Control TreatmentFemale Male

Research Cohort within Gender



Page 16 of 40

Figure11:DifferencesinPAIscoresDistributedEquallyAccordingtoStudentEthnicity

AccordingtoanalysisofPAIresultsbasedonwhetherornotstudentsreceivedFree/ReducedLunchprovisions(Figure12),itappearsthatthePDPTreatmentschools’low-incomestudentsshowedagreaterunderstandingofartsandartsintegrationprocessesthandidthetreatmentschoolhigherincomestudents.Thisfindingwasthereversedinthecontrolschools,wherelow-incomestudentstrailedboththetreatmentlow-incomestudentsandthecontrolhigherincomestudents.

Figure12:ComparisonofC-TPAIScoresbyFamilyIncome

AswiththeISATscores,thepoolofELLstudentsisnotlargeenoughtomakeconclusiveinferencesregardingtheimpactofPDP.

PDP PAI Average Scores by Ethnicity by Research Cohort

0

0.5

1

1.5

2

2.5

3

3.5

4

Mea

n(PA

I AVG

)2.04

2.24 2.17 2.261.88

Control Treatment Control Treatment Control TreatmentBlack, Non-Hispanic Hispanic Other

Research Cohort within Ethnicity


PDP PAI Average Scores by Free/Reduced Lunch by Research Cohort

0

0.5

1

1.5

2

2.5

3

3.5

4

Mea

n(PA

I AVG

)

2.082.28 2.14 2.01

Control Treatment Control TreatmentFree/Reduced No

Research Cohort within Free/Reduced Lunch



Page 17 of 40

C-TPerformanceAssessmentInterview(PAI)DistributionofScoresAccordingtoPDPExpectationsbytheFinalYearofthePDPProject

BenchmarksforstandardsofPAIscoresinthefinalyearofPDPweredeterminedbytertileclassificationwithinthenormaldistributionofstudentperformanceratings.Meets/Exceeds(MEX)profileanalysisofthePAIdatainTable8revealsthattreatmentschoolstudentswerefarmorelikelytoexceedthebenchmarkstandardsofartslearningandfarlessliketoratebelowthesebenchmarkswhencomparedtothecontrols.

Table8:C-TDifferencesinBenchmarksforPAIRatings

BenchmarkCategories

Below Meets Exceeds

Control n=8/2236.4%

n=11/2250.0%

n=3/2213.6%

Treatment n=4/3710.8%

n=18/3748.7%

n=15/3740.5%

SummaryPoint4:AnalysisoftheIndividualstudentPerformanceAssessmentInterview(PAI)revealedpositiveevidencefortheeffectofPDPonthetreatmentschoolstudents,therebysuggestingpreliminaryevidenceforpossiblecausallinksbetweenstudentunderstandingofartworksandart-makingprocesses,thePDPteacherprofessionaldevelopmentprogramingeneral,andincreasedISATscoresreportedearlier.

***

Arts/ArtsintegrationOutcomesMeasure2:TreatmentSchoolArtsPortfolioArtifactsAssessment(PAA)DuringFinalProjectYear

Thisvariablewascreatedtoassessthequantityandqualityofstudentportfoliowork.PortfolioworksamplescollectedinthefinalyearofthePDPprojectwereanalyzedfor(a)“numberofartifacts”asanindicatorofteacherlevelofsupportforthePDPprojectand(b)“qualityofstudent”workproductsratedaccordingtotherubricspresentedinAppendixC:1.2.Table9specifiestherelativedistributionofstudent’sabilitytosuccessfullymaintainanarts/artsintegrationportfoliosystem,aprimaryobjectiveofthePDPteacherprofessionaldevelopmentprogram.Fifty-sevenoutoffifty-nineoftheteachers’studentsmetorexceededexpectationsforasuccessfulPDPstudentportfoliosystem,astatisticthatindicatesthatallteacher’smetorexceededtheirresponsibilitytocreateaportfoliosystemforvirtuallyallstudentsinthetreatmentschools.

Table9:StudentPortfolio“QuantityofArtifacts”DistributionofRatingsaccordingtoLevel

ofPDPExpectationsbytheFinalYearoftheProject

Below≤20 20<Meets>40 40≥ExceedsQuantityofStudentPortfolioArtifacts

n=2/593.4%

n=23/5939.0%

n=34/5957.6%


Page 18 of 40

Table10specifiestherelativedistributionofstudents’abilitytoproducehighqualityartsplusartsintegrationworkproducts,anotherprimarygoalofthePDPteacherprofessionaldevelopmentprogram.ThespectrumofstudentworkratingsrevealsthatwhilevirtuallyallPDPteachershadprovidedtheopportunityforstudentstocreateportfolioworkandmoststudents(69.5%)metorexceededPDPexpectations,manystudents(30.5%)haddifficultycreatingdetailedormultidimensionalartisticwork.

Table10:TreatmentSchoolStudentPortfolio“QualityofArtifacts”Distributionof

AveragedRatingAccordingtoLevelofPDPExpectationsbytheFinalYearoftheProject

Below<=2.0(general,diffuse,singledimensional)

2.0<Meets>2.3(multiplesingle

dimensions,somedetail)

2.3>=Exceeds(towardinter-relational

perspectiveshighlydetailed)QualityofStudentPortfolioArtifacts

n=18/5930.5%

n=30/5950.9%

n=11/5918.6%

Arts/ArtsintegrationOutcomesMeasure3:TreatmentSchoolArtsIntegrationStudentPortfolioConference(AIPC)AssessmentResults

Thisvariablewascreatedtoratethequalityofindividualstudentperformanceduringtheirparticipationinfacilitatedportfolioconferenceprotocol(AppendixB:1.2).PerformanceratingswerebasedonthequalityofdescriptionanddialoguewiththefacilitatorandpeersbasedonexamplesofstudentportfolioworkdiscussedthroughouttheAIPCprotocol.StudentswereratedforqualityofresponseindicatorsaccordingtothesamerubricusedtoscorethePAIresponses(AppendixC:1.1).

Table11specifiesthedistributionoftreatmentschoolstudents’leveledabilitytoreflectonthequalityofartsplusartsintegrationworkproducts,anotherprimaryobjectiveofthePDPteacherprofessionaldevelopmentprogram.Thespectrumoftreatmentschoolstudentportfolioconferenceresponseratingsrevealsthat,contrarytothequalityratingsofthestand-aloneportfolioworksamplesintheprevioustable,alargemajorityofstudents(82.6%)metorexceededexpectationsforcriticalthinkingandreflectiveunderstandingofmeaningfulartsandartsintegrationlearningprocesses,basedontheinterpretationoftheirownandtheirpeerportfolioworkproducts.

Table11:TreatmentSchoolStudentPortfolioConferencePerformanceAssessmentDistributionofAveragedRatingsAccordingtoPDPExpectations

bytheFinalYearoftheProject

Below<=2.0(general,diffuse,singledimensional)

2.0<Meets>2.3(multiplesingledimensions,

somedetail)

2.3>=Exceeds(towardinter-relational

perspectiveshighlydetailed)

StudentPortfolioWorkAveragedRatings

n=12/6917.4%

n=26/6937.7%

n=31/6944.9%


Page 19 of 40

SummaryPoint5:AnalysisoftheindividualstudentArtsPlusArtsIntegrationPortfolio“NumberofArtifacts”inthetreatmentschoolsprovidedpositiveevidenceofPDPteacherprofessionaldevelopmentoutcomesbythefinalyearoftheproject.Theprofileofthe“QualityofArtifacts”ratingsintheportfoliosandstudent“LevelofResponse”ratingsdistilledfromtheirPortfolioConferenceperformanceassessmenttasksprovidedevidenceoftheimpactofportfoliopracticesontreatmentstudentsbythefinalyearoftheproject.

***PairwiseInter-correlationsBetweenAllThreeStudentLearningOutcomeVariables

Researchersemployedmultivariate“patternsanddegreeofcorrelation”analysistechniquestotestforthedegreeofassociationamongalltreatmentschoolstudentlearningvariables.ThedatasummarizedinTable12suggestthatastatisticallysignificantdegreeofassociationexistsbetween:

(a) “QualityofStudentPortfolioArtifacts”and“QualityofStudentResponseinPortfolioConference”AverageScores[positivetrend]

(b) “QuantityofStudentPortfolioArtifacts”andthe“QualityofStudentPortfolioArtifacts”AverageScores[weak,yetstatisticallysignificantcorrelation]

(c) “StudentPortfolioConference”andthe“StudentPerformanceAssessmentInterview”AverageResponseScores[strong,statisticallysignificantcorrelation]

Table12:PairwiseCorrelationsAmongTreatmentSchoolStudentArtsLearningVariables

byArtsTeacherType

Variable1 Variable2 CompleteSpearmanr

CompleteProb>|p|

MusicSpearmanr

MusicProb>|p|

VisualArtsSpearmanr

VisualArtsProb>|p|

QuantityofStudentPortfolioArtifacts

QualityofStudentPortfolioArtifacts

0.2649

0.0426*

N.S.

N.S.

N.S.

N.S.

QuantityofStudentPortfolioArtifacts

StudentPCResponseAverageScore

N.S.

N.S.

N.S.

N.S.

N.S.

N.S.

StudentPortfolioNumberofArtifacts

PAIAverageScore

N.S.

N.S.

N.S.

N.S.

N.S.

N.S.

QualityofStudentPortfolioArtifacts

StudentPCAverageScore

0.2510

0.1045

N.S.

N.S.

0.4226

0.0634t

StudentPortfolioQualityofArtifacts

StudentPAIAverageScore

N.S.

N.S.

N.S.

N.S.

N.S.

N.S.

StudentPCAverageScore

PAIAverageScore

0.4744

0.0053**

N.S.

N.S.

0.8246

0.0010*

N.S.=nonsignificant;t=positivetrend;*=significant(pvalue<.05);**=verysignificant(pvalue<.01)


Page 20 of 40

SummaryPoint6:AnalysisofInter-correlationsamongthePDPStudentLearningOutcomesintreatmentschoolssuggestthatquantityislinkedtoqualityofportfolioproducts,qualityofportfolioartifactspredictsqualityofreflectioninportfolioconferences,andindividualandinteractivegroupinterviewreflectivecommentsarestronglylinkedtogetherbysimilarlevelsofunderstandingofartsandartsintegrationlearningprocesses.Incidentally,thedivisionofteachertypesrevealsthatvisualartsinstructorswerefarmoreeffectivethantheirmusicteachercounterpartsinbringingabouttheseassociationsbywayoftheirparticipationinPDP.Lateroninstatisticalanalysis,however,itwasdeterminedthatthesesmallsampleteachertypedifferencesdidnotsignificantlyinfluencefundamentalimpactofPDPonstudentlearningoutcomes.

***

5.TheExaminationofTreatmentSchoolTeacherPDandPerformanceVariablesandTheirLinkstoStudentArtsandAcademicLearningOutcomesDuringtheFinalYearoftheProjectInordertomaptheentirechainofevidencefortheimpactofPDPonthemeasuresofstudentlearningdescribedearlier,teacherdatawascollected,validated,andreliablyquantifiedbytheresearchteam.Thesedataareorganizedintotwocategories:(a)teacheroutcomevariableslinkedtotheirparticipationinprofessionaldevelopmenteventsand(b)datacollectedandcodedasaresultofteacherperformanceassessmenttasksandprotocols.DescriptionandNumberingofSevenTeacherOutcomeVariablesI:FourArtsTeacherProfessionalDevelopmentOutcomeVariables

1A:ArtsTeacherAttendanceData.Basedonthenumberofexitsurveysfilledout,thesedatarepresentabasicmeasureofteacherengagementinPDPprofessionaldevelopmenteventsthroughoutthethreeyearsoftheprojectadministration.AttendanceDatarevealthatgenerallythat8of10artsspecialistteachersattendedmorethan50%ofPDevents,while2of10teachersattendedlessthan50%ofthePDsoffered.

1B:ArtsTeacherSelf-AssessmentPre-PostSurveyData.Basedonaveragedpre-postagreementresponses(never–sometimes–mostofthetime–allofthetime)toquestionsaboutsupportforartsintegrationlearningpracticesintheclassroom,theaveragedresultsfromallsurveyquestionsconsistentlyrevealsignificantdifferencesinteacherresponsestothesurveyquestionsabout:

• thedepthofengagementwithPDPpracticesintheclassroominteractions,• themaintenanceofbothstudentandteacherportfoliosystems,• theconnectionsofPDPworktobotharts&academicwork,and• thefocusonprovidingopportunitiesforstudentreflectionandself

assessments.


Page 21 of 40

Consistentwiththegoalofthetreatmentschoolclassrooms,therearenolowperformingteacheroutliersinthetreatmentschools(3/9teachersintoptertile;6/9teachersinmiddletertile;0/9teachersinbottomtertile).

IC:ArtsTeacherProfessionalDevelopmentEventExitSurveyAverageScore(AppendixD:1.1).Thesedataarebasedonself-esteemandconfidenceratingsbasedonaveragedself-reportratings(1–2–3–4)thatmeasurethedegreeofteacherunderstandingandconfidencewithPDPartsintegrationlearninggoals,contentandprocessstandards,teachingstrategies,andtheirapplicationtoclassroompracticesthroughoutthethreeyearsofprojectadministration.Thoughtherearesomedifferencesamongtheteachers,theoverallrangeofPDPsurveyresponsesregisteratauniformlyhighlevel.Thatis,10/10overallteacherssurveyresponsesaveragedinthetopquartileofpossiblescoresbytheendoftheproject.

ID:TheArtsTeacherCombinedPDOutcomeVariableisacompositeteacheroutcomevariablecreatedbyaveragingcomparablelevelrubricratingsfromthethreepreviousteacherPDoutcomemeasures,theArtsTeacherObservationAveragedScore,TeacherQuantityofStudentPortfolioWork,andtheArtsTeacherPortfolioConferencePerformanceAssessmentAverageScore,whichfollow.

II.ThreeTeacherPerformanceAssessmentOutcomes

IIA:ArtsTeacherQuantityofStudentWork.Asdiscussedpreviously,thisteacherfactorismeasuredinthefinalyearofPDPbythenumberofportfolioartifactscollectedfromeachtreatmentschoolstudentandisaveragedbyeachclassroomtorepresenteachartsteacher’scommitmenttocreateandsustainanindividualarts/artsintegrationstudentportfoliosystemaccordingtothegoalsandPDpracticesmodeledinthisproject.AlthoughtheaveragednumberofportfolioartifactsishighintermsofthePDstandardsbythefinalyearoftheproject(8of9artsteachersmeetorexceedexpectationsofthePDPproject),thedistributionofaveragedstudentnumberofartifactsnonethelessisusedtorankordertheteachersintermsoftheirstudents’abilitytogenerateportfolioworkproducts.IIB:ArtsTeacherObservationAveragedScore.Thismeasurewasbasedonexpertratingsofteacher-studentengagementandreflectionduringPDPclassroomactivitiesasdescribedintheTeacherObservationProtocol(AppendixD:1.3).Averagedratings(1-2-3-4)encompassinteractivefactors(withanequalfocusonteacherandstudentbehaviors)suchasexchangeofquestions,curiosities,bigideas,explicitattentiontolearningtransfer,discussionofchoices,creativeprocessesandstudent-centricartisticbehaviorssuchasactiveexperimentation,imaginativeideas,multiplemodesofexpression,improvisation,“whatif”questions,reflectionongoals,self-assessment,respectforothers,andcollaboration.UnlikepreviousteacherPDoutcomemeasures,theaveragedteacherobservationratingsscoreddirectlyafteraclassroomvisitrevealedthatmostPDPartsteachersfailedtodisplayidealbehaviorsduringtheirclassroomobservations:0of10teacherobservationexceededthegoalsoftheproject,1of10teachersmetPDP


Page 22 of 40

standardsofobservedclassroombehavior,and9of10teachersperformedatbelowstandardlevelofclassroompracticesassumedtorepresentidealclassroomcultureforartsintegrationintheartsclassroom.IIC.ArtsTeacherPortfolioConferencePerformanceAssessmentAverageScore.ThestructureoftheteacherportionoftheAIPCprotocol(AppendixB:1.2)showsthattheartsteacherswerechallengedtodescribethegoalsandpracticesofPDPandtointerpret&assesstheirstudents’previousvoiceddiscussionoftheirworkintheearlierpartoftheportfolioconference.ThetranscriptsoftheseconversationswerecodedandsuccessfullyscoredbytheCMAIEresearchteamonlyinthefinalyearoftheproject.TranscribedteacherresponsesduringtheAIPCwerescoredforlevelsofrelevance,detailandperspectiveusingthescoringrubricsystempresentedinAppendixC:1.1.Becausethereflectionprocesswasbasedonhighqualityportfolioworkchosenbytheteacherandthestudentsandthattheindexofinter-raterreliabilitywashigh—over97%oftheratingswerewithintheacceptablerangeofagreementandallproblematicexampleswerescoredtwiceandaveragedbetweentwoindependentscorers—theresearcherswereconfidentthatartsteacherrankorderedaveragedratingsrepresentavalidandreliablemeasureofteacherreflectiveunderstandingofthecontributionofartsintegrationportfoliostostudentlearninginthePDPclassroom.TherangeofArtsteacherAIPCratingswerenormallydistributedthroughoutthespectrumofteacherlevels(5of9teachersintoptertileand4of9teachersinmiddletertile).

PairwiseInter-correlationsAmongAllTeacherPDandPerformanceAssessmentOutcomeVariablesSimilartotheinter-correlationalanalysisofstudentperformancevariables,PDPresearchersemployedmultivariate“patternsanddegreeofcorrelation”analysistechniquestotestforthedegreeofassociationamongalltreatmentschoolteacherprofessionallearningvariables.Inthiscase,however,resultssuggestthatoutof21permutationsofteacherPDandperformanceoutcomes,virtuallynostatisticallysignificantdegreeofassociationexistsexceptinthecaseoftwopairedvariables:

(d) IA:TeacherPDAttendanceandIIB:TeacherObservationRatings(e) IB:TeacherSelf-AssessmentPre-PostSurveyandIC:TeacherSelf

Esteem/ConfidencePDEventExitSurveyInTable13thefirstpairsuggeststhatattendanceinPDeventsdidpredictteacherobservationratingsmoderatelywell,particularlyinthecaseofthemusicteachers.Thesecondpairsuggeststhattheteacherself-assessmentandattitudesurveysarelinkedintermsofcontentarea.ThethirdpairisacalculationofhowtheimpactofteachersuccessingeneratingproductiveandrichportfoliosandhowthatenhancestheirabilitytoarticulateanddemonstratethegoalsandimpactofthePDPprogramintheirclassrooms.Thelackofcorrelationamongtheremainingpermutationsofpairedvariablessuggestthesecombinationsofvariableswereeitherrelativelyindependentofeachother(asindicatedbynegativeorrandomcorrelations)orwere,bydesign,alreadycorrelatedsignificantlywiththecompositeteacherratingsvariable.


Page 23 of 40

Table13:ThreeSignificantCorrelationsOutOf22TreatmentSchool

ArtsTeacherPDandPerformanceVariables

Variable1 Variable2 AllArtsSpearmanp

AllArtsProb>|p|

MusicSpearmanp

MusicProb>|p|

VisualArtsSpearmanp

VisualArtsProb>|p|

IA:TeacherPDAttendance

TeacherObservationRatings

0.3402 <0.0001** 0.4178 0.0002** -0.6783 <0.0001**

IC:TeacherSelf-AssessmentRatings

TeacherSelf-Esteem/ConfidenceRatings

0.2989 0.0002** N.S. N.S. 0.5472 <0.0001**

IIA:TeacherQuantityofStudentPortfolioWork

TeacherPCPerformanceAssessmentRatings

0.2880 0.0314* N.S. N.S. 0.5282 0.0080**

N.S. = not statistically significant; * = significant (p value <.05); ** = very significant (p value < .01)

SummaryPoint7:Only3of22pairedteacherPD/performanceassessmentvariableswerestronglyandpositivelyinter-correlated:thatis,(a)strongteacherPDattendanceappearstopredicthighqualityteacherobservationratings(andviceversa),(b)highlevelsofself-esteemorconfidenceimplementingPDPteachingpracticescorrespondstohighlevelselfassessmentratings,and(c)ahighlevelofclassroomstudentportfolioworkproductivity(i.e.,numberofstudentworkartifacts)predictstoacertainextenttheteachers’levelofsophisticationofresponseduringthePDPportfolioconferenceprotocol.Thefirstcasesuggestsacausallinkbetweenteachertrainingandhighqualityarts/artsintegrationteachingpractices;thesecondcasesuggeststhatsignificantoverlapexistsbetweenthetwoseparateteachersurveyinstrumentssuchthatastrongself-perceptionofsuccesswiththeprogramistiedcloselywithhighlevelsofconfidenceinincorporatingtheprogramintoteacherclassrooms;thethirdcasesuggeststhattheartsteacher’sabilitytodocumentalargeamountofstudentworkartifactsintheportfoliospredictshigherlevelsofarticulationabouttheprogram’sgoalsandtheimpactoftheprogramonthequalityofstudentarts/artsintegrationwork.Overall,thelackofcorrelationamongalargemajorityoftheteachervariablessuggeststhattheteacherdatacollectioninstrumentsrepresentedrelativelyindependentmeasuresofteachereffectivenessinPDP.

***6.LinkingtheChainofEvidenceI:DirectPairwiseCorrelationsBetweenTeacherPDandStudentAcademicPerformanceOutcomesOnceallteacherandstudentvariableshavebeendescribedandvalidatedinisolationofoneanotherortheirinterdependencywithoneanother,thenextstepinthe“chainofevidence”evaluationistosearchandtestfor“patternsanddegreeofcorrelation”between:(a)thesevencategoriesofteacherPDandperformanceoutcomevariables,(b)thethree


Page 24 of 40

categoriesofstudentartslearningvariables,and(c)thetwomeasuresofstudentperformanceonstandardizedacademictests.86A.CorrelationBetweenTeacherVariablesandStudentAcademicOutcomes:Baseline–FinalYearISATTestGainScores

Table14revealsthatonlytwooutofseventeacherfactorssignificantlyrelatestoISATCombined(readingandmath)AverageGainScoresfromthebaselinetothefinalyearofimplementation.Thatis,onlyIA:TeacherAttendanceinPDEventsand(2)IIA:TeacherQuantityofStudentPortfolioWork(i.e.,numberofportfolioworkartifacts)correlatedsignificantlywithstudentacademicperformance.Thedirectinfluenceofteacherattendanceonlong-termacademicgains—particularlyinthecaseofthevisualartsteachers—providesanessentialevidentiarylinkbetweenPDParts/artsintegrationteachertrainingandthestudentlearninggainsthatdifferentiatedthetreatmentfromthecontrolschoolsbythefinalyearoftheprojectreportedinsection4(Arts/ArtsIntegrationOutcomesMeasure1).Theteacherabilitytoproduceahigherquantityofdocumentedstudentlearningartifactsintheirstudentportfoliosthatcorrespondedtoincreasesinstudenttestscoresappearedtobemorelikelythecaseinmusicclassroomsthanwithvisualarts.

Table14:Correlationof7TeacherVariableswithStudentISATGainScores

fromBaselineToFinalYearofthePDP

TeacherVariableCorrelationwithBaselinetoFinalYear(2011-13)FinalYearISATCombinedAverageGainScores

CompleteSpearmanr

CompleteProb>|p|

MusicSpearmanr

MusicProb>|p|

VisualArtsSpearmanr

VisualArtsProb>|p|

4TeacherPDOutcomeVariablesIA:ArtsTeacherPDAttendance

0.1871 0.0391* N.S. N.S. 0.4364 0.0008**

IB:ArtsTeacherPre-PostSurveySelf-Assessment

N.S. N.S. N.S. N.S. N.S. N.S.

IC:ArtsTeacherExitSurvey(Self-Esteem/ConfidencewithPDP)


1D:CombinedArtsTeacherPDOutcomeVariable


3TeacherPerformanceAssessmentOutcomeVariablesIIA:ArtsTeacherQuantityofStudentPortfolioWork(#artifacts)

0.3223 0.0175* 0.5265 0.0020** N.S. N.S.

IIBArtsTeacherClassroomObservationRating


IIC:ArtsTeacherPortfolioConferencePerformanceAssessment


N.S. = not statistically significant; t = positive trend; * = significant (p value <.05); ** = very significant (p value < .01)

8N.B.SeeAppendixFforcompletesummarychartofallteacherandstudentoutcomevariables.SeeFinalFigure13Correlation-RegressionMultivariateMapforaflowchartrepresentationofallprincipalinter-relatedvariables.


Page 25 of 40

***

6B.CorrelationBetweenTeacherVariablesandStudentAcademicOutcomes:FinalYearISATTestScoresTheassociationofteachervariableswithacademicachievementintheculminatingyearoftheprojectwasinvestigatedtodeterminewhichfactorsinfluenceacademiclearningintheculminatingyearoftheartsteachers’portfoliopractices.Table15indicatesthatastrongsignificanceexistsbetweentheISATCombinedAveragescoreandtheTeacherSelf-EsteemandConfidencewithartsintegrationpractices(distilledfromtheIC:TeacherExitSurveyresults),particularlyinthecaseofmusicteacherswhohadlesspreviousfamiliaritywithportfoliodocumentationpracticesthandidthevisualartsteachers.WeakbutstatisticallysignificantnegativecorrelationsbetweenstudentISATscoresandTeacherperformanceratingsintheAIPCPortfolioConferenceperformanceassessmentsandTeacherObservationratingsduringthefinalyearofPDPsuggestteacherunderstandingofportfolioconferencestudentperformancewasnotyetsufficientlyaddressedintheteacherPDprogram.

Table15:Strong,SignificantCorrelationsExistBetweenOneArtsTeacherOutcome

VariablesandFinalYearStudentISATTestScores

TeacherVariableCorrelationwith2012-13FinalYearISATCombinedAverageScore

CompleteSpearmanr

CompleteProb>|p|

MusicSpearmanr

MusicProb>|p|

VisualArtsSpearmanr

VisualArtsProb>|p|

4TeacherPDOutcomeVariablesIA:ArtsTeacherPDAttendanceData

N.S.

N.S.

-0.4347

0.0002**

N.S.

N.S.


N.S.

N.S.

N.S.

N.S.

N.S.

N.S.


0.2472

0.0046**

0.3994

0.0008**

N.S.

N.S.


N.S.

N.S.

N.S.

N.S.

N.S.

N.S.


N.S.

N.S.

N.S.

N.S.

N.S.

N.S.


-0.1739

0.0496*

N.S.

N.S.

N.S.

N.S.


-0.1938

0.0284*

-0.2419

0.0522 t

N.S.

N.S.


SummaryPoint8:TeacherartsintegrationprofessionallearningoutcomesinPDP,investigatedfortheirinfluenceonstudentacademiclearning,havedeterminedthatteacherparticipation,positiveselfassessment,andresponsetoportfolioconferenceprotocolsfocusedontheimpactofPDPonstudentworkarehighlyassociatedwithtreatmentschoolacademicgains.TheseresultsprovideevidencethatexplainswhyPDPstudentsimprovedatagreaterratethandidthecontrolschoolsaspresentedinsection3ofthisreport.


Page 26 of 40

***

7.LinkingtheChainofEvidenceII:DirectPairwiseCorrelationsBetweenTeacherPDandStudentArtsLearningPerformanceOutcomesBecauseevidenceexiststhatteacherPDoutcomesarelinkedwithstudentacademicsuccess,possiblecorrelationsbetweenPDoutcomesandartslearningoutcomescanbeexploredtodeterminewhetherachainofevidencecanbedrawnthroughthestudentartslearningvariablesinwaysthatmaybelinkedtoeitherorbothteacherprofessionallearningandstudentacademictestperformancedata.7A.PatternandDegreeofCorrelationBetweenTeacherVariablesandFinalYearStudentArts/ArtsIntegrationPerformanceAssessmentInterview(PAI)Resultsfromcorrelationanalysissuggestthatthereisnodirectevidenceofstatisticalcorrelationbetweenanycombinationofteacherandstudentartslearningoutcomes.SummaryPoint9:ThelackofanysignificantcorrelationbetweenanyoftheteacherPDorperformanceassessmentdataandfinalyearstudentPAIresultssuggestthattheartsteachers’responsestothePDPprofessionaldevelopmentprogramandtotheirabilitytodevelopmentproductivearts/artsintegrationportfoliosystemsintheirclassroomweremorelikelytobolsteracademicratherthanartsstudentlearningoutcomes.ItappearsthatitisthePAIperformanceratings—andnotteacherPDorperformanceassessmentvariables—thatcorrespondtostudentperformanceinthevariousformsofPDPprojectartslearning,suchasstudentportfolioconferenceorPAIperformanceassessmentratings.

***7B.PatternandDegreeofCorrelationbetweenTeacherVariablesandFinalYearStudentQualityofPortfolioWorkArtifactsTable16revealsevidencethatbothvariablesIIA:TeacherQuantityofPortfolioArtifactsandIIC:TeacherPortfolioConferencePerformanceAssessmentRatingscorrelatesignificantlyandpositivelywithIIIA:QualityofStudentPortfolioWorkbythefinalyearoftheproject.Thepredictivepowerofthesetwovariablesdidnotsurprisetheresearchersbecause:(a)ahighernumberofstudentartifactsistheresultofhighlevelsofteachersupportfortheportfoliopracticesintheclassroomand(b)thehighqualityofstudentartifactsshouldbelinkedwithhigherratingsofteacherreflectiononstudentachievementgoalsinPDPasdemonstratedbythehigherlevelofsophisticationoftheirportfolioconferenceinterviewratings

Converselytherearealsotwoinstancesofsignificant,yetnegativecorrelationsbetweenIIA:StudentQualityofPortfolioWorkRatingsandboth(a)IA:TeacherPDAttendanceDataand(b)IIC:TeacherPortfolioConferenceresults.Theserathersurprisingpairedcorrelationresultsmaybeduetothefocusandtimingofthedatacollection.Thatis,bythefinalyearoftheproject,teacherattendanceinPDeventsmayberegardedasmore


Page 27 of 40

supplementalthancentraltothequalityofstudentwork,andthattheobservedclassroomteachingpracticeswerelessgermanetoevidenceofartslearningthanwastheleveloftheteachers’abilitytopromoteandunderstandtheimplicationsofhighqualitystudentportfoliowork.

Table16:PatternandDegreeofSignificantCorrelationsbetween7TeacherVariablesandStudentPortfolioQualityofPortfolioWorkArtifactsbytheFinalYearoftheProject

TeacherVariableCorrelationswithStudentQualityofPortfolioWork

CompleteSpearmanr

CompleteProb>|p|

MusicSpearmanr

MusicProb>|p|

VisualArtsSpearmanr

VisualArtsProb>|p|


-0.3605 0.0054** N.S. N.S. -0.4663 0.0108*








0.2649 0.0426* N.S. N.S. N.S. N.S.


-0.3639 0.0058** -0.3293 0.0657 t N.S. N.S.


0.2880 0.0314* N.S. N.S. 0.5282 0.0080**


SummaryPoint10:Table16providesevidencethatbythefinalyearoftheproject,thequalityofstudentportfolioworkdependslessoncontinuedexposuretoteacherPDtrainingorobservablechangesinclassroomteachingpractices,butratherreliesmoreonthesuccessoftheartsteacheringeneratinghighqualitystudentworkthat,inturn,canbelinkedtotheincreasinglysophisticatedmetacognitiveperspectiveonstudentlearningrevealedintheteacherportfolioconferenceratings.

***

7C.PatternandDegreeofCorrelationbetweenTeacherVariablesandFinalYearStudentPortfolioConferencePerformanceAssessmentResponseRatings.

Table17indicatesthatadifferentkindofteacherPDoutcomemeasure,IC:TeacherExitSurveyRatings,aselfreportattitudevariablethatfocusedonissuesofteacherselfesteemandconfidenceasitpertainedtothetheirabilitytosupportarts/artsintegrationportfoliopracticesintheirclassroom,positivelycorrelateswithIIIB.StudentPortfolioConferenceResponseRatings.Thisteacherattitudevariableappearsfarmoreimportanttothemusic


Page 28 of 40

teacherswho,incomparisontovisualartsteachers,werefarlesslikelytohaveinitiatedorsustainedsystematicdocumentationandassessmentofstudentworkintheirclassroomspriortotheproject.NegativecorrelationsuggeststhatahighdegreeofIA:TeacherAttendancedidnotcorrespondtopositivestudentperformanceduringtheportfolioconferenceperformancetaskswithPDPpractices.Thisresultsuggeststhatbytheendoftheproject,teachereffectivenessmaydependmoreonconfidencedevelopedthroughpersonalexperiencewithPDParts/artsintegrationpracticesthanonattendingmorePDsessions.

Table17:Strong,SignificantCorrelationsbetween7TeacherVariables

andStudentPortfolioConferenceResponseRatingsbytheFinalYearoftheProject

TeacherVariableCorrelationswithStudentPortfolioConferenceResponseRatings

CompleteSpearmanp

CompleteProb>|p|

MusicSpearmanr

MusicProb>|p|

VisualArtsSpearmanp

VisualArtsProb>|p|


-0.2412 0.0510t -0.3739 0.0321* -0.3073 0.0820t




0.3132 0.0104* 0.4607 0.0070** N.S. N.S.








N.S. N.S. -0.3698 0.0373* 0.4813 0.0046**


SummaryPoint11:Relativelyfewmeasuresofartsteachers’professionaldevelopmentorperformanceoutcomeswerelinkedpositivelytostudentacademicorartslearningoutcomes.Nonetheless,thepatternsanddegreeofcorrelationovertimerevealedthatspecificteachervariablesmatteredgreatlywithregardto(a)artslearningforitsownsakeand(b)artsintegrationforthesakeofPDP’seventualimpactonISATtestscores.Thus,IIA.:TeacherQuantityofStudentPortfolioWorkandIIC:TeacherPortfolioConferencePerformanceAssessmentRatingsstronglyinfluencesIIIA:QualityofStudentArts/ArtsIntegrationPortfolioWork,whileIC:TeacherExitSurveyresultsstronglylinkedtostudents’understandingofhighqualityartslearningandthepossibleimpactofartsintegratedlearningonacademiclearningasdemonstratedbyIIIB:StudentPortfolioConferencePerformanceAssessmentRatings.


Page 29 of 40

Thenextsectionofthereportfocusesonthecorrelationofstudentartslearningoutcomeswithacademicoutcomes.

***

8.LinkingtheChainofEvidenceIII:DirectPairwiseCorrelationsBetweenStudentArtsLearningandAcademicPerformanceOutcomesTheprevioussectionsexploredthecorrelationallinksthatexistedbetweenthesevenartsteacherPDandperformanceassessmentfactorsandthetwostudentstandardizedtestresultsandthethreestudentartslearningoutcomes.ThecorrelationsandconnectionsbetweenartslearningoutcomesandISATstandardizeacademicperformanceoutcomesareinvestigatedbelow.

***

8A.PatternsandDegreeofCorrelationBetweenStudentQualityofPortfolioWorkArtifactsandISATAcademicAchievementTestScoresTable18showsthatastrongandhighlysophisticatedcorrelationexistsbetweenIIIA:StudentQualityofPortfolioWorkRatingsandtheIVB:StudentISATFinalYearCombinedAverageScores.Table18:CorrelationofStudentQualityofPortfolioWorkandAcademicPerformance

IIIB:StudentQualityofPortfolioWorkRatingsandtheirCorrelationwithISATAcademicAchievementTests

CompleteSpearmanr

CompleteProb>|p|

IVB:StudentBaseline-to-Final-YearISATCombinedAcademicPerformanceAverageGainScores(2010-2014)

N.S. N.S.

IVA:StudentFinalYearISATB=CombinedAcademicPerformanceISATCombinedAverageScores(2013-2014)

0.4489 0.0005**

N.S. = not significant t = positive trend; * = significant (p value <.05); ** = very significant (p value < .01)

SummaryPoint12:AverystrongdegreeofcorrelationexistsbetweenIIIA:StudentQualityofPortfolioWorkRatingsandacademicachievement,suggestingthatthesuccessfulimplementationofPDPinmusicorvisualartsclassroomsoptimizestheeffectofartslearningwithinportfoliosonacademicachievementThereasonthiseffectwasnotobtainedinthebaseline-to-final-yearacademicgainscoresisprobablyduetoseveralfactors:(a)qualitativeassessmentofportfolioworkwasnotconducteduntilthefinalyearoftheproject,(b)theacademicgainsforthePDPtreatmentschoolswasnotachievedsignificantlyuntilthethirdyearoftheproject,and(c)theimpactofartsintegrationskillsinthecontextofartslearningclassroomswerenotshowntohaveanyconnectiontoprioracademicperformance.

***


Page 30 of 40

8B.PatternsandDegreeofCorrelationBetweenStudentPortfolioConference(PC)PerformanceAssessmentRatingsandISATAcademicAchievementTestScores

CorrelationanalysisdeterminedthatnosignificantassociationsexistbetweentheIIIB:StudentPCPerformanceAssessmentRatingsandIVA:StudentFinalYearISATTestScoresortheIVB.StudentISATBaselinetoFinalYearGainScores.

Summarypoint13:Unfortunately,researcherswerenotabletodeterminethedegreeandpatternofthiscorrelationintheearlieryearsoftheprojectbecauseofunreliableimplementationoftheStudentPCprotocolthatledtouncorrectablescoringandcodingproblems.

8C.PatternsandDegreeofCorrelationBetweenStudentPerformanceInterview(PAI)ResponseRatingsandISATAcademicAchievementTestScoresStatisticalanalysisrevealedthatstudentIIIC:PAIRatingsdonotcorrelatesignificantlywitheithertheIVA:ISATFinalYearISATCombinedAcademicPerformanceTestScoresortheIVB:ISATBaseline-to-Final-YearTestGainScores.SummaryPoint14:AlthoughtheIIIC:PAIRatingsarestatisticallyisolatedfromtherestoftheotherPDPprogramfactors,thesedataaresignificantlylinkedwiththeIIIB:StudentPortfolioConferencePerformanceAssessmentRatings.Inaddition,thelinkbetweentheStudentPAIandPortfolioConferencePerformanceAssessmentRatingsvalidatesbothinstrumentsasmeasuresofarts/artsintegrationteachingandlearninginartsintegrationlearningenvironments.

***9.DeterminingtheStrongestLinks:StepwiseRegressionTestingforMostSignificantTeacher,StudentandFamilyDemographicPredictorsofAcademicAchievement

CMAIEresearchersemployedmultivariate“patternsanddegreeofcorrelation”analysistechniquestotestforthedegreeofassociationamongalltreatmentschoolteacherprofessionallearningvariables.Thetworegressionmodelsinvestigatedinthisreportfocusonsortingoutwhichofthe7teacherPDandperformanceoutcomevariables,3studentarts/artsintegrationlearningoutcomevariables,and5studentfamilydemographicvariablesbestfittheshapeandtrajectoryofthe(a)IVA:StudentISATFinalYearCombinedAcademicTestScoreDataandtheIVB:StudentISATBaseline-to-FinalYearCombinedAcademicTestGainScores.TheteacherPD,studentarts/artsintegrationand/ordemographicvariablesthatbestfittheacademicoutcomedatafitthusbecometheprincipalpredictorsofacademicachievement.


Page 31 of 40

9A.PDPStepwiseFitforIVB.Baseline-to-FinalYearStudentISAT(2011—2014)CombinedAcademicTestGainScoresStepwiseRegressionmethodsfocusedonIVB:StudentISATBaseline-to-FinalYearCombinedAcademicPerformanceGainScores(2011-2014)resultedinidentifyingthemostprominentpredictorsofacademicachievementinthetreatmentschools.Thefactor-by-factorstepwisefitforthedifferenceinISATscoresbetweenthe2010-2011andthe2013-2014academicyearsinTable19revealsthattheIA:TeacherAttendanceinPDworkshopsandtheconcomitantincreasedabilityofteacherstoproduceagreaterIIA:QuantityofStudentPortfolioWorkArtifactsoverthespanoftheprojectarebyfarthetwostrongestandstatisticallysignificantfactorsthatexplainthedifferenceinlevelsofISATachievementfromthebaselinetofinalyearoftheproject9.Table19:StepwiseRegressionModelingFitforIVB.StudentISATBaseline-to-FinalYear

ISATCombinedAcademicAchievementAverageScores

MajorPredictorsofAcademicAchievementGainScores

Fratio(Effectsize)

Prob>F R2(Degreeofexplainedvariancepervariable)

CumulativeR2(wholemodelexplainedvariance)

IIA.TeacherQuantityofStudentPortfolioWorkArtifacts(determinedbythenumberofportfolioworksamplescollectedinthefinalyearoftheproject)

30.558 0.000004** 0.3499 0.3499

TeacherPDAttendance(determinedbythesubmissionofexitsurveys)

14.251 0.00151** 0.2965 0.6464


***

9B.RegressionFitforFinalYear2013-2014ISATCombinedAverageScores

ThestepwiseregressionfitfortheIVA:FinalYear(2013-2014)StudentISATCombinedAcademicTestScoresresultedinidentifyingtheoneprominentandthreerelativelyancillarypredictorsofacademicachievementinthetreatmentschoolsduringtheculminatingyearofPDPprojectimplementation.InTable20,thestepwisefitfortheIVA:FinalYearISATCombinedAverageScoresrevealsthattheIIIA:StudentQualityofPortfolioWorkArtifactsisthemostsignificantfactorinpredictingacademicachievement.Threeotherfactors:(a)IA:TeacherPDAttendance,(b)IB:TeacherPre-PostSelf-AssessmentSurveyRatingsand(c)Free/ReducedLunchBenefits

9SeeAppendixE1.1forregressioneffectestimatesandcompletestephistorydetails.


Page 32 of 40

(FamilyIncome)DemographicClassificationDataallinfluenceacademicachievementsignificantly,thoughwithfarlessexplanatorypowerassummarizedinthetablebelow.

Table20:StepwiseRegressionFactorFittoStudentISATFinalYearCombinedAcademicTestScores

MajorPredictorsofPDPFinalYearAcademicAchievementTestScores

Fratio(Effectsize)

Prob>F R2(Degreeofexplainedvariancepervariable)

CumulativeR2(wholemodelexplainedvariance)

IIIA:StudentQualityofPortfolioWorkRatings(ScoredbytheCMAIEResearchTeam)

22.182 0.00041** 0.3067 0.3067

Free/ReducedLunchBenefitsStudentDemographicClassificationData(FamilyIncome)

7.167 0.019* 0.1230 0.4297

IB:TeacherPre-PostSurveySelf-AssessmentRatings(ArtsteacherabilitytoimplementPDPintheclassroom)

10.829 0.00585** 0.0872

0.5169

1A:TeacherPDAttendance(determinedbythesubmissionofexitsurveys)

9.824 0.00791** 0.0739 0.5908


SummaryPoint15:StepwiseregressionanalysesfocusedonmultiplePDPteacherandstudentoutcomevariableshavesucceededindeterminingtheprincipalfactorsleadingtoacademicachievement.Fourstrategiesformeasuringacademicprogresswereinvestigated:(a)longitudinalviewofpatternsofacademicachievementnowshowthatteachercommittedparticipationinhighqualityPDprogramsandtheconsequentproliferationofportfolioworkismosthighlyassociatedwithdifferencesinacademicgainscorecomparisonsandfinalyearresultsbetweenmatchedpaircontrolandtreatmentschoolacademicschools,(b)thepatternanddegreeofpairwisecorrelationbetweenteacherPDfactorsandstudentartslearningoutcomes,(c)thepatternanddegreeofpairwisecorrelationbetweenartslearningfactorsandacademictestscores,and(d)thecombinationofallteacherPDoutcomes,studentarts/artsintegrationlearningoutcomesandstudent/familydemographictraitsweremeasuredinthecontextofoneanotherthroughregressionanalysistodeterminestatisticallyboththesignificanceandthedegreeofinfluenceonacademicachievement.Insum,“baselinetofinalyear”academicprogresswasmostclearlylinkedwithlong-termparticipationinTeacherPDtrainingsessionsandthequantityofstudentworkproduced;“finalyearresults”weremostclearlylinkedwiththequalityofstudentportfoliowork,positiveratingsonteacherselfassessmentsurveys,students’familyincomestatus,andcontinuedengagementinPDservices.Inthefinalsectionofthereport,allthecorrelationandregressionlinksaremappedtogethertoexpressthecomplexityandflowofsuccessfulartsintegrationprogramdevelopmentinurbanpublicelementaryschoollearningenvironments.

***


Page 33 of 40

9C.TheFullChainofEvidence:ASummaryCorrelation-RegressionFlowChartMapDepictingtheRelationshipsBetweentheTeacher&StudentVariablesandtheirHierarchicallyOrdered“PatternandDegreeofAssociation”withStudentAcademicLongitudinalGainScoresandFinalYearTestResults.MultivariateanalysishasprovidedausefulthoughsomewhatcircuitousroutetowardexplainingthedevelopmentofportfoliopracticesinChicagoPublicSchoolartsclassroomsanditsimpactonbothartsandacademiclearning.The“chainofevidence”approachtakesintoconsiderationasequenceof7teacherand5studentoutcomevariablesthathave,uptothispoint,establishedthebasisforartsintegrationPDtrainingandprogramdevelopmentcriteriaaimedatincreasingbothartsandacademiclearning.Thelongitudinalcohortacademicoutcomeshavedevelopedovertimetothepointthatresearcherscanmakecontrol-treatmentschoolcomparisons,canconstructaflowchartoffactorsthatshowhowteacherPDresponseoutcomesleadtoteachingoutcomesandhowteacherperformanceoutcomesleadtonewformsofstudentlearning,andcanshowthatallofthefactorscontributetoacademicachievement.Table21isasummaryofallsignificantandpositivecorrelationsandregressionfactorsthataccountforthesuccessoftheacademiccontrastwithcontrolschools,therisinglevelofsophisticationofstudentportfoliowork,andthereflectivethinkinginbothteacherPDsessionsandstudentportfolioperformanceassessmentprotocolsthatprovideindicationsofthegrowingofartsintegrationteachingandlearningpracticesbygrade6inthefinalyearoftheproject.Table21:SummaryofAllCorrelationandRegressionFactorsastheBasisfortheFinalPDP

Correlation-RegressionMultivariateAnalysisTable

FourTeacherPDOutcomeVariables

IA:ArtsTeacherPDAttendanceData

CorrelationwithIVB:StudentBaselinetoFinalYearCombinedAcademicPerformanceAverageTestScore(Prob>|p|=0.0391)

SignificantregressionfactorofIVA:StudentFinalYearCombinedAcademicPerformanceAverageTestScore(Prob>F=0.00791)(r2=.0739)

SignificantregressionfactorofIVB:StudentBaselinetoFinalYearCombinedAcademicPerformanceAverageTestScore(Prob>F=0.00151)(r2=.2965)


N.S.



CorrelationwithIIIB:StudentPortfolioConferencePerformanceAssessmentRatings(Prob>|p|=0.0104)

StrongcorrelationwithIVA:StudentFinalYearCombinedAcademicPerformanceAverageTestScore(Prob>|p|=0.0046)

N.S.


StrongCorrelationwithIIA:ArtsTeacherQuantityofStudentPortfolioWork(#ofartifacts)(Prob>|p|=<0.0001)

N.S.


Page 34 of 40

ThreeTeacherPerformanceAssessmentOutcomeVariables

IIA:ArtsTeacherQuantityofStudentPortfolioWork(#ofartifacts)

CorrelationwithIIIA:StudentQualityofPortfolioWorkRatings(Prob>|p|=0.0426)

CorrelationwithIVB:StudentBaselinetoFinalYearCombinedAcademicPerformanceAverageTestScore(Prob>|p|=0.0175)

Strongcorrelationwith1D:CombinedArtsTeacherPDOutcomeVariable(Prob>|p|=<0.0001)

SignificantregressionfactorofIVB:StudentBaselinetoFinalYearCombinedAcademicPerformanceAverageTestScore(Prob>F=0.0000369)(r2=.3499)

IIB:ArtsTeacherClassroomObservationRatings

N.S.

N.S.


CorrelationwithIIIA:StudentQualityofPortfolioWorkRatings(Prob>|p|=0.0314)

N.S.

ThreeStudentArtsLearningAssessmentOutcomeVariables

\

IIIA:StudentQualityofPortfolioWorkRatings

CorrelationwithIIA:ArtsTeacherQuantityofStudentPortfolioWork(#ofartifacts)(Prob>|p|=0.0426)

CorrelationwithIIC:ArtsTeacherPortfolioConferencePerformanceAssessment(Prob>|p|=0.0314)

StrongcorrelationwithIVA:StudentFinalYearCombinedAcademicPerformanceAverageTestScore(Prob>|p|=0.0005)


IIIB:StudentPortfolioConferencePerformanceAssessmentRatings

CorrelationwithIC:ArtsTeacherExitSurvey(Self-Esteem/ConfidencewithPDP)(Prob>|p|=0.0104)

StrongcorrelationwithIIIC:StudentPerformanceAssessmentInterviewRatings(Prob>|p|=0.0053)

N.S.

IIIC:StudentPerformanceAssessmentInterviewRatings

StrongcorrelationwithIIIB:StudentPortfolioConferencePerformanceAssessmentRatings(Prob>|p|=0.0053)

N.S.

OneStudentDemographicFactor

StudentDemographicFactor:Free/ReducedLunch(Gender,HALclassification,Ethnicity)

N.S.

RegressionfactorofIVA:StudentFinalYearCombinedAcademicPerformanceAverageTestScore(Prob>F=0.019)(r2=.1230)

(continuedonnextpage)


Page 35 of 40

TwoStudentAcademicAssessmentOutcomeVariables

IVA:StudentFinalYearCombinedAcademicPerformanceAverageTestScore

StrongcorrelationwithIC:ArtsTeacherExitSurvey(Self-Esteem/ConfidencewithPDP)(Prob>|p|=0.0046)StrongcorrelationwithIIIA:StudentQualityofPortfolioWorkRatings(Prob>|p|=0.0005)

Significantregressionfactors:

IA:ArtsTeacherPDAttendanceData;(Prob>F=0.00791)(r2=.0739)

IB:ArtsTeacherPre-PostSurveySelf-AssessmentofPDPpractices(Prob>F=0.00585)(r2=.0872)

IIIA:StudentQualityofPortfolioWorkRatings(Prob>F=0.00041)(r2=.3067)

SignificantRegressionStudentDemographicFamilyIncomeFactor:Free/ReducedLunch(Prob>F=0.019)(r2=.1230)

IVB:StudentBaselinetoFinalYearCombinedAcademicPerformanceAverageTestScore

CorrelationwithIA:ArtsTeacherPDAttendanceData(Prob>|p|=0.0391)

CorrelationwithIIA:ArtsTeacherQuantityofStudentPortfolioWork(#ofartifacts)(Prob>|p|=0.0175)

Significantregressionfactors:

IA:ArtsTeacherPDAttendanceData(Prob>F=0.00151)(r2=.2965)

IIA:ArtsTeacherQuantityofStudentPortfolioWork(Prob>F=0.0000369)(r2=.3499)


***CorrelationandRegressionAnalysisFindingsinthecontextofthePDP“ChainofEvidence”FlowChartUsingdatainTable21asthefoundation,Figure13(mentionedpreviouslyinsection1ofthisreport)summarizesthecausallinksinthechainofevidencethatflowsfromasequenceofevidencefrom:

I.TeacherPDOutcomesto

II.TeacherPerformanceAssessmentto

III.StudentArts/ArtsIntegrationLearningto

IV.AcademicTestsGainScoresandFinalYearresults.Significantpairedcorrelations(thindottedlines):

• Delineatetheassociationof“teacherPDattendance”and“quantityofstudentportfoliowork”onthelong-termacademicgainscores,afindingthatshowsthatsupportforteacherdevelopmentofartsintegrationhadadirectinfluenceontheteachers’abilitytopromoteanexpansivedocumentationofstudentworkthathadasubstantialeffectonacademicperformance

• Tracetheinfluenceofteacher“quantityofstudentportfoliowork”and“portfolioconferenceperformanceassessmentratings”onstudent“qualityofstudentportfoliowork,”afindingthatsubstantiatesthatnotonlywasthePDPprogramfullydevelopedintotheartslearningclassroomsafterthreeyears,butthatthequalityofteacherreflectiveunderstandingofthegoalsandimpactoftheprogramwastiedtolevelsofqualitystudentwork.


Page 36 of 40

• Demonstratetheimportanceofsurveydatathatsuggesthowhighlevelsofteacher“self-confidence”withPDPartsintegrationportfoliopracticescanlikelyleadtostudents’abilitytoexpressincreasinglysophisticatedlevelsofreflectiveunderstandingoftheirownworkanditsconnectionstobothartsandartsintegrationteachingandlearningasindicatedby“studentportfolioconferenceperformanceassessmentratings”.

Highlysignificantpairedcorrelations(thinsolidlines)extendthelineofevidencebyshowingthat:

• “Qualityofstudentportfoliowork”connectedpreviouslywithteacheroutcomesisalsosignificantlylinkedto“finalyearISATacademicachievement”levels,afindingthatsuggeststhat,asPDParts/artsintegrationpracticesmaturedinthemusicandvisualartsclassrooms,sodiditsinfluenceonacademicachievement.ThusPDPnotonlyoptimizedacademicandartslearningcomparedtothematchedcontrolschoollongitudinalcohortsasdescribedinthefirstpartofthisreport,butalsobecametheintermediarycausalchainoffactorsthatproceededfromteacheroutcomestohighqualitystudentartslearningoutcomesthatinturnpredictedlevelofacademicachievement.

• The“teacherself-confidence”PDoutcomeratingspositivelyassociatedwith“studentdemonstrationandreflection”portfolioconferenceratings(describedpreviouslyinsection9),alsorelatestronglyto“finalyearacademictestscores”.Thischainofevidencesuggeststhatteachers’confidentattitudesabouttheirowncompetencewitharts/artsintegrationportfoliopracticesislinkedsubstantiallytostudentacademicperformance.

• ApositiveprofileofaveragedteacherPDandperformanceassessmentoutcomescorrespondswithahighdegreeofcertaintytoahighamountofstudentportfoliowork.ThisfindingisanotherindicationofthesuccessofthePDPprofessionaldevelopmentprogramtakingrootinartslearningclassroomsinwaysthatsupportincreasesinacademicachievementovertime.

• Thereisacloserelationshipbetweenstudent“performanceassessmentinterview”and“portfolioconference”ratings,indicatingthatthesetwomeasuresprovidevalidatedalternativeassessmentsofstudent’sunderstandingofarts/artsintegrationlearningprocessesandtheirpossibleimpactonacademicperformance.

StepwiseregressiontechniqueswereusedinPDPanalysistodeterminewhichvariablesemergedasleadingpredictorsofstudentacademicachievementincomparisontoothercompetingfactors,includingstudentdemographicdata.Significantandhighlysignificantregressionfactors(thinandthicksolidarrows)inFigure13indicatethat:

• Long-termbaseline-to-finalyeartestacademicscoreresultsarepredictedprimarilyby“teacherparticipationinthePDPprofessionallearningevents”andthe“abilityofteacherstogetstudentstogeneratesubstantialamountofstudentportfoliowork.”AsindicatedinTable21,noothervariablescomeclosetothatlevelofinfluence.

• Theculminatingyearacademicresultsarepredictedprimarilyby“studentqualityofportfoliowork”andtoalesserextentbyteacherPDattendance,teacherselfreportsregardingthesuccessoftheirclassroomarts/artsintegrationportfoliopractices.Therelativeimportanceofstudentfamilyincomefactorsremindsusofthedifficultyofanyeducationinterventiontotranscendtheinfluenceofpoverty.

Documents

Principal Investigator’s Report