DevelopingNGSS-AlignedAssessmentstoMeasureCrosscuttingConceptsinStudent

ReasoningofEarthStructuresandSystems

By

GaryWeiser

Submittedinpartialfulfillmentofthe

requirementsforthedegreeofDoctorofPhilosophy

undertheExecutiveCommitteeoftheGraduateSchoolofArtsandSciences

COLUMBIAUNIVERSITY

2019

©2019

GaryWeiser

Allrightsreserved

Abstract

DevelopingNGSS-AlignedAssessmentstoMeasureCrosscuttingConceptsinStudent

ReasoningofEarthStructuresandSystems

GaryWeiser

Thepasttwodecadesofresearchonhowstudentsdeveloptheirscienceunderstandingsas

they make sense of phenomena that occur in the natural world has culminated in a

movementtoredefinescienceeducationalstandards.Theso-calledNextGenerationScience

Standards(orNGSS)codifythisnewdefinitionintoasetofdistinctperformanceexpectations,

whichoutlinehowstudentsmightrevealtowhatextenttheyhavesufficientunderstanding

ofdisciplinarycoreideas(DCIs),sciencepractices(SEPs),andcrosscuttingconcepts(CCCs).

Thelatterofthesethreedimensionsisuniquebothinbeingthemostrecenttothefieldand

inbeingtheleastsupportedbypriorscienceeducationresearch.Morecrucially,asapolicy

document,theNGSSalonedoesnotprovidethesupportsteachersneedtobringreformsto

theirclassrooms,particularlynotsummativeassessments.Thisdissertationaddressesboth

ofthesegapsusingacombinationofquantitativeandqualitativetechniques.First,Ianalyze

differentialcategorizationofproblemsthatrequirerespondentstoengagewiththeirCCC

understandings via confirmatory factor analysis inference. Second, I use a set of Rasch

modelstomeasurepreliminarylearningprogressionsforCCCsevidentinstudentactivity

withinacomputer-assistedassessmentexperience.Third,Ianalyzestudentartifacts,think-

aloud interviews, and post-task reflective interviews via activity theory to adapt the

progression into a task model in which students explain and predict aspects of Earth

systems.Theculminationof these threeendeavorsnotonlysets forthamethodology for

researchingCCCsinawaythatismoreintegrativetotheotherdimensionsoftheNGSS,but

alsoprovidesaframeworkfordevelopingassessmentsthatarealignedtothegoalsofthese

newstandards.

i

TABLEOFCONTENTS

TABLEOFFIGURES,GRAPHS,ANDTABLES iii

ACKNOWLEDGEMENTS iv

CHAPTER1 1

INTRODUCTION 1RESEARCHQUESTIONS 5

CHAPTER2 7

LITERATUREREVIEW 7DIMENSIONSOFTHENGSS 7ANCHORINGLEARNINGANDASSESSMENTCONTEXTS 14ELEMENTSOFRIGORINTHEDEVELOPMENTOFASSESSMENTS 19DESIGNINGTHEDAT-CROSSASSESSMENTSUITE 23DEVELOPINGCOGNITIVELABPROTOCOLS 28FRAMEWORKFORANALYSIS 31

CHAPTER3 37

METHODOLOGY 37DATACOLLECTION 38DATAANALYSISMETHODOLOGY 41ETHICSANDREFLEXIVITY 46

CHAPTER4 47

RESULTS 47SUMMARYSTATISTICS 47RESEARCHQUESTION1 48RESEARCHQUESTION2 51RESEARCHQUESTION3 55

CHAPTER5 62

FINDINGSANDDISCUSSION 62RESEARCHQUESTION1 62RESEARCHQUESTION2 64

ii

RESEARCHQUESTION3 71MAKINGAMENDMENTSTOTHEECOSYSTEMTASKSTORYBOARD 73IMPLICATIONSFORTHEDESIGNOFTASKSINNEWSYSTEMCONTEXTS 74LIMITATIONSTOMYFINDINGS 75

CHAPTER6 77

CONCLUSION 77

REFERENCES 79

APPENDICES 93

APPENDIXA:TASKMODELFORASSESSMENTOFECOLOGICALSYSTEMS,STRUCTURES,ANDFUNCTIONS–ADAPTEDFROMMS-LS2-3,MS-LS2-4,&MS-LS2-5 93APPENDIXB:HYPOTHESIZEDLEARNINGPROGRESSIONS 105APPENDIXC:ECOSYSTEMTASKITEMSWITHDESIGNRATIONALES 114APPENDIXD:POST-TASKINTERVIEWPROTOCOL 128APPENDIXE:DAT-CROSS:BACKGROUNDQUESTIONNAIRE 130APPENDIXF:DATACOLLECTIONMATRIX 132APPENDIXG:ETICCODINGSCHEMEFORINTERVIEWDATA 133APPENDIXH:RUBRICFORASSESSINGCONSTRUCTEDRESPONSES 134APPENDIXI:RSCRIPTFORSTRUCTURALEQUATIONANDRASCHMODEL 146APPENDIXJ:DEFINITIONSOFSOMEKEYTERMS 148

iii

TableofFigures,Graphs,andTables

Figure2.1FundamentalsoftheECDValidityArgument.AdaptedfromMislevyetal.(2017)........................................................................................................................................................................................21 Figure2.2DiagramofEcologicalSystemsTaskModelDevelopment............................................27 Figure2.3Question2ofDAT-CROSSwithItemDesignRationale,below....................................29 Figure2.4ExampleofaWrightMap.AdaptedfromGotwals&Songer(2013).........................33 Figure2.5ExampleofEngestrom’s(2000)ActivityTheoryModelinStudentWork.............34 Figure3.1Question5ofDAT-CROSSEcosystemStoryline.................................................................40 Figure3.2ExampleofActivityTheoryModelatPlayinClassroomWork..................................44 Figure4.1HypothesizedFactorStructureofDAT-CROSSEcosystemAssessment..................50 Figure4.2ResultingCorrelationsfromCFAAnalysis............................................................................51 Figure4.3Person-Item(Wright)MapofStructureFunctionItems...............................................52 Figure4.4Person-Item(Wright)MapofSystemsandSystemModelItems...............................54 Figure4.5DiagramofActivityDisruptionWhenStudentInteractswithModelingTool.......58 Figure4.6DiagramofActivityDisruptionDuringStudentInteractionwithSimulation.......59 Figure4.7DiagramofActivityDisruptionDuringStudentConstructedResponses................61 Figure5.1DAT-CROSSQuestion5..................................................................................................................65 Figure5.2Usability32'sresponsetoDAT-CROSSQuestion5..........................................................66 Figure5.3:Usability41'sresponsetoDAT-CROSSQuestion5..........................................................67 Figure5.4Usability7'sresponsetoDAT-CROSSQuestion5.............................................................68 Figure5.5DAT-CROSSQuestion10..............................................................................................................69 Figure5.6ExampleQuestionfromWaterUseSystemStoryline......................................................75

Table3.1Inter-raterReliabilityRatingsforConstructedResponses............................................41 Table4.1AveragePerformanceonAssessmentItemsbySubgroup.............................................47 Table4.2FitStatisticsofStructureandFunctionItems.....................................................................53 Table4.3FitStatisticsofSystemsandSystemModelItems..............................................................55

iv

Acknowledgements

MyexceedingthanksgoouttoProf.EmdinandProf.Andersonfortheircontinuingexpertiseandguidancealongmyjourneytothedissertation.AdditionalthanksareduetoProf.RivetfortheinitialpushthatsetmyworkinmotionandtoProf.Mensahforseeingitattheend.ThisworkwouldnothavebeenpossiblewithoutthementorshipandfriendshipofLeiLiuandtherestoftheDAT-CROSSteam.Iextendtothemsincereappreciation.Lastly,Iwanttothankmyfamilyandfriendswhosecontinueddedicationmadethisworkpossible.The project described henceforth was partially supported by the U.S. Department ofEducation. Institute of Education Sciences, National Center for Education Research,throughgrantnumberR305A170456.ThecontentissolelytheresponsibilityoftheauthoranddoesnotnecessarilyrepresenttheofficialviewsoftheInstituteofEducationSciences.

1

Chapter1

INTRODUCTION

TheNextGenerationScienceStandards(NGSS)takeanovelapproachtothewaysthe

science education community thinks about what counts as acceptable expressions of

students’ science understandings (Krajcik & Merritt, 2012). Previous reform documents

suchastheAmericanAssociationfortheAdvancementofScience’sBenchmarksforScience

Literacy(henceforthAAAS;AAAS,1993)describedexpectationsintermsofwhatstudents

shouldknowratherthanwhatstudentscando(Duschl,Schweingruber,&Shouse,2007).The

NGSS reforms, however, describe expectations for student understanding in terms of a

performance,somethingstudentsneedtodoasawayofexpressingcompetenciesinuseof

relevant disciplinary core ideas, science and engineering practices, and crosscutting

concepts(NationalResearchCouncil,2012).Thoughthesestandardsarenot, themselves,

assessments,theyplayanessentialroleininformingassessmentdesign(DeBarger,Penuel,

Harris,&Kennedy,2015)byhelpingtoconstrainpossibleperspectivesontheassessment

triangle: Cognition, Observation, and Interpretation (Pellegrino, Chudowsky, & Glaser,

2001).Thethree-dimensionalviewofscienceunderstanding(Duschletal.,2007)whichfuse

disciplinary core ideas (DCIs), crosscutting concepts (CCCs), and science-engineering

practices(SEPs)detailedintheNationalResearchCouncil’s(henceforthreferredtoasthe

NRC) A Framework for K-12 Understanding (NRC, 2012) (henceforth referred to as

Framework) sets expectations for how the community should think aboutwhat students

know (the cognition corner of the assessment triangle model; Pellegrino et al., 2001).

Similarly,theexpectations,themselves,makesalientwhatstudentunderstandingofscience

should look like (the observation corner of the triangle model; Pellegrino et al., 2001).

2

However, where assessment observations should be found, and how to interpret

observationsmade(Pellegrinoetal.,2001),remainunderspecifiedinreformliterature.

Uniquely underspecified in existing literature are supports for eliciting and

interpreting observations of the crosscutting concepts (CCCs), which has left some

questioningwhythedimensionappearsatall(Osborne,Rafanelli,&Kind,2018).Whilethe

subtlety of CCCs is undeniable, they have immense power in shaping how students use

languagetoexplainphenomena(Weiser,Lyu,&Rojas-Perilla,2017)and,byextension,how

assessorsvaluetheresponsesstudentsprovide.Nonetheless,ifmyphilosophicalpositionis

that student reasoning of the crosscutting concepts adds value to their science

understandings in a way that the other dimensions do not, then failing to capture that

reasoning is failing to capture the full richness of their performances. It is, therefore, a

continuingchargeplacedontheNGSS-supportingscienceeducationcommunitytodesign

assessmentsthataligntoallpartsofthenewstandards.Inthefollowingsections,Idescribe

aframeworkthattheevaluationcommunitymightusetodesignassessmentsfortheNGSS,

fillinginthegapsontheassessmenttrianglealongtheway.

Thisresearchstudyto followsaparadigmknownasdesign-basedresearch,which

emergedfromthelearningsciencesduringthelate1990s(Barab&Squire,2004).Unlike

clinicalinterventionresearch,design-basedresearchmakesatradeoff;i.e.,sacrificingtypical

treatmentcontrolsinfavorofdatathatcanonlybecollectedinthebustleoftherealworld,

butnolessinasystematicandevidence-basedway(Hoadley,2004).DiSessaetal.(2004)

describethegoalsoftheseendeavorsas‘ontologicalinnovation’,inwhichtheresearchadds

somethingnewtoanimportanttheorythatcouldonlyhavebeenaddedwithdatacollected

fromthenonclinicalsetting.Partofthisgoalof innovation(asDiSessaandhiscolleagues

3

describeit)isanewformofvaliditydubbedconsequentialvaliditybyHoadley(2004)that

definesthequalitybothoftheoryandresearchintermsoftheirabilitytosolveaproblem

that actually exists in the real world. Simply put, a good educational theory ought to

successfullysupportgooddesignprinciplesthatinturnshouldsuccessfullysupportagood

educational product or practice. The framers of the Next Generation Science Standards

certainlyhaveanabundanceofgoodeducationaltheory(setforthexceptionallybyDuschl,

2008) and evidence-centered design is a growing set of design principles that are often

describedintermsoftheiralignmenttothesenewstandards(DeBargeretal.,2016).The

timenowcomestoengagewiththeseprinciplesinordertoproduceneededassessmentand,

insodoing,innovateon(asPellegrino,Chudnowsky,andGlaserputit)howbesttoknow

thatstudentsknow(Pellegrinoetal.,2001).

The validity argument that foregrounds the evidence-centered design framework

hingesontheabilityoftheassessmenttosuccessfullysupportaparticularclaimregarding

whatstudentsknow(Mislevyetal.,2017).However,beforeevidencebackingaclaimcanbe

elicited,designersfirstneedtodefinewhatclaimsareusefulforrelevantstakeholders;such

asteachers,students,administrators,andpolicymakers(Debargeretal.,2016).Whilethe

broadstrokespresentedintheNextGenerationScienceStandards(NGSS;NGSSLeadStates,

2013)maybesufficient foradministrators lookingtomakeoverarchingclaimsaboutthe

science proficiency of students in their local purview (Pellegrino, 2013), teachers and

students requiremore detailed information about their present understandings that are

capableofbeingmappedtoalearningtrajectorythatendsatsomeproficiencygoal(Harris

etal.,2016). In ideal cases,prior literatureprovidesempirically-backedmodels forwhat

claimsoughttobemadeatcertaingradebands(Duncan&Rivet,2013).Forcross-cutting

4

concepts(CCCs),however,thelackofresearchintowhatstudentsknowastheybuildoverall

sciencecompetencymakesthedeterminationofappropriateclaimsdifficult.Forthisreason,

theprimaryresearchquestionofmydissertationis:Whatdoesstudentdemonstrationsofthe

crosscuttingconceptual reasoningaspectof their three-dimensional scienceunderstandings

looklikeatdifferentlevelsofoverallscienceunderstanding?

Toaddressthisbroadquestion,aprojecttitled:DevelopingAssessmentsandToolsto

Support the Teaching and Learning of Science Crosscutting Concepts (DAT-CROSS) was

created in collaboration between the Educational Testing Service (ETS) and Indiana

University.Theprojectwassupportedbyagrant(R305A170456)fromtheNationalCouncil

for Education Research (a division of the U.S. Department of Education’s Institute of

EducationSciences)todevelopanempirically-validatedprogressionforCCCunderstanding

(particularly regarding theCCC’sSystemsand SystemModels andStructure andFunction;

NGSSLeadStates,2013c).BuildingonmypreviousworkingrelationshipswithseveralETS

researchers,aswellasoneofmypriordevelopedexpertiseonCrosscuttingconcepts(Rivet

et al., 2016;Weiser et al., 2017), Iwas invited to design an assessment suite capable of

meeting the learning progression goals of the project. As with many IES-funded grant

projects,theDAT-CROSSendeavorisdesignedtocontinueacrossseveralyears,culminating

inmaterials that canbe readilyprovided to teachers to formativelyassess students’CCC

understandings.Thedataformyresearchquestionderivedfromthefirsttwoyearsofthe

project,inwhichtheteaminvestigatedthe‘usability’(Zaharias&Poylymenakou,2009)of

the assessment in order to validate the chosen tasks (Johnstone, Bottsford-Miller, &

Thompson,2006)andtominimizetheroleofnon-focalskills(suchasELAproficiencyor

5

computer-usecompetency)inmediatingstudents’abilitytoengagewiththetasks(Hoadley,

2004).

IntheprocessofworkingontheDAT-CROSSproject,itbecameclearthatpartofthe

challengeinthedesignofassessmentsforcrosscuttingconceptswasthelackofclarityover

howtheconceptsweredistinctfromoneanother.Asrecentlyas2018,researchersinscience

educationhaveexpresseddoubtsovertheutilityofevenattemptingtomeasurecrosscutting

conceptsin-use(Osborne,Rafanelli,&Kind,2018).Answeringmyinitialresearchquestion

entailed, first, answering a brand-new question: Is there evidence that the crosscutting

concepts of the NGSS are distinct constructs that can be measured as students use them?

Simultaneously, the goal of design research is not merely to confirm that the designed

product works as intended. The goal of design research, particularly when student

assessment is involved, is to identify all the impediments tomaking a locally functional

solutionwhilekeepingtoexistingdesignprinciples(Joseph,2004).Thisgoalmanifesteda

thirdresearchquestion,describedbelow,thatfocusesonthechallengesfacedbymysubjects

astheyinteractedwithassessmenttaskelements.

ResearchQuestions

Therearethreeresearchquestionsforthisstudy:

1. IsthereevidencethatthecrosscuttingconceptsoftheNGSSaredistinctconstructsthat

can be measured as students use them? Answers to this question undergird the

possibilityofameasurementmodel.

2. Whatdoesstudentdemonstrationsofthecrosscuttingconceptualreasoningaspectof

their three-dimensional scienceunderstandings look likeatdifferent levelsofoverall

6

science understanding?Answers to this questionwill inform the development the

evidencemodel.

3. What challenges do middle-school-aged subjects face in presenting their CCC

understandingswhileengagingwithan interactivesuitedesignedtotargetSystems-

Models and Structure-Function dimensions of their three-dimensional science

understandings?WhileCCCsrepresentanimportantconstituentofstudents’science

reasoning,thedimensionisoftenthesubtlestconstituentofagivenperformance.As

students face challenges in the task, determining that their barriers stem from

underdevelopedCCCunderstanding iscritical todeterminingtheusefulnessof the

assessment.

7

Chapter2

LITERATUREREVIEW

DimensionsoftheNGSS

The first step in understanding the rationale for this thesis study comes from

unpackingthemajordimensionsoftheNextGenerationScienceStandards(NGSS).

PerformanceExpectations:ThestructureoftheNGSS.

TheNextGenerationScienceStandardsarecomposedofperformance-based tasks

whichcombineall threedimensionsofsuccessfulscienceeducation: a)disciplinarycore

ideasthatmakeuptherelevantsciencecontent(e.g.,therelationshipbetweenenergyand

forces),b)scientificpracticesthattieinstructiontoactivityofthescientificcommunity(e.g.,

developingandusingmodels),andc)crosscuttingconceptsthatreflectthecommonalities

ofthetaskandmainideastoallsciencedisciplines(e.g.,causeandeffect)(Krajcik&Merritt,

2012;Krajciketal.,2014;NRC,2012).Thesynthesisofthesethreedimensionsatarelevant

grade-bandresultsinaperformanceexpectation(orPE)statedlikethis:“HS-PS3-5.Students

whodemonstrateunderstandingcan:developanduseamodelof twoobjects interacting

throughelectricormagneticfieldstoillustratetheforcesbetweenobjectsandthechanges

inenergyoftheobjectsduetotheinteraction.”(NGSSLeadStates,2013a).Thisreflectsa

radically altered view of the nature of science, and its relation to science education,

comparedtothatoftheNationalScienceEducationStandards(NationalResearchCouncil,

1996)andtheBenchmarksforScientificLiteracy(AAAS,1993).Bothfocusedondeveloping

inquiry practices and on replacing students’ alternative conceptions with those AAAS

consideredmoreaccurate (Kesidou&Roseman,2002).Whereearliernational standards

8

suggested that students have sufficient knowledgewhen they “know” particular science

content,theNGSSstatesthatstudentsrevealtheirknowledgeby“doing”ataskthatrequires

useofthatknowledge(Krajciketal.,2014).

DisciplinaryCoreIdeas

Disciplinarycoreideas(DCIs)arethedomain-specificcontentofscienceknowledge

writ-large,best exemplifiedby the informationcontainedwithin textbooksor flashcards.

IdeaslikeNewton’slawsortheprocessofnaturalselectionareexamplesofthisdimension

that represent key building blocks to modern scientific understandings (Krajcik, 2015).

These building blocks have been featured in science standards for about as long as the

conceptofeducationalstandardshaveexisted(DeBoer,1991).Unsurprisingly,thehistoric

salienceofthecontent,nowcategorizedasDCI,toooftendominatesintheclassroomatthe

expense of the other dimensions of the NGSS (Stroupe, 2015). Successfully planning

instructionaimedtowardsachieving thegoalsof theNGSSentailsacarefulbalancingact

betweenhelpingstudentsdevelopsophisticationaroundthesecoreideasandhelpingthem

understandthewaysthesecoreideasareactuallyusedbyscientists(Krajciketal.,2014).

Withoutadditionalsupportsforteachers,particularlyinassessingstudentunderstandings

oftheothertwodimensionsoftheNGSS,itissuspectedthatinstructorswilldefaulttoafocus

onDCIs(Pellegrinoetal.,2014).

ScienceandEngineeringPractices

Much of the recent literature on science learning that emerged from the learning

science fieldhascenteredonhowknowledgegetsused(Brown,Collins,&Duguid,1989;

CognitionandTechnologyGroupatVanderbiltUniversity,1992;Edelson,2001;Krajcik&

Merritt,2012;Nersessian,2002;Stroupe,2015).Thoughdescribedinearlierstandardsin

9

terms of scientific inquiry (DeBoer, 1991; Edelson, 2001), the various means by which

scientists(andbyextensionsciencestudents)engagewiththeirknowledgeandbuildnew

understandingarefeaturedastheseconddimensionoftheNGSS:ScienceandEngineering

Practices (SEPs) (Krajcik & Merritt, 2012). There are eight such practices: a) asking

questionsanddefiningproblems;b)developingandusingmodels;c)planningandcarrying

out investigations; d) analyzing and interpreting data; e) using mathematical and

computationalthinking;f)constructingexplanationsanddesigningsolutions;g)engagingin

argument from evidence; and h) obtaining, evaluating and communicating information

(Osborne,2014).Whileeachofthesepracticeshaveappearedinalitanyofpasteducational

standards(Osbourne,2014),priorresearchhasbeenunevenonwhatthosepracticeslook

like both at the expert and novice level (Stroupe, 2015). Overwhelmingly, research has

centeredonthedevelopmentofsciencemodels(Pluta,Chinn,&Duncan,2011;Schwarzet

al., 2009; Schwarz &White, 2005; Svoboda & Passmore, 2011) and the construction of

scientificarguments(Berland&Reiser,2009;Osborneetal.,2016;Sampson&Clark,2009).

Whilethisfocusisareasonablefunctionofthenatureofthescientificenterprise(McComas,

Clough, & Almazroa, 2002), it means that there is a dearth of empirically-validated

instructionalapproachestohelpingstudentsdeveloppracticalskillsalongmanyoftheother

SEPs.

CrosscuttingConcepts.

Crosscuttingconceptsareacomparativelynewadditiontothecorecomponentsof

sciencestandards(firstappearingaspartof the“unifyingconceptsandprocesses” in the

NationalScienceEducationStandardsproducedbytheNRCin1996[p.104])andrepresent

themes that underlay the scientific enterprise (Rivet et al., 2016) in all domains and

10

disciplines.Theseseventhemesare:a)patterns;b)causeandeffect;c)scale,proportionand

quantity;d)systemsandsystemsmodels;e)energyandmatter;f)structureandfunction;

andg)stabilityandchange(NGSSLeadStates,2013b).TheNGSSpridesitselfonitsempirical

backing,anditsframersroutinelyclaimthatthescopeandsequenceencouragedbytheNGSS

anditssupportingdocumentswasderivedfromevidencearoundhowstudentslearnscience

concepts(Duschl,2008;Duschletal.,2007;NRC,2012).Whilethisclaimismostlytruefor

disciplinarycoreideas,andpartlytrueofscienceandengineeringpractices,thereisawell-

establishedlackofevidenceforcrosscuttingconcepts(Rivetetal.,2016).Forthisreason,

somecriticshaveclaimedthatCCCsmakeapooradditiontothenewstandards(Osborneet

al.,2018),butIdisagreewiththeassessmentthatthecurrentabsenceofevidenceforCCCs

instudentworkisevidencethatthedimensionisnotequallyimportantinstudentlearning

(Weiseretal.,2017).

As in the case of SEPs, what prior research does exist regarding student

understandingoftheconceptsatplayfortheCCCdimensionoftheNGSSislimitedtojusta

fewoftheenumeratedthemes.Whilemuchhasbeenwrittenaboutmatterandenergy(Jin

&Anderson,2012;Neumann,Viering,Boone,&Fischer,2013;Stevens,Delgado,&Krajcik,

2009), this research generally treats such concepts as bounded by the uses that are

particulartoasciencedomain(akintoDCIs).Oneofthefewinstancesinwhichresearchinto

CCCs seems to consider the researched construct as a concept that might apply across

contexts iswith regards to students’useof systemsandsystemsmodels.Examining this

literatureacrossbiological,physical,andearthsciences(Breslynetal.,2016;Gunckeletal.,

2012;Jin&Anderson,2012:Mohanetal.,2009;Songeretal.,2009),Ifoundfourgeneral

pathwaysbywhichsophisticationinsystemsthinkingbuildsovertimeasfollows.

11

1. System Phenomena – As students build sophistication in their thinking around

systems, they are better able to describe phenomena as a system of many

simultaneousinteractions.

2. System Components – As students build sophistication in their thinking around

systems,theyarebetterabletobreakdownthecomponentsofasystemintotheir

constituent,dynamicparts.

3. System Relationships – As students build sophistication in their thinking around

systems, they are better able to describe the relationships between previously

identifiedcomponentsofthesystem.

4. System Boundaries – As students build sophistication in their thinking around

systems,theyarebetterabletodefinetheboundariesofthesystemandtrackthe

flowsofinputsandoutputsacrossthoseboundaries.

Thesepathwayswilllaterserveaskeyindicatorsofprogressusedinthedevelopmentofthe

tasksthatmakeupmyresearchendeavor(seeAppendicesAandB).

GiventhephilosophicandepistemicnatureoftheCCCs,Ibelieveitismorelikelythat

appropriateinstrumentsforfindingCCCsstillneedtobedeveloped(Rivetetal.,2016).In

previous research, my team found a set of four roles that students expressed in their

understandingof CCCswhen constructing scientific explanations (Weiser et al.,2017) as

listedbelow.

1. CCCsasLenses–TheroleoftheCCCistohighlightsalientfeaturesthatmaynotbe

immediatelyobviousduetoscale,scope,orsize.

2. CCCsasBridges–TheroleoftheCCCistodrawconnectionsbetweentwoentities,

facilitatingtransferofunderstanding.

12

3. CCCsasLevers–TheroleoftheCCCistocombineseveral,relatedideas,entities,or

representations in order to make understanding take on a new form towards a

particulargoal.

4. CCCs as Rules – The role of the CCC is to validate a representation’s utility in

explainingorpredictingthenaturalworld.

Whicheverrolestudentsemploy, itradicallyalterstherelationshipbetweentheevidence

usedintheexplanationandthesubject-phenomenarelationship.BythinkingaboutCCCsin

termsoftheseroles,IbelieveIcanconstructassessmentsthatarebothbetterabletoelicit

evidence of CCCs, and better contextualize that evidence in terms of students’ three-

dimensionalscienceunderstandings.

LearningProgressions

OneoftheadvertisedfeaturesoftheNGSSistheirbasisinempiricallyderived(Duschl

etal.,2007)researchintohowstudentslearn.ExemplifiedintheworkofGotwalsandSonger

(2013),learningprogressions(LPs)arethedominantframeworkforidentifyingwhatthat

empirical evidence looks like (Corcoran,Mosher, & Rogat, 2009).While discussions that

appear in Berland and McNeill’s (2010) work paint a very rosy picture of the positive

implicationsoftheLPframework,thereisalsomuchdiscussionfocusingonthenotionofa

“messy middle” (Gotwals & Songer, 2013, p. 599) where the pathways between well-

established stepping-stone ideas are less clear. The dominant perspective on learning

progressions and learning science research (Lehrer et al., 2001; Rivet & Krajcik, 2008)

suggestthatprogressionsareanchoredbythecontextsofaparticularphenomenon,making

ithardtogeneralizeonelearningprogression(forexample,onmoonphases[Plummer&

Krajcik,2010])toanotheraboutadifferenttopic,suchasNewton’slaws(Alonzo&Steedle,

13

2009).Someexistingprogressionsattempttobeapplicableacrossdisciplinarycontextsby

focusingonsciencepractices,asinthecaseofBerlandandMcNeill(2010);oronconcepts

thatappearsinallsciencedomains,asinthecaseofNeumannetal.(2013).However,such

progressions are few and far between (Duschl et al., 2011). LPs continue to be a useful

paradigm for research and theory, but incorporation into instructional practice requires

greaterelucidation.

ContemporaryCritiques

No reformmovement iswithout its constructive critics, andwhile therehasbeen

longstanding agreement that past standards have failed to live up to their lofty goals

(Eisenhart,Finkel,&Marion,1996;Lee,1997),thebeststepforwardalwaysremainsatopic

ofdebate.FortheNGSS,anearlycritiqueofitsemphaseswasthediminishedfocusonthe

nature of science,whichwas initially absent from the standards until its inclusion as an

addendum document (Appendix H of the NGSS [NGSS Lead States, 2013]). For science

educatorswhoarguedthatnatureofscienceisacriticalcontentelementthatneedstobe

explicitly discussed in classrooms (Abd-El-Khalick, Bell, & Lederman, 1998; Lederman&

Zeidler,1987),relegatingnatureofsciencecontenttoanappendixthatonlyappearsonsome

performanceexpectationswouldleadteacherstoavoidsuchtopicswhenweighedagainst

their other instructional responsibilities (Lederman& Lederman, 2014). Coupled to this

discussionwerecritiquesoverthepresenceofnewdimensionsandtheunevendegreeto

which existing learning progressions could describe how these dimensions grew in

sophisticationasstudentsmature(Duschl,Maeng,Sezen,2011).Returningtothenotionof

consequentialvalidity,thetruetestofthetheoriesundergirdingtheNGSSwillbethedegree

towhich instructional-materialdevelopers(includingassessmentdesigners)canproduce

14

thetoolscapableofhelpingstudentsreachthesupposedgoalsofthenewstandards(Furtak,

2017).

Concernsof equity in science learninghave alsobeena locusof critical reviewof

educational standards reform. As researchers like Lee (1997) and Brown (2005)

investigatedduringtherolloutand,later,enactmentofTheBenchmarksforScienceLiteracy

(AAAS,1993),thedefinitionsfortermslike“scienceliteracy”(Lee,2005)becomedefinitions

forwhohasaccesstoeffectresources.AsstatesbegintheprocessofimplementingtheNGSS,

anewgenerationofresearcherslikeMutegi(2011)andElam-Respass(2018)arewondering

hownewstandardswillaffecthistoricallyunder-representedteachersandstudents.Equity

intheNGSS,asasetofstandardsdrivenbystudentdoingoverstudentknowing,isuniquely

tiedtoassessmentequity(Rodriguez,2015).Nowmorethanever,alignmenttoboththetext

and the goals of the NGSS requires the design of assessments responsive to students’

linguisticskillsandsocially-mediatedcognitions(Mislevy,2016).

AnchoringLearningandAssessmentContexts

Theconceptoflearningascognitiveapprenticeship,developedbyBrown,Collins,and

Duguid(1989),tellsusthatlearningrepresentsamaster-novicerelationshipbetweenthe

teacherand thestudent inwhich thestudentbecomesenculturated inauthenticpractice

throughthedoingofrelevantactivity,muchastheapprenticeofoldpracticedhiscraftunder

thetutelageofamastercraftsman.Tothisend,theyarguethatalllearningissituatedinthe

contextinwhichitwastaught,withthiscontextbeingkeytothelearner’sabilitytotransfer

conceptsfromthelearningenvironmenttonewscenarios(J.Brownetal.,1989).Criticalto

developingmasteryoftransferis“authenticity.”Authenticityreflectsthedegreetowhicha

15

cognitive tool that develops during learning in the classroom context is, later, usable to

studentsasexpertpractitionersmightwieldit.TouseanexampledevelopedbyBrownetal.

(1989),ifsomeoneusesahammerasapaperweight,itisavaliduseofthetool;but,itisnot

an authentic use. In contrast, developing the use of a hammer in context of building a

birdhouserepresentsauthenticpracticeofusingthetoolasacarpenterorcraftsmanwould

useit.Theimportantdifferenceisthatourearlierexamplewouldnotbeextendabletomany

otherinstancesinwhichahammermightbeuseful,whileourlatterexampleprovidesuseful

knowledgeaboutthekindsofscenariosinwhichauthenticuseofahammermaybecalled

for.Inthemostfundamentalsense,everyinstanceinwhichatoolmaybeusedauthentically

hassomecommonalitywitheveryotherpossibleinstanceofauthenticuse.Contextualizing

instructionintermsofthisauthenticusehelpsthenovicelearnerrecognizecommonalities,

whichassistsinbothintegration(Rivet&Krajcik,2008)andtransfer(Yilmaz,Eryilmaz,&

Geban,2006)ofnewlearning.

However,cognitiveapprenticeship ismoreadescriptionofwhat itmeansto learn

thananexplicitmeansofevaluatingthequalityofagivenlearningenvironment.Thereis

somesense inwhichall learningenvironments are instancesof cognitiveapprenticeship

(Brown et al., 1989) and, moreover, such instances are not inherently sufficient for

developing a successful learning environment. Extending their theory on cognitive

apprenticeship, Lave andWenger (1991) presented amore complex analysis of learning

environments; i.e., legitimateperipheralparticipation(LPP),whichexamineslearningnot

onlyasapprenticeship,butasameansofmovingperipheralmembersofacommunityof

practiceintofullparticipants.Legitimateperipheralparticipationrepresentsananalytical

perspective for evaluating cognitive apprenticeship environments for their ability to

16

producedesirablechangesnotjustinnovices’abilities,butintheirrolewithinacommunity

ofpracticeandintheirself-identityinrelationtothatcommunity.Thisisnosmalltask,even

a well-established apprenticeship can fail if the novices do not perceive themselves as

movingfromperipheralparticipationtocentralparticipation(Lave&Wenger,1991).

IntheensuingdecadesofresearchfollowingthatofLaveandWenger,keyideasfrom

the frames of cognitive apprenticeship and legitimate peripheral participation have

continuedtoproveuseful.Examiningtheroleofscienceidentity(asinfluencedbypeerand

by teacher evaluation of language use), Brown et al. (2005) reiterate the importance of

identityshift(Lave&Wenger,1991).Understandinguseoftheestablishedtechnologyofa

community of practice is needed in all learning environments, and LPP presents such

understandingthroughtheconceptoftransparency.Transparencyrepresentsaconjunction

oftheabilitytorecognizetheuseofaparticulartoolandthescenariosinwhichithasutility

(visibility)andtheabilitytousethattoolinordertoachieveadesiredend(invisibility).Lave

andWenger(1991)presenttheexampleofawindow,whichoneusestolookoutsidefrom

theinsideofaconfinedspace(itisinvisible);butsimultaneously,wemustrecognizeitas

somethingwecanlookoutofinordertodoso(ithasvisibility).Atoolmustbetransparent

tothelearnerforthemtoeffectivelyuseitinbothperipheralandcentraltasks.Themedical

school studentmust be able to use a stethoscope both in their novice, peripheral stages

working with dummies and in their central role on actual humans once they become

residents.Failurestoseeacognitivetoolbothinitspresent,noviceuse,andhowexperts

similarlyuseitcanoftenhavedisastrousresultsforstudents’viewsontheutilityoftheir

scienceknowledge(Sandoval,2005).Thisidearemergedincritiqueofthen-presentscience

standards by Eisenhart, Finkel, and Marion (1996) who highlighted the importance of

17

couchingwhat is taught in the light of activities students find relevant to them and, by

extension,arebetterabletoseethedesiredend(s)ofapieceofscienceknowledge.Authority

also matters in terms of classroom discourse. Legitimate learning sets itself apart from

traditional instruction inwhich themaster will talk about a practice, opting instead for

encouragingperipheralmemberstotalkwithintheestablishedrulesofthecommunity(Lave

&Wenger,1991).Ford(2008)reiteratesthisideainsuggestingthatateacher’sauthority

shouldextendfromtheirabilitytohelpstudentsengagewiththescientificdiscipline(and

itsrules)ratherthanfromtheirpositionwithintheschoolhierarchy.

RichPerformanceTasks

For assessment development, all this researchhas served to show that the things

students have learned are much more contextually dependent than might have been

suspectedbypriorreformmovements(whichfocusedmoreonwhatscientistsknowthan

whatstudentknow;Eisenhartetal.,1996).Consequently,assessmenttasksthatmeasure

learningoughttomirrorthecomplexityofthelearningprocessbyacceptingawidearrayof

potentialperformancesthatallsuccessfullymeetthecriteriaofatask(Mislevy,2017).Such

tasks are known as ‘rich performance tasks’ and are well aligned to the philosophical

underpinnings of the NGSS (Gorin & Mislevy, 2013). Unfortunately, they are also more

resource intensive (in capital, cognitive load, and time) than more traditional tasks.

Determininganddesigning learningtoachievetheappropriatebalancebetweenrichness

anddomainrangeremainsatopicofcontinuedresearch(Weiser&Liu,2018).

Socio-scientific issues, cases where there is a salient intersection between the

understandingsgeneratedbyscientistsandongoinghumanbehavior(Kitcher,2010),are

productive contexts for creating rich tasks (Morin, Simonneaux, & Tytler, 2017).

18

Unfortunately, students struggle to readily engage with their science knowledge once a

politicallenshasbeenappliedtoaproblemcontext(Morin,Simonneaux,&Tytler,2017)and

teachersarehesitanttowadeintoacontextthatmayproverifewithpoliticalpitfalls(Kilinc,

Demiral,&Kartal,2017).Designingtowardsasocio-scientificcontextforanassessmentmay

provefruitfulforelicitingrobustevidence(whichmaybeneededwhenthetargetconstruct

is sosubtleasareCCCs [Weiseretal.,2017]),but careshouldbe taken tochoose those

contextswheretheinstructoralreadyhasstrong,positiveaffect(Kilincetal.,2017).

ConnectiontotheDimensionsoftheNGSS.

TheNextGenerationScienceStandardsrepresentamajorshiftawayfromBruner’s

inquiry(DeBoer,1991)andtowardsmoretransformative,social-constructionbasedactivity

(Blumenfeld et al., 2000). This shift, in addition to the redevelopment of benchmarks as

performance expectations (Krajcik et al., 2014) represents clear attempts to establish

technologic transparency (Lave & Wenger, 1991) within science content and to induce

knowledgecyclesamongpeersandnear-peersthroughdiscourse,cooperation,andcritique

(Lave &Wenger, 1991; NRC, 2012). Along these dimensions, the NGSS represent a step

forward fromProject 2061 in establishing legitimate peripheral participationwithin the

classroom.However, thechoicetopresentthesestandardsasbeingwhollydifferentiated

fromcurriculummaterials(Duschletal.,2007),whileconducivetoteacher-studentcentered

instructionaldesign,alsoplacesasizableburdenoninstructorstoredeveloptheirlessons

withoutasmuchinstructionalsupportasAAASprovided(AAAS,2000).

19

ElementsofRigorintheDevelopmentofAssessments

ValidityandGeneralizability

Assessmentsexistintermsoftheirvalidity(Pellegrinoetal.,2001):dotheymeasure

whattheyclaimtomeasure?Inorderformyassessmentstohavecontentvalidity,theyneed

toprovidethesupportsthatwillallowassessorstomeasurestudents’CCCunderstandings.

However,thethree-dimensionalframeworkofstudentsciencelearningthatisthebasisfor

the NGSS (NRC, 2012) is clear in its assertion that evaluators cannot measure any one

dimensioninisolation,becausetheothertwodimensionsalwaysaffectthecontextofthe

evaluation.Thus,assessmentsdesignedtoaligntotheNGSSneedaveryparticulartypeof

structuralvalidityinwhichtherelationshipsbetweenthethreedimensionsareconsidered.

This is not a simple task and the central concept ofmy dissertation research is that all

currentlyexistingassessmentslackthisformofvalidity,becausetheydonotconsiderthe

roleofCCCs.Generalizabilityisanotherchallengeformyresearchproject.Alldesignprojects

arebuiltupondevelopingsolutionsforlocalproblems;thisisespeciallytrueofframeworks,

likeactivitytheory,whichtakeethnographicapproaches.BarabandSquire(2004)suggest

areconceptualizationofgeneralizabilitythattakestheformofwhattheycall“consequential

validity” (p. 8). For consequential validity, the designed solution is not the end goal of

research.Rather, it isatoolforevaluatingatheoryofdesign(Cobb,Zhao,&Dean,2009).

Therefore, my research seeks generalizability not by suggesting that the particular

assessment I have designed should be used in novel contexts, but rather by showing a

successful method for design (Clarke & Dede, 2009) of science assessments that makes

conclusionsaboutstudents’three-dimensionalunderstandingsinamorestructurallyvalid

way.

20

TheAssessmentTriangle

Theassessmenttriangle(Pellegrino,Chudowsky,&Glaser,2001)sitsatthecenterof

allclaimsregardingtheconsequentialvalidityofanypsychometricinstrument.Thegoalfor

anyassessmentistomakeaconclusionaboutalatentcognitiveprocessfromacollectedset

ofobservableevidence.Whileevidencefromobservationderivesfromcognitionprocesses,

theyarenotthecognitionitself.Aninterpretationschememustbeusedtoconvertfromthe

observations to the desired claims. These three corners of the assessment triangle

(cognition,observation,andinterpretation)needtoagreeiftheassessmentistobevalid–

observablesshouldbetargetedtobestelicitrelevantevidence,theinterpretationscheme

shouldbeabletoaccountforallmeaningfulobservations,andtheclaimsaboutcognition

shouldbelimitedtothosewhereobservablescouldbereasonablygenerated.

Evidence-CenteredDesign

Fromtheevidence-centereddesignperspective,assessmentservesasanopportunity

forstudentstoprovideevidenceofwhattheyknow,andasanopportunityforassessorsto

collect,evaluate,anddrawconclusionsfromthatevidence(DeBargeretal.,2015;Mislevy&

Haertel, 2007). Mislevy and Haertel (2007) suggest that an early step in developing

assessmentistheconstructionofaconceptualassessmentframeworkwhichdictateswhat

youareinterestedinaboutasubject(thestudentmodel),howyouwillcreateopportunities

for subjects to provide assessors with evidence (the task model), and what collected

evidenceneededfortheanalysislookslike(theevidencemodel).Theevidencemodelcanbe

further subdivided into an observationmodel that dictates what evidence will count as

applicable,andameasurementmodelthatdictateshowobservationswillbeconvertedinto

a form suitable for analysis (Mislevy & Haertel, 2007). The evidence model (and its

21

subdivisions),whichseemstocorrelatewiththeunderspecifiedinterpretationcornerofthe

assessmenttriangle(Pellegrinoetal.,2001),isthefocusofmyresearch.

Modelsoftheevidence-centereddesign(ECD)process(Figure2.1)frequentlytakea

formsimilar to thatofToulminarguments (Mislevy,2017) inorder tohighlighthowthe

validityofanassessmentisnotsomemetriclikeprecisiontobequantified,butratherisan

ongoing argument that the instrument is successfully supportinguseful claimsabout the

subjects(Baxter&Mislevy,2004).

Figure2.1Fundamentalsof theECDValidityArgument.Adapted fromMislevyetal. (2017)“Assessing model-based reasoning using evidence-centered design: A suite of research-baseddesignpatterns.”Cham,Switzerland:SpringerInternationalPublishing.ECDandNGSSPerformanceExpectations

Thegoalofassessmentineducationistomeasurecurrentlevelsofproficiencyina

particulardomain(inthiscase,science)inordertosupportfuturegrowth.TheNGSShave

22

redefinedwhatitmeansforstudentstobeproficientinscience(Pellegrino,2013)bylisting

a particular set of claims regarding what proficient students can do (elucidated in the

EvidenceStatementswhichaccompanyeachperformanceexpectation[NGSSLeadStates,

2013]). Thismakes the PEs of NGSS (or a bundle of them)well suited to the evidence-

centereddesignapproach,whichframesassessmentsastheinstrumentsthatelicitevidence

insupportofproficiencyclaims(DeBargeretal.,2016).OncewebundleasetofPEs,soalso,

wehaveselectedasetofclaimstomakeaboutstudents;i.e.,knownasthestudentmodel

(Mislevy&Haertel,2006).FromtherethePEscanbeunpackedintocomponentdimensions

aspartofadomainanalysis(Harrisetal.,2016)thatseekstooutlinethemanywaysdifferent

partsof thedimensional constructs integrateas studentsdemonstrate three-dimensional

understanding.Thenextstepistoconstructanevidencemodel(Mislevy&Haertel,2006)

capableofdelineatingvaluableevidenceofstudentproficiencyfromevidenceattributable

tothenon-focalknowledge,skills,orabilitiesthatagiventaskmightrequire.Ourevidence

model takes the form of a design pattern (Mislevy & Haertel, 2006), which highlights

recurrent themes in the assessment of any one PE from the bundle linked by a viable

sequenceof science/engineeringpractices.Finally,weconstructa taskmodel (Mislevy&

Haertel,2006) thatdefines theway the item/tasksetwillelicitevidence in the lightofa

provided scenario. In later subsections, Iwill further elaborate on the use of ECD in the

creationoftaskstargetingCCCconstructsfromabundleofperformanceexpectations.

EthicsofAssessment

PartoftheToulmin-esquevalidityargumentmadeviatheevidence-centereddesign

approach to assessment development is accounting for alternative explanations by

establishingthatthetasksusedwereappropriateforboththeusecase(Gorin&Mislevy,

23

2013)andthepopulation(Cohen&Lotan,1995).Studentswithminimalexperiencewiththe

terminologythatispresentedwithinataskmaystruggletoengagewithtaskitems(Nobleet

al.,2012).Thisstruggle,therefore,maybearesultofalackoflanguageratherthanalackof

conceptual understanding (Brown et al., 2005). Equity in education considers what is

sociallyrelevant tostudents (Freire&Macedo,1995)so that theyhave thebestpossible

opportunities to learn. Likewise, equity in testing considers what is socially relevant to

students(Wiliam,2010)sothattheyhavethebestpossibleopportunitytopresentwhatthey

know to assessors.To assuage some, thoughnot all, concerns allDAT-CROSS taskswere

passedthroughanequityandfairnessreviewprocessthatconsideredcontextrelevanceand

ELAdifficultyindeterminingtheappropriatenessofthetasks.Additional

DesigningtheDAT-CROSSAssessmentSuite

Crosscutting concepts are broad themes that consistently appear in similar forms

across disciplinary boundaries and, despite differences in conceptual content, seem to

perform similar roles within student understanding across contexts (Rivet et al., 2016).

These roles are subtle and frequently lose salience to assessors when compared to the

students’ more obvious disciplinary core idea competency. Assessing CCCs, therefore,

requiresrichperformancetasks(Gorin&Mislevy,2013)thatallowstudentstobringawider

swath of personal experiences and understandings to bear during the activity. Mislevy

(2017)providessomeinsightintohowrichtaskscanstillbeimplementeddespitetheirhigh

cognitiveload.Althoughtheperformanceofanyonestudentonanyonerichperformance

taskmaybetoomessytosuccessfullymakeindividual-levelconstruct-relevantconclusions,

it is possible touse repeatedmeasures of the construct acrossmany contexts andmany

24

individuals that combine (or overlap) to produce a clearer picture of students’ CCC

understandings.

DAT-CROSSwasdesignedtousethreesuchrichtasks,eachwithastorylinearounda

familiar system: infestation of a farm by a dangerous insect (ecological systems), the

shrinkingwateraccessofafarmtown(earth-hydrologysystems),andtheconsequencesof

starch-richdietsonhumanhealth(humanbodysystems).Eachstorylineisgroundedinreal

problemsthatscientistsseektosolve(Brownetal.,1989)thatstudentscancontextualize

(Noble et al., 2012) in light of their own experienceswith insects, waterways, and food

consumption.ThefollowingparagraphsexemplifythedesignprocessthatIusedtodevelop

theecologicalsystemsstoryline.

Once I selected the ecological system of the farm as being a viable context for

anchoring the assessment, thenext step is to find appropriateperformance expectations

(PEs)thattargettherelevantCCCconstructs,sciencepractices,andcore ideasapplicable

bothtothescenarioandtotheobjectiveoftheresearchproject(theCCCs‘systemandsystem

models’and‘structureandfunction’;NGSSLeadStates,2013c).IselectedabundleofMS-

LS2-3,MS-LS2-4,andMS-LS2-5(NGSSLeadStates,2013a)whichjointlydescribeataskof

modelingenergyandmatterflowswithinanecosystem,howthoseflowscanbeaffectedby

changestothephysicalcomponentsoftheecosystem,andhowhumansmightengineera

solutionthatseekstomaintainapre-existingecology.ItshouldbenotedthatthePEsofthis

bundledonotfullyaligntothedesiredCCCconstructs(acommonproblemduetouneven

distributionof CCCs across thePEsof theNGSS).However, addendumdocuments to the

NGSSfocusingonCCCs(NGSSLeadStates,2013c)notethatCCCsrarelyexistinisolationand

25

canfrequentlybegroupedthematically(i.e.‘energyandmatter’canbeausefullensinthe

understanding of systems or ‘stability and change’ can be critical to the continued

functioningofaparticularstructure).AppendixAhighlightshowthePEswereadaptedfrom

theiroriginalformtoplacegreateremphasisonthenewfocalCCCs.

Inprocessofmakingsenseofthephenomenon(Krajcik&Merritt,2012)regarding

the consequences of an insect infestation to a pre-existing ecology, and the steps a local

community might take to protect their farmland, students will naturally present their

understandingsofsystems(Breslyn,etal.,2016;Gunckel,etal.,2012;Jin&Anderson,2012:

Mohan,etal.,2009;Songeretal.,2009;Yoonetal.,InPress),foodwebs(Griffiths&Grant,

1985;Hogan,2000;Hokayem,Ma,&Jin,2015),argumentation(Berland&McNeill,2010),

and modeling (Schwarz et al., 2009). The prior literature on practices, DCI learning

progression,andexpecteduseoftheCCCSystemsandSystemsModelsservedtosettheclaims

regardingwhatitmeansforstudentstobescientificallyliterateintermsofthereal-world

scenario.Thisactedasaformofdomainanalysis(Mislevy&Haertel,2006)thatfacilitated

creationofataskmodeldescribingabroadsetoftasksthatcouldpotentiallybeconstructed

forthisresearchstudy;onlyasubsetofthosepossiblewereproduced.Figure2.4outlines

theprocessbywhichthethreeselectedPEswereunpackedandtranslatedintoataskmodel

(describedinAppendixA)appropriatetowardstheCCC-assessinggoalsoftheDAT-CROSS

project.

Oneofthebiggestchallengesassociatedwiththedesignprocesswashowtomaintain

thesalienceofCCCsevenafterintegratingthemorewell-supporteddimensionsoftheNGSS.

Unlike the integration process used by Harris et al. (2016), which combined all three

26

dimensionsatthesametimefollowingtheirdomainanalysis,Istaggeredtheintegrationof

theDCI.Thispermittedgreater in thedesignprocess toward the interactionofSEPsand

CCCs,beforethesalienceoftheDCIoverwhelmedthedesignprocess.

27

Figure2.2DiagramofEcologicalSystemsTaskModelDevelopment

28

DevelopingCognitiveLabProtocols

Inassessmentdevelopment,theroleofcognitivelabs–inwhichstudentsoftarget

ageand/orcontentfamiliarityworkthroughthetaskwhile“thinkingaloud”(Johnstoneet

al.,2006)–istovalidatetestelementsthatwillnotbereadilyaccessibleoncethetaskmoves

into production stages. It is of critical importance that test itemshave construct validity

(Peterson et al., 2017). However, in multiple choice questions especially, it may not be

possible to know if the reasons why students select a particular response is actually

reflectiveofreasoningwithintherelevantconstruct(Howelletal.,2013).Simultaneously,

thetargetaudienceofthefinalproductneedstobeviewedasusableanduseful(O’Brienet

al.,2008).Accordingly,thefinalresearchprotocolwasmanifestedintwohalves:acontent

validation half and a reflective use half. These two halves, described in subsequent

subsections,arepresentedinAppendicesCandD.

Aprotocolforcontentvalidation.Howelletal.(2013)describetworesearchgoals

addressed by the cognitive interview method. The first goal is confirming items are

appropriatelykeyedsuch that thestatedbestanswer is theonlyone thatstudentscould

successfullyjustifywithinthecontextofthequestion.Thesecondgoalistoreviewresponses

todistractoroptionstoensuretheyreflectareasonedguessratherthantheresultofsome

non-focal barrier. This requires the support of “design rationales” (Howell et al., 2013),

whichdescribeboththeintendedreasoningthateachtaskofanassessmentistargeting,and

theplausible reasonings thatwould reflect awell-reasoned inaccuracy rather thanause

issue. The answer to these questions support what Snow and Katz (2009) call the

“interpretiveargument,” a stageof theevidence-centeredvalidityargument inwhich the

designercanarguethattheinterpretationofevidencecollectedduringthetaskisactually

29

usefultowardsclaimsaboutwhatstudentsknoworcando.Petersonetal.(2017)notesa

cog-lab research cycle in which good understanding of the goals of each item supports

identificationofbothsuccessesandbarrierstostudentperformance.Below,Iexemplifyan

itemdesignrationaleforquestion2oftheDAT-CROSSassessment:

Figure2.3Question2ofDAT-CROSSwithItemDesignRationale,below

Key: Corn–Producer,Goat–PrimaryConsumer,

Cornrootworm–PrimaryConsumerHuman–PrimaryConsumerandSecondaryConsumer

ItemDesignRationale Themajordecisionstobemadeinthedesignofitemsisaddressedbelow.Whatistheassessmentitemasking?Thisitempresentsavarietyoftrophicrolesorganismscantakeonwithinanecosystemandasksrespondentstocorrectlyassociatestoryelements(corn,goats,people,andcornrootworms[CRWs])withthoseroles.What information is important? It is important to pay attention to respondents’ability to differentiate primary and secondary consumption, identifywhich kind oforganismsareconsideredproducers,andtoseeCRWsasinvasiveconsumersratherthankeydecomposers.

30

Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistforsomeselectdistractorresponses.Furtherrationalesmayneedtobeinvestigatedwithinterviewprobes.

1. The corn are producers – Corn produces edible biomass by convertingmatter(carbonintheairandwaterfromair/soil)byusingenergyfromthesun.

2. Thegoatsareprimaryconsumers.–Thegoatdirectlyconsumesthecorn.3. ThepeopleofTownsvillearebothprimaryconsumersandsecondary

consumers.–Humansconsumecornbothbydirectlyeatingitandbyeatingorganisms(thegoats)thateatthecorn.

4. The people of Townsville are one of either primary or secondaryconsumers-Humansconsumecorneitherbydirectlyeatingitorbyeatingorganisms(thegoats)thateatthecorn.(Note:checkrespondent’sreadingofthestoryandwhethertheynotedthatbothconsumptionsoccur).

5. The corn rootworms are primary consumers. – The CRWs directlyconsumethecorn.

6. Thecornrootwormsaredecomposers.–TheCRWscausethecorntodie(decompose).

In addition to outlining the tasks undertaken by students, Appendix C includes my

documentationofreasoninganddesignrationalethatundergirdseachresponseoption.

A protocol to inspect usability barriers. Nonetheless, it is not enough that

questions are construct relevant. Theymust also be perceived as authentic by students

(O’Brienetal.,2008)andaligntowhattheyshouldbeabledoattheirstageofdevelopment

(Johnstone et al., 2006). Inauthentic tools will feel frustrating and foreign, potentially

limitingabilityorwillingnesstoengagewithatask;whiletoolsthatarebeyondthescopeof

students’skillswillnotprovideconstruct-relevantinformation.Bensonetal.(2002)provide

aseriesofheuristicsforidentifyingusability,particularlyincomputer-assistedtasks(like

DAT-CROSS), which served to inform a second, semi-structured protocol. During these

interviews,Iaskedstudentstoreflectontaskelementsandidentifychallengesintaskclarity,

ability to error check, and their comprehension of the content in the task storylinewith

questionssuchas: “Was thevideouseful tohelpyouunderstand the relationships in the

31

system?”;“Basedonthevideo,whatwasthepurposeofthesimulation?”;and“Whatdidyou

learn from the video?” A more thorough detailing of the usability reflection protocol is

providedinAppendixD.

FrameworkforAnalysis

ConfirmatoryFactorAnalysis

Factor analysis is a critical tool in this process of supporting a validity argument

(Kline,1994).Factoranalysisuseslinearalgebratobreakdownthescoresassignedtoeach

itemintoalessernumberoflatentvariableswhichshould(iftheinstrumenthasconstruct

validity) map to the target constructs. Confirmatory factor analysis (as opposed to

exploratory factoranalysis) isused to ‘validate’ a setof constructs;whereas, exploratory

factor analysis is used to analyze data to potentially discover underlying constructs or

factors.

RaschModelsandWrightMaps

Whilemanyanalysisapproachesfitwithinthelearningprogressionparadigm(Rivet

&Duncan,2013),acurrentdominantmeansofinterpretingstudentresponsesinorderto

empirically supportahypothesizedprogressionmakesuseof theRaschmodel (Briggs&

Alonzo,2012).TheRaschmodel is a special classof item-response-theorymodelswhich

supposesthattheonlymeaningfulfactorsindeterminingthelikelihoodthatstudentigets

task item n correct is the relativemagnitudes of the student’s latent ability (δi) and the

difficultyoftheitem(βn).AsinEquation1,below,thesetwofactorscombinetodictatealogit

model.

32

Pr{$%& = 1} = +,-./01 + +,-./0 (3456789:2.1)

TheRaschmodelcanbeamendedtoconsiderpartialcredit(asinGotwals&Songer,

2013) and some examples exist inwhich a greater number of item-parameters (such as

discriminability)areincluded(asinWilson&Draney,2005),buttheprimaryend-goalofall

RaschmodelsistheWrightMap.Sincethetwofactors(abilityanddifficulty)aremodeled

ontoasharedintervalscale,wecangraphthemalongasharedaxis(e.g.,Figure2.2),whereby

each item (the right side of the map) sits at the same height at which a student of

corresponding ability should have a 50% chance of getting the item correct. By coding

questionstoparticularconstructlevels,theWrightmapcanprovidesomevisualevidenceof

patternsinthedifficultyofitemsrelativetotheconstructlevels(Rivet&Kastens,2012).In

the figure below (adapted from Gotwals and Songer [2013]), questions of code E5

consistentlysitbelowquestionsofcodeC8,suggestingthatconstructE5(regardingability

todetectscientificevidence)iseasierthanconstructC8(regardingabilitytodetectscientific

claims). This visual evidence can be further supported by ANOVA of difficulty scores to

determineifobserveddifficultydifferencesarestatisticallysignificant(Lee&Liu,2010).

33

Legend:The orange histogram bars indicate the number of accurate responses for eachevaluation item ina seriesof increasingdifficulty, someofwhichare indicatedby labelslistedatthebaseofthehistogram.Figure 2.4 Example of a Wright Map. Adapted from Gotwals & Songer (2013). Validityevidenceforlearningprogression-basedassessmentitemsthatfusecoredisciplinaryideasandsciencepractices.JournalofResearchinScienceTeaching,50(5),597–626.ActivityTheory

Developing an observationmodel requires deep thinking regarding what kind of

evidenceassessorscanobserveandwhatkindofevidence isuseful toobserve.Acareful

consideration of contextual factors can help assessors make these needed distinctions

betweenthemanyobservations.Fromthesituatedactionperspective(Nardi,1996),context

plays the most important role in transforming a potential assessment space into an

actualizedassessmentspace,oftentransformingthesespacesinwaystheassessorcannot

possiblypredict. Incontrast, thedistributedcognitionperspective(Nardi,1996)suggests

thatassessmentspacesexistasinstancesofsharedcognitionbetweenassessors(whoput

thinkingintomakingaproblemsolvable)andsubjects(whoputthinkingintoproducingthe

solution).Activitytheorysitsasasortofmiddlegroundbetweenthetwo(Nardi,1996)by

34

implyingthatwhilecontextultimatelydirectsthesubject’scognition,andthatcognitioncan

beconstrainedbytheassessortowardsaparticulargoal.Engestrom(2000),asthefounder

of activity theory, describes the framework as a method for analyzing the interaction

betweensubjects,theobjectoftheirperformances,theinstrumentsortoolsusedtooperate

ontheobjects,therulesforacceptableperformance,thenormsofthecommunity,andthe

division of labor among relevant parties. These six factors, which all contribute to the

achievement of a desired outcome, are frequently depicted as vertices of interconnected

triangles.Studentwork,too,isaformofactivity–depictedinhypotheticaltriangleformas

Figure2.3.

Figure2.5ExampleofEngestrom’s(2000)ActivityTheoryModelinStudentWork

Jonassen and Rohrer-Murphy (1999) explains how activity theory can serve as a

framework for evaluating and developing learning environments. While they focus on

activitytheoryasamechanismforevaluatingdesigninterventions,theideascancarry-over

to evaluationof studentperformances (as in theactivity theory-baseddesignprocessby

SparksandDeane,2015).Successfulstudentperformancesinvolveeffectivelyusingamodel

35

asatoolfortransformingtheobjecttoalignwiththeperformancegoal,activitytheorycan

beusedtounderstandwhentheseperformancesaredisrupted.Cohen(1998)suggeststhat

these disturbances are a function of mismatches between expectations of subjects and

objects, usually due to the differing contexts by which the two live. Anchoring context

(Cognition andTechnologyGroup atVanderbilt, 1992)may, therefore, prove effective in

constrainingcontextwhilemakingexplicittostudentsthekindsofperformancesdeemed

acceptablebythecommunity.

Ibelievethefindingsthatemergedbyemployingtheactivitytheoryframeworkwere

invaluableindirectingmuchoftheanalysisintohowtheDAT-CROSSassessmentmightbe

improved.Indeed,muchhasbeenwrittenontheaffordancesofactivitytheory(Nardi,1996;

Roth&Tobin,2002).Sincetheearly2000s,however,increasedattentionhasbeenplacedon

adapting activity theory to account for the roles culture and history play (Nussbaumer.

2012).Thecultural-historicalactivitytheorymodel(CHAT)considersthatwhatscriptsfor

activityexistforthesametoolsdevelopunderdifferentcontextsandmanifestindifferent

formsundertheinfluenceofboththelivedexperiencesofthesubjectbutalsothecollective

experiencesoftheculturethesubject inhabits.Morenuance isrequired, therefore, inthe

claimsImakeregardingthekindsofstudentswhomightbewillingtoperformusabilitytests

foracompanythatspecializesinassessmentdesign.

Accordingly, I cannot claim that participants are representative of the general

populationof futureusersbutIcansupposethattheseparticipantsareat leastbest-case

use-participantsfortheirgradeband.Thatis,anyusabilitychallengesfacedbythisbest-case

sample of users are likely to be faced by most other students, and possibly in a more

demandingway.Moreover,fortheveryfirstimplementationofanovelresearchenterprise,

36

itisoftenusefultobeginwithmoretractableparticipantstobetterestablishafoundation

forbroaderfutureresearch.Itislefttofutureresearchendeavorstoexaminehowstudents

useCCCsnotonlyacrossdisciplines(assetforthbytheNRC[2014]),butalsoacrosscultural

contexts.

37

Chapter3

METHODOLOGY

Inthischapter,Idescribetheproceduresentailedinbothcollectingandanalyzingthe

dataformydissertationresearchendeavor.Iuseddifferentformsofdataand,byextension,

employeddifferentanalyticproceduresforeachofmythreequestionsinterwovenamongst

eachotherinwhatCreswellwouldcallan“analyticspiral”(Creswell,2009,p.183).Below,I

restatemythreeresearchquestions.

1. IsthereevidencethatthecrosscuttingconceptsoftheNGSSaredistinctconstructsthat

canbemeasuredasstudentsusethem?

2. Whatdoesstudentdemonstrationsofthecrosscuttingconceptualreasoningaspectof

their three-dimensional scienceunderstandings look likeatdifferent levelsofoverall

scienceunderstanding?

3. What challenges do middle-school-aged subjects face in presenting their CCC

understandingswhileengagingwithan interactivesuitedesignedtotargetSystems-

Models and Structure-Function dimensions of their three-dimensional science

understandings?

A more succinct overview of these methods can be found in the data collection matrix

(AppendixF).

Addressing these research questions required two forms of data: a) themultiple-

choiceresponsesdirectlyproducedbyrespondentswhentheyengagedwiththetestsuite,

and b) selected interview responses obtained from the respondents’ narrative as they

participatedinathink-aloudtaskthatinvolvedareviewofallthecomponentsoftheactivity

(Hoadley,2004).The firstof thesedatasets isquantitative innaturewhile thesecond is

38

qualitative;thus,all-togetherrequiringamixed-methodsapproachcommonlyemployedin

thedesign-basedresearchparadigm(Brown,1992).However,unlikesomenarrativeforms

ofqualitativeresearchthatemphasizethesemanticmeaningsofthenarrativeasarecordof

theindividual’s livedexperiences(as inthecaseofethnography,casestudy,orgrounded

theory (Creswell, 2009); the goal of these interviews is to confirm that theprompts and

stimuliused in the assessment are successfully inducing students toprovideevidenceof

relevantcognitiveprocesses.Ideally,theassessmentsituationestablishesapositiveclimate

for activity, where the student is not faced with significant barriers from the non-focal

processesinvolvedinthedoingoftasks(i.e.,thesubjects’knowingthecorrectbuttonsto

clickonscreentologtheirdesiredmultiple-choiceselection,etc.).Priorlivedexperiences

play a role – as they do for all activities of life – in how students both engage with

assessments-writ-large and how they do so in the specific, somewhat rare context of

usabilityinterviews.However,theanalysisofsuchimpactsremainsatopicoffuturestudy.

DataCollection

ParticipantsandExperimentalSetting.

Thecognitivelabsthatmakeupthedatacollectionformydissertationresearchtook

placeatETSheadquartersinPrinceton,NJduringtheautumnof2018.TheDAT-CROSSteam

recruited45,middle-school-agechildren,primarilybyreachingouttoETSstaffwhohave

appropriately aged offspring. Middle-schoolers are the target participants, because they

representthetargetageofboththeDAT-CROSSassessmentsuiteandthegradebandofthe

NGSSperformanceexpectationsusedduringthedevelopmentprocess.Ultimately,about15

participantsfromeachofgrades6,8and10wererecruited,generatingaspreadinstudent

39

performances.ParticipantswereremuneratedfortheirtimewithaVisagiftcardatanETS

standardrateof$30.

Regardingmyparticipants,47%percentwereFemale,44%werewhite,38%were

AsianorAsian-Indian,11%wereAfrican-AmericanorBlack,whiletheremaining7%were

Hispanicorofotherracial/ethnicbackground.Amore

Process/Procedures

The formatof theDAT-CROSS suite isdetailed inAppendixC, includingquestions

usedduringtheassessment(whichwereamixtureofmultiplechoice,multipleselect,drag-

and-drop, and subject-constructed style items). Each question has a dedicated stimulus

designedtomapsuccessfulcompletionofthetasktothekeyprogressindicators,whilestill

beingcapableofmappingpotentialdistractorstolowerlevelsofthelearningprogression.

Duringtheassessment,subjectsengagedwithastorylinethatrequiredthemtointeractwith

simulationsandmodels inorder todraw theevidenceneeded to support socio-scientific

arguments(likethoseofEggertandBögeholz,2010).Theseargumentstakethegeneralform

ofhowbesttoimprovethefunctioningofanengineeredsolutiontoalocalproblem(likean

infestationofcrop-destroyinginsectstoafarm).Figure3.1exemplifiesonesuchquestion

(Question5)fromtheactivity.

Duringthisphase,theassentingparticipants(consentalsoprovidedfromtheirlegal

guardian)willbeaskedtocompletethetaskswhileverbalizingtheirthinkingabouteachof

their choices. Hammer (1994) has noted that even advanced students struggle to

spontaneouslyengageinthesetypesofinterviews,soaDAT-CROSSteammember(though

40

nottheauthorofthisproposal)waspresenttoguidethesubjectsthroughtheactivitiesin

ordertohelpparticipantsrecognizeinstancesinwhichsubjectthinkingmayhaveoccurred.

Figure3.1Question5ofDAT-CROSSEcosystemStoryline

Questiontext:Jonahandthescientistswanttoknowhowthecornrootwormswillaffecttheamountofcornthatsurviveuntilharvest.Usethetooltomakeamodelforthefoodwebtoshowtherelationshipbetweenthecorn,thegoats,theSun,andthecornrootworms.Key:Sun->Corn;Corn->Goat;Corn->Rootworm.

Each of the subjects responded in a ‘think aloud’ format of the assessment suite.

Additional questions were posed by the researcher targeting places where subjects

struggled(Hoadley,2004).Thiswascrucialtoestablishingevidenceoftheusabilityofthe

assessmentsuite,so that the future iterationsof thedesigncouldmoreeffectivelydevise

assessmentswithminimal interruptions by a proctor to explain aspects of the task. The

think-aloudprocesswasmoretimeconsumingthananon-verbalparticipationapproach–

closertotwohoursofparticipation,ratherthanone.

41

I scored (with theassistanceof fellow researchers) all non-constructed responses

according to their key – listed below each item in figures 2.5, 3.1, and the remainder in

AppendixC.Iscoredeachconstructedresponse(orCR)question(Questions10,12,and13)

accordingtotherubricinAppendixG,andthenIassignedvarioussubsetsoftheresponses

tofiveassistantraters.ForeachofthethreeCRs,35ofthe45responseswereratedbyat

least three people. Since constructed response scores are ordinal, and measured by

nonunique sets of multiple raters, a weighted kappa (Maclure &Willett, 1987) seemed

appropriate. I alsoconsideredsomeothermeasuresof interrater reliability suggestedby

Hallgren(2012),includingpercentagreementandintraclasscoefficient.Table3.1describes

theseratingsforeachconstructedresponsequestion.

Table3.1Inter-raterReliabilityRatingsforConstructedResponses

Question PercentAgreement

Krippendorf’sAlpha

Fleiss’Kappa(weighted)

Intra-ClassCoefficient

Question10 90.5% 0.506 0.501 0.603Question12 90.3% 0.444 0.439 0.533Question13 91.6% 0.608 0.603 0.671

I should also note thatmy role as a consultant precluded the proctoring of these

usability/cognitivelabs.Myanalysisinthisresearchstudywasderivedfrompseudonymized

dataoftheselabs,simultaneouslydiminishingpossiblesourcesofbiasthatImightotherwise

bringintothethink-aloudprocess,butalsolimitingmyagencyoverwhatquestionswere

askedbythelabproctors.

DataAnalysisMethodology

Theoverallgoalofmyresearchquestionsistodevelopaneffectiveevidencemodel

forstudents’understandingsofcrosscuttingconcepts,particularlyregardingtwooftheCCCs

42

Systems and System Models and Structure and Function (NGSS Lead States, 2013c). As

mentionedintheintroductorychapter,however,developinganevidencemodelrequiresthe

ability isolate relevant observations of CCC reasoning from observations of the other

knowledges,skills,orabilitiesthatareevokedbythetask(addressedbyResearchQuestion

3)while also differentiating evidence reflecting progressing levels of CCC understanding

(addressedbyResearchQuestions1and2).Addressingallthreeresearchquestionsrequires

amixofbothquantitativeandqualitativeanalyticalmethods.

QuantitativeAnalysis–ResearchQuestions1and2

DatagatheredtoaddressResearchQuestions1and2formthequantitativeaspectof

the research to be answered via factor analysis (Kline, 1994) and item response theory

(Gotwals&Songer,2013).Factoranalysisservesasthefirststepinconstructvalidation,in

which we examine if variation in performance across a broad set of questions can be

accountedforintermsofalessernumberoflatentdimensions(Kline,1994).Ideally,this

analysisrevealsthatavariablethatcutsacrosscontexts(muchasCrosscuttingConceptsdo)

isasignificantexplainerofoverallperformance.Usinganitem-responseRaschmodel,Ithen

placebothstudentsandCCCperformancealongthesamedimension,reflectinganincrease

initemdifficultyassociatedwithcomplexityofCCCunderstandingelicitedbythetask(the

equationforthismodelcanbefoundinChapter2ofthisthesis).

QualitativeAnalysis–ResearchQuestion3

ResearchQuestion3fitssquarelywithinaphenomenologicalparadigm(asdefined

byCreswell,2009).LiketheapproachtakenbyDuncanandTseng(2011)andbyHoganand

Fisherkeller(1996),phenomenologicalstudyregardingtheexperienceofinteractingwith

curricular or assessment materials can serve as productive frames for evaluating the

43

usabilityofeducationalmaterials.Bothgroupsofresearchersmadeuseofeticcoding(where

thematiccodesaregeneratedpriortoanalysis;Bricker&Bell,2008)tocheckalignmentof

studentworktothedesignedgoalsoftheinstructionalmaterialsandemiccodes(inwhich

themesareallowedtoemergeintheprocessofanalysis)inordertoprobeinstancesinwhich

misalignmentwasfound.Iderivedeticcodesformyanalysisfrompriorpublishedresearch

onthenatureofCCCs(Rivetetal.,2016;Weiseretal.,2017)aswellasknownavenuesof

progressionsinsystemthinking(Breslynetal.,2016;Gunckeletal.,2012;Jin&Anderson,

2012:Mohanetal.,2009;Songeretal.,2009).ThesecodescanbefoundinAppendixG.

Hermeneutics. Translating the words of students into conclusions about their

thinking involves hermeneutic analysis (Eger, 1992) in which the language choice of

students in their crafting of arguments and explanations is evidence for their level of

understanding.Eger(1992)exemplifiesthisbynotingthattheclaimthatanobject‘has’a

forcemaysound likeanobject ‘exerts’or ‘isacteduponby’a force, thusentailingavery

different understandings of Newton’s laws. As in the work of Rivet et al. (2016), such

analysesdrawtheirarrayofconclusionsfromsubtledifferencesinwording(allowingfora

hostofalternativeinterpretationsofthedata).Inordertofurthersupportmyanalysis,the

findings were analyzed quantitatively in terms of their alignment to the hypothesized

evidencemodelandmembercheckedwiththeotherfacilitatorsoftheusabilitystudy.

Activity theoryasananalytical framework.Within theprevioussections, Ihave

describedthebroadproblemofdesigningassessmentappropriatetotheNGSS(DeBargeret

al., 2015) and have outlined activity theory as a framework for thinking deeply about

potential solutions to that problem. Figure 2.3 from the literature review (reprinted as

44

Figure3.2,below)exemplifiesonepossibleactivitytheorytrianglethatdemonstratesthe

interactionbetweentheresponses/activitiesofthestudentsandtheevidence:

Figure3.2ExampleofActivityTheoryModelatPlayinClassroomWork

Thesetrianglesservetoinformtheevidencemodeloftheconceptualassessmentframework

(Mislevy&Haertel, 2007) for this research, and highlight a clear problem space for the

designofanassessment.Mydesigninterventionrequired:(a)assessmentwhichmakesthe

evidencedictatedbytheactivitytheorymodelsalient(Mislevy&Haertel,2007)and(b)tools

for the teacher thathelps themmeasure thatevidence (Joe,Tocci,Holtzman,&Williams,

2013;vanEs&Sherin,2002).

Roth and Tobin (1992), in their use of activity theory to improve teacher

preparedness programs, describe several different forms of contradictions that impede

successfulengagementinaparticularactivity.Ofthese,twoareofparticularrelationtothis

research project: (a) differences between the language of the classroom and that of the

student,and(b)differencesinmotivebetweenassessmentactivityandtheactivityofexpert

scientists.Intheformercontradiction,thegreatestchallengetoelicitingevidenceofCCCsis

45

gettingstudentstounderstandandrecognizeCCCuseintheirownwork.CCCslikepatterns

andsystemshavemeaningsinthesciencesthataredistinctfromtheireveryday,English-

language meanings (NGSS Lead States, 2013b). Because assessment materials are

linguistically constrained, students receive some prompt in English and, then, have to

interpretwhatthatmeansrelativetotheirscienceknowledge(Lee,Quinn,&Valdes,2013).

Inpilotstudies,lookingforevidenceofstudent’sCCCunderstandings–asinWeiseretal.

(2017)–studentsroutinelyaskedforclarificationofthemeaningofpatternsandsystems.

Thisputresearchintoadilemma:clarifymeaningatthecostofpossiblyimposingourown

understandingsoftheCCCsontostudents,orrefusetoclarifymeaningatthecostofstudents’

abilitytoprovideevidenceoftheirreasoning.Bycombiningfieldnotesoftheconfederate

thinkaloudfacilitatorswiththepost-taskinterviewquestions,Iusedactivitytheoryasalens

toexamineifthelanguageoftheassessmentwassuccessfulinelicitingstudents’displayof

their CCC understandings. In the second-referenced contradiction (differences inmotive

between assessment activity and the activity of expert scientists), students generate

epistemic cleavages between their own way of knowing and the ways of knowing

appropriatetoscientists(furtherexpoundedinSandoval,2005).Activitytheoryservesasa

useful framework for ensuring that the finalized, three-dimensional assessment is an

effectivestepping-stonetowardsthepracticeofexpertscientists(Stroupe,2015).Forboth

contradictions of interest, activity theory is not the framework for how the assessor

interpretsevidence,butratherhowitsolicitsevidence(thetaskmodelofDeBargeretal.,

2015).Returningtothenotionsofvalidity,Ihavediscussedinaprevioussection,activity

theory serves as a framework for analyzing field notes, student responses, and teacher

probestoensurethattheassessmentshaveconstructvalidity(wherebytheconstructisthe

46

activitydonebypracticingscientists)andconsequentialvalidity(wherebytheassessment

activitiesdonothaveglowingcontradictionswhichinhibittheiruseintheclassroom).

EthicsandReflexivity

Theapproachusedtoanswermyresearchquestionshasseveralfeaturesthatpose

both general and unique ethical challenges falling into three broad categories: (a) the

concerns associated with any study carried out on human subjects, (b) the concerns

associated with remunerating subjects for their participation, and (c) the concerns

associatedwiththedisconnectbetweenmyroleasdataanalyzer/researchcoordinatorand

theroleofotherDAT-CROSSteammemberswhowerethePIsoftheusabilitystudy.Ofthe

foremost, this study was approved by the appropriate Institutional Review Boards as

fulfillingtheirrequirementsandstandards.

47

Chapter4

RESULTS

SummaryStatistics

Table4.1presentsperformanceaverages foreachassessment item(listedas sub-

itemswhererelevant)oftheDAT-CROSSecosystemtask(AppendixC)categorizedbyboth

gradeandgender.Thefirstitem,Q1,wasunscoredasitservedasanactivatorthatcouldnot

bekeyedtosomebest-possible-response.Differencesbetweengradesreflectsomeevidence

ofalearningprogression,whiletheabsenceofgenderdifferencesissurfaceevidencethat

itemswerenotgenderbiasedincontentorstructure.

Table4.1AveragePerformanceonAssessmentItemsbySubgroup

OVERALLAVG G6AVG

G8AVG

G10AVG

MALEAVG FEMALEAVG

Q2CORN

.91(.28)

.88(.33)

.94(.25)

1.00(0.00)

1.00(0.00)

.81(.40)

Q2GOAT

.62(.48)

.71(.47)

.44(.51)

.82(.4)

.71(.46)

.52(.51)

Q2PEOPLE

.04(.21)

0.00(0.00)

0.00(0.00)

.18(.4)

.04(.2)

.05(.22)

Q2ROOTWORM.04(.21)

.06(.24)

0.00(0.00)

.09(.30)

.08(.28)

0.00(0.00)

Q2SUM1.62(.77)

1.65(.70)

1.38(.62)

2.09(.83)

1.83(.70)

1.38(.80)

Q3.93(.25)

.88(.33)

.94(.25)

1.00(0.00)

1.00(0.00)

.86(.36)

Q4.33(.47)

.24(.44)

.38(.50)

.36(.50)

.25(.44)

.43(.51)

Q5.39(.49)

.35(.49)

.20(.40)

.73(.47)

.39(.49)

.38(.5)

Q6PT1.73(.44)

.71(.47)

.75(.45)

.82(.4)

.71(.46)

.76(.44)

Q6PT21.00(0.00)

1.00(0.00)

1.00(0.00)

1.00(0.00)

1.00(0.00)

1.00(0.00)

Q6SUM1.73(.44)

1.71(.47)

1.75(.45)

1.82(.4)

1.71(.46)

1.76(.44)

Continuednextpage

48

Table4.1AveragePerformanceonAssessmentItemsbySubgroup,continued

OVERALLAVG G6

AVGG8AVG

G10AVG

MALEAVG

FEMALEAVG

Q6SUM1.73(.44)

1.71(.47)

1.75(.45)

1.82(.4)

1.71(.46)

1.76(.44)

Q7CORN.96(.21)

.88(.33)

1.00(0.00)

1.00(0.00)

.96(.2)

.95(.22)

Q7GOATS

.87(.34)

.76(.44)

.88(.34)

1.00(0.00)

.83(.38)

.90(.30)

Q7PEOPLE

.33(.47)

.41(.51)

.25(.45)

.27(.47)

.25(.44)

.43(.51)

Q7SUM2.16(.67)

2.06(.90)

2.13(.50)

2.27(.47)

2.04(.69)

2.29(.64)

Q8.6(.49)

.65(.49)

.5(.52)

.73(.47)

.67(.48)

.52(.51)

Q9.18(.38)

0.00(0.00)

.38(.50)

.18(.40)

.17(.38)

.19(.40)

Q101.38(.85)

1.06(.75)

1.63(1.02)

1.55(.69)

1.25(.85)

1.52(.87)

Q11.02(.15)

0.00(0.00)

0.00(0.00)

.09(.30)

.04(.20)

0.00(0.00)

Q121.11(.77)

.94(.90)

1.13(.62)

1.36(.81)

1.08(.78)

1.14(.79)

Q131.16(.89)

.94(.83)

1.25(.86)

1.45(1.04)

1.21(.93)

1.1(.89)

TOTAL11.6(2.95)

10.47(3.26)

11.63(2.45)

13.64(2.25)

11.63(2.92)

11.57(3.12)

Note1:Standarddeviationsinparenthesis.Boldtextsignifiesstatisticallysignificantdifference(α=0.05)fromgrade6scores

ResearchQuestion1

IsthereevidencethatthecrosscuttingconceptsoftheNGSSaredistinctconstructsthatcanbemeasuredasstudentsusethem?

Researchquestion1addressedtheissueifcrosscuttingconceptslikeStructureand

Function(StFinFigures4.1and4.2)orSystemsansSystemModels(SyMinFigures4.1and

4.2) are different and measureable constructs that can be assessed. For DAT-CROSS,

questions2 through8weredesigned to target theStFCCC.Thesequestionsentailed the

followinggeneralaspects:a)constructingamodeltounderstandthetrophicstrutureofthe

49

farm,b) identifying theroleof thecornasa trophicproducer,andc)predictinghowthe

introductionofanewconsumer(theWesterncornrootworm,D.virgifera)mightthreaten

thattrophicstructure.

Followingthosequestions,studentsinteractedwithasimulationthatdemonstrated

theoutcomeofimplementingatwoalternativestrategiesforcontrollingtheinfestationof

thefarm.Throughtthatinteraction,studentsrespondedtoQuestions9through13,which

weredesignedtotargettheSyMCCC.Thesefivequestionsentailedthefollowingaspects:a)

understanding the consequencs of not implementing any infestation control strategy, b)

comparingandexplainingtheefficacyofthedemonstratedcontrolstrategies,andc)making

arecommendationregardinghowthecontrolstrategyoughttobeusedinthefuture.

Includinggradeasapossiblefactorinfluencingperformance,Ihypothesizefromthe

designoftheDAT-CROSSassessmentfactorstructureillustratedinFigure4.1.

Figure4.2describes the resulting standardizedparameters from theconfirmatory

factor analysis. The number within each arrow represents the standardized correlation

betweenvariables.

50

Figure4.1HypothesizedFactorStructureofDAT-CROSSEcosystemAssessment.

Thehighlight of this analysis is the lackof correlationbetweenStF andSyMwith

confidence interval [-0.070, 0.033]. Following the convention outlined by Garver and

Mentzer (1999), a standardized inter-construct correlation that excludes 1 within its

confidenceintervalisevidencefordiscriminabilityofconstructs.

Legend:Grd=Student's Grade-year; StF=Structure and Function; SyM=Systems and SystemModels; Q#=the scorestudentsgotonanitemnumber#.Squareboxessignifyobservedvariableswhilecirclesconveylatentvariables

51

Figure4.2ResultingCorrelationsfromCFAAnalysis

ResearchQuestion2

Whatdoesstudentdemonstrationsofthecrosscuttingconceptualreasoningaspectoftheirthree-dimensionalscienceunderstandings look likeatdifferent levelsofoverallscienceunderstanding?

PreliminaryLearningProgression.FollowingtheresultsoftheCFAanalysis,Iused

separateRaschpartialcreditmodelstoestimatestudentperformanceonboththeStructure-

Function-targeting and Systems-and-System-Models-targeting items. These resulted in

Wright Maps (Figures 4.3 and 4.4, respectively) and fit statistics (Tables 4.2 and 4.3,

respectively).Ingeneral,using45participantsisonthelowendofacceptableforvalidating

aRaschmodel(deAyala,2010),sotheseresultsshouldbeviewedaspreliminaryandasa

meansoffindingavenuesofimprovementratherthanproofuntoitself.

Legend:Grd=Student's Grade-year; StF=Structure and Function; SyM=Systems and SystemModels; Q#=the scorestudentsgotonanitemnumber#.Squareboxessignifyobservedvariableswhilecirclesconveylatentvariables

52

Figure4.3Person-Item(Wright)MapofStructureFunctionItems

FromFigure4.3,thefirststandoutareQuestions2(inwhichstudentsidentifythe

trophiclevelofvariousorganismsonthefarm)and7(inwhichstudentsmakeaprediction

abouttheoutcomeoftheoncomingrootworminfestation).TheRaschpartialcreditmodel

(adaptedfromEquation2.1fromtheliteraturereview)estimatesthatitwaseasiertoget

fourresponsesoutoffourcorrectinQuestion2,thanitwastogetthreeresponsescorrect

and, similarly, estimates that it was easier to get two responses out of three correct in

Question7,thanitwastogetjustone.Thesestrangeresultsbearoutinthefitstatisticsas

Question2hadanoutfitTbeyondacceptablelevels(deAyala,2010).Questions2and7were

alsosomewhatuniqueinthattheypromptedmultipleselectresponses(i.e.,studentswere

askedtoselectallthatapply).

Legend:Top:TheordinatedimensionisahistogramofnumberofstudentsinrelationtoLogitsofdifficulty/abilityontheabscissa(theLatentDimensionsharedbyitemsandpeople).Bottom: The ordinate dimension lists Structure-Function-targeting questions (with partial scores asapplicable)inrelationtoLogitsofdifficulty/abilityontheabscissa(theLatentDimensionsharedbyitemsandpeople).

53

Table4.2FitStatisticsofStructureandFunctionItems

CHISQ DF P-VALUE OUTFITMSQ

INFITMSQ

OUTFITT

INFITT

Q2_SUM 22.588 44 0.997 0.502 0.469 -2.59 -2.86Q3 21.636 44 0.998 0.481 0.802 -0.78 -0.34Q4 54.053 44 0.142 1.201 1.147 1.07 1.16Q5 30.923 43 0.915 0.703 0.741 -2.02 -2.49

Q6_SUM 34.631 44 0.843 0.77 0.838 -1.01 -0.99Q7_SUM 31.003 44 0.93 0.689 0.725 -1.43 -1.26Q8 54.678 44 0.13 1.215 1.174 1.38 1.39

Overall,theSystemsandSystemModelsitemsseemtobehavewellwithsomedecent

evidence.Forexample, the responses toDAT-CROSSQuestion9 (inwhichstudentswere

asked to describe patterns in the data produced from the simulation as it simulated the

efficacyofthefirststrategyforcontrollingtherootworminfestation–addingnewpredators)

andtheconstructedresponsequestions(questions10,12,and13inAppendixC)alignsto

the hypothesized learning progression (detailed inAppendixB). That is, Question 9was

designedtoaligntolevel4ontheSystemslearningprogression(inwhichstudentsareable

tomakepredictionsaboutthefuturestateofasystem),aboutequaltothetargetalignment

oftheconstructedresponsesthatearnedathree-pointscorebasedonthescoringrubrics

(furtherdetailedinAppendixH).

54

Figure4.4Person-Item(Wright)MapofSystemsandSystemModelItems.

InFigure4.4,wefindexactlythatthealignmentbehaviorbearsoutinthemodeled

itemdifficulty(the latentdimensionaxis).Forexample,Question9hasanearequivalent

itemdifficultymeasureasthree-pointscoresinQuestion10(thefirstconstructedresponse

questioninwhichstudentsareaskedtoexplainwhythepredator-introductionstrategythat

was also the subject of Question 9 was not as effective as predicted). Examining the fit

statistics (Table 4.3), further examination is needed to understandwhy Question 12 (in

whichstudentsexplainwhyasecondcontrolstrategy–plantingatrapcrop–waseffective

at reducing the impacts of the infestation) has poor outfit T, and why students found

Question11sodifficult(whichwassimilarinformtoQuestion9butregardingtheefficacy

ofthesecond,trap-cropstrategyratherthanthefirst).

Legend:Top:TheordinatedimensionisahistogramofnumberofstudentsinrelationtoLogitsofdifficulty/abilityontheabscissa(theLatentDimensionsharedbyitemsandpeople).Bottom:TheordinatedimensionlistsSystems-and-System-Models-targetingquestions(withpartialscoresasapplicable)inrelationtoLogitsofdifficulty/abilityontheabscissa(theLatentDimensionsharedbyitemsandpeople).

55

Table4.3FitStatisticsofSystemsandSystemModelItems

CHISQ DF P-VALUE OUTFITMSQ

INFITMSQ

OUTFITT

INFITT

Q9 29.343 42 0.93 0.682 0.973 -0.51 -0.05Q10 37.621 42 0.663 0.875 0.846 -0.57 -0.74Q11 19.178 42 0.999 0.446 0.827 0.2 0.04Q12 23.946 42 0.989 0.557 0.562 -2.45 -2.42Q13 35.926 42 0.734 0.835 0.83 -0.75 -0.82

TheseRaschmodelsallowedmetoisolatesixsubjects(threeeachfromStFandSyM)

whoperformedatthebest,worst,andmedianofparticipants.Theirresponsesbothduring

thethinkaloudandinthereflectiveinterviewareexaminedinthediscussionchapter.

More on the Structure and Function and the Systems and Systems Models

Constructs.Togain furtherevidence that theStructureandFunction construct isunique

compared to the Systems and SystemModels construct, I compared themodeled student

abilityscores(deltafromEquation2.1,alongthelatentdimensionaxisfromFigures4.3and

4.4) fromtheirperformanceonthequestionsfromeachconstruct. If theconstructswere

equivalent, we would expect that the distribution among the population of each was

equivalent. Using a Wilcoxon signed-rank test (Kerby, 2014) to account for potentially

nonparametricvalues,thedistributionsofstudents’abilityinthetwoconstructsrejectedthe

nullhypothesisofequivalentdistributions(W=830,p<<.01).

ResearchQuestion3

What challenges do middle-school-aged subjects face in presenting their CCCunderstandingswhileengagingwithan interactivesuitedesignedtotargetSystems-Models and Structure-Function dimensions of their three-dimensional scienceunderstandings?

UnderstandingDisruptionsinStudentPerformance

56

InlightofthefindingsfromResearchQuestion2,inwhichcertainitemswerefound

to be poorly functioning, understanding the non-construct relevant challenges students

facedbecomesallthemorecritical.ToanswerResearchQuestion3,IreturntoEngestrom’s

(2000)activitytheorymodelasameansforanalysis.Withinhisframework,thestudentwho

weremysubjectshavedevelopedascriptforbeingassessedintheirscienceclassroomthat

hasbeencrafted,muchliketheconstructivistschema,inresponsetotheirpriorexperiences.

Astheyworkedwithinthenovelcontext(understandingecosystems)oftheassessmentand

inanovelsetting(onacomputeroutsideoftheirnormalclassroom),studentsmayfindthat

theirpriorscriptsareunproductiveorcounterproductive–leadingtoadisruptionintheir

engagementwiththetask.Inthefollowingsubsections,Ioutlinethreepatternsofdisruption

thatcommonlyoccurredasstudentsworkedwiththeassessment–eachdiagrammedusing

activitytheorytriangleswithredlinkages.

Disruption1:ParingtheFoodWeb.Thefirstsetofactivitiesinthefarmecosystem

assessment was designed to introduce students to modeling ecosystems as a set of

relationships betweenorganisms thatmight grow in complexity as neworganismswere

addedintothemodel.Whilestudentshadminimaldifficultyconstructingtheinitialmodel

(Q3 of the task as described in Appendix C) with 44 of 45 correctly connecting model

elements, theyexpressedgreater challenge inunderstanding the roleof themodelwhen

asked to incorporate the rootworm into their prior creation. Usability Participant 15

expressestheirconfusion...

Facilitator: So,thefirstisaboutthefirstmodelthatyoucreated.So,wasiteasytointeractwiththecomponents?

Usability15:Yes.Itwaseasy,butIthinkyoushouldlikeexplainmoreaboutwhatyouwantustodonexttime.Like,Igotthepartthatyouwantedustoshowliketherelationshipbetweenthem,butlikewhatkindofrelationship?Likethefoodchain

57

relationshiporlike,isitlikethesunfeedsthecornorisitthegoateatsthecorn?Thecorntakesfromthesunorisitthesunfeedsthecorn?Thecornfeedsthegoat?Isitbackwardsorisitforwards?

Student’schallengesintheseearlymodelscouldpropogateintolateritemsdespiteleveling

(inwhich studentswere shown the correct answer as part of interactionwith the task).

UsabilityParticipant30initiallyexplainswhytheharvestmenstrategymightnothavebeen

effective…

Usability30:Adding theharvestmendidnothelp increase thepercent corn yieldbecausethenumberofrootwormeggswerestillincreasingalso.

Only on later reflection, during the post-task interview portion, did they describe the

phenomenonusingideasfromthemodel…

Facilitator: So,wehavethesecondpartthatitwasrelatedtotheharvestmen.So,whatdidyounoticeintermsfromthedatafromthesimulationvideoandfromthechartandthegraphs? Whydoesitremain? Thinkthewayyouhavetoexplaintosomeone.Usability30:The corn harvested was going down but it wasn’t going down asgraphicallyasbefore.Andthenumberofrootwormeggswhereinashighasbefore.Facilitator: Doyouthinkthatthisstrategywaseffective?Usability30:Notreally,becausethey’restilllosingcornyieldandtherootwormeggsarestill,like,theyarestillincreasingandmultiplyasmuch.Facilitator: Whydoyouthinkthat thenumberofharvestmen introducedbythescientistwasthesameeveryyearfromyearthreetofive?Usability30:Because they want the harvestmen becoming -- they want theharvestmenhavingthepredatorsalsoandtakingoverthe--disturbingthefoodchain.Facilitator: Okay.Whatdotheharvestmendointhisecosystem?Usability30:Theygatheruptherootworms.Facilitator: Andhowdotheygetridoftherootworms?Usability30:Ithinktheywouldeatthem.

I call this pattern (as diagrammed in Figure 4.5) “paring the foodweb”, in which

studentspaidselectiveattentiontoideasfromthemodelingtaskwhichdidnottransferinto

makingsenseofthesimulation.

58

Figure4.5DiagramofActivityDisruptionWhenStudentInteractswithModelingTool

Disruption2:FailuretoInteractwithEvidenceAvailableintheSimulation.As

studentsmoveintothesecondphaseofthetask,theyareaskedtointeractwithsimulated

implementationsofthecontrolstrategiesinordertocollectevidenceabouteachstrategy’s

efficacy.Drawingasuccessfulconclusionentailedmergingevidenceavailablefromseveral

sources – information provided to students before the simulation regarding how each

strategy might work, a visualization of organisms in the farm ecosystem interacting

according to thecontrol strategy,adata table indicating thepopulationnumbersofeach

organismbyyear,andasetofgraphs(seeAppendixC).However,inpractice,studentsoften

restrictedthemselvestoonlyonesetofevidence,usuallyasinglegraphorthedatatable.

Below,UsabilityParticipant27explainshowheusedtheevidenceinreasoningaboutthe

harvestmenstrategy.

Facilitator: Yeah. So, the term harvestmen, are you okay with that term?Harvestmen?

Usability27:Well,Ididn’treallyknowitbefore. Facilitator: Okay,butwhenyouseethis,it’skindofbug Usability27:Itjustseemedlike,(00:42:50)isjustabug. Facilitator: Yeah,okay.So,forthesecondvideo,doyourecallwhatitwastryingto

tellyou?It’sabouttheharvestmen.

59

Usability27:Howtheygrewwhenaddedwhenlikeyouaddedtheharvestmenwiththerootworms.

Facilitator: Yeah. Usability27:Ifeellikeitdidn’treallymakeachangethatmuch. Facilitator: So,hereisthedatatablehere. Usability27:Istilldidn’tlookatthedatatable.Ilookedatthegraphs. Facilitator: Youstilllookedatthegraphs? Usability27:Yeah. Facilitator: So,youarelookingatwhetherthe--Ithinkyoulookatthisgraphalot,

right? Usability27:Yeah. Facilitator: Youarelookingwhethertherootwormeggsaregrowingordecreasing. Usability27:Because in thisone, it increased(00:43:37) likewenta littlebitand

wentbackup. Facilitator: Stillincreased? Usability27:Yeah. Facilitator: Okay.Andforthe--didyou--canyouexplainwhydoyouthinkthe

harvestmenmethodsdidn’twork? Usability27:BecauseIdon’tthinktheharvestmenworkedbecausehere,ifyoulook

at--Ilookedatthisonealotbecauseitstartedupthishigh,butthenitlookslikeit’sslowlystartingtodecrease.

These types of behaviors (diagrammed in Figure 4.6) resulted in constructed responses

which were based on changes in the population of rootworms, or on changes in the

populationofcorn,butnotboth(correspondingtoalevel1scoreintherubric).

Figure4.6DiagramofActivityDisruptionDuringStudentInteractionwithSimulation

60

Disruption 3: Salience of Systemic Relationships. The final disruption pattern

(depictedinFigure4.7)camewhenstudentslackedrelevantpriorknowledgetoengagewith

keytaskcomponentsintheireverydaylifeexperiences.Muchasoccurredindisruption1,

studentswereunable todescribeecosystemfunctioning in termsofsystemrelationships

(like predation or energy usage); instead they resorted to accounts, best described as

laypersonterms.Forexample,UsabilityParticipant26describeshowtheharvestmenscare

awaytherootworms…

Facilitator: Okay, perfect. We’re moving next to the two control methods, theharvestmenand the alfalfa. This video and this videowas about theharvestmen.Basedonthedata,whatisthemaintakeawayfromthissituation?

Usability26:Thattheharvestmen,theyhelpedbutnotthatmuchbecauseIfeellikeifthere(00:47:30)hadmoreharvestmenandwehadmoreharvestmeneggslikethatthenthatwouldkindofhelpbecausetheywereonlyliketenandthenumberofeggswereincreasing.ItwashardfortheharvestmentogotoeachandhelpthecornsoIfeelthatifwehadmoreharvestmen,thatwouldhavebeeneasierandwehavehadmorecornharvested.

Facilitator: Howdotheharvestmenhelpthecorn? Usability26:I thinktheyhelpedbecauseI thinktheykindofscareawaythecorn

rootwormsbecausethey’rebiggerorlike,yeah.

While such lay accounts could sometimes allude to complex interaction dynamics

between the corn, the rootworm, and the control strategy, more often the result was

explanations that misread available information. Usability Participant 1 inaccurately

describesthetrapcropmethodasineffective,because“Thealfalfaincreasedtherootworms

morejustbyprovidingmoreshelterforthemtoplanttheireggs.”

61

Figure4.7DiagramofActivityDisruptionDuringStudentConstructedResponses

62

Chapter5

FINDINGSandDISCUSSION

Thedesign-basednatureofmyresearchquestions ledmetoadoptamethodology

akintootherdesign-basedresearch(Brown,1992)byblendingquantitativeandqualitative

elements.Mirroringthe“analyticspiral”describedbyCreswell(2009,p.183),myqualitative

data served as a means of phenomenologically validating the activities that produced

quantitativefindings,whilethequantitativedataservedasatooltoidentifyandtriangulate

newavenuesofqualitativeanalysis.Morespecifically,thequalitativedatafromthethink-

aloudsandreflectiveinterviewshelpedtosupporttheclaimthatstudentswereusingthe

relevantconstructsintheirreasoningwhilethequantitativedatafromstudentperformance

ontheassessmenthelpedidentifytargetassessmentitemsinneedofrevision;andtarget

students for whom a deeper analysis of their statements would be most fruitful. This

spiralinganalysishelpedidentifyplaceswheretheassessmentcouldberevisedaswellas

providedsomeguidanceforthedevelopmentofbrand-newitems.Inthefollowingsections,

Idiscusstheresultsofmystudybyresearchquestioninrelationtothisoverarchinggoalof

implementingCCC-consciousassessment.

ResearchQuestion1

IsthereevidencethatthecrosscuttingconceptsoftheNGSSaredistinctconstructsthatcanbemeasuredasstudentsusethem?

The highlight finding from the CFA used for Research Question 1 was the

indistinguishable-from-zerocorrelationbetweentheStructureandFunction(StFinFigures

4.1and4.2)andtheSystemsandSystemModels(SyMinFigures4.1and4.2).Theremaining

resultsfromtheCFAareonlymoderate,withsomequestionshavingmuchbettercorrelation

63

totherelevantCCCconstructthanothers.Whilethep-valueofthechi-squaregoodnessoffit

was much greater than 0.05, the root-mean-squared-error of approximation (Mueller &

Hancock,2015)wasjustbarelywithintheacceptablerangewitha90%confidenceinterval

of[0,0.11].Overall,theitemswerebettercorrelatedandmorereliableintheSystemand

System Models CCC than in Structure and Function. Overall, these findings point to two

implicationsfortheDAT-CROSSassessment:a)moreitemsareneededinordertoimprove

intra-construct reliability, and b) understanding of how to design systems-reasoning-

oriented assessment exceeds understanding of how to do so for assessments oriented

towardsStructure-and-Function-basedreasoning.

Implicationa)isbeingaddressedbydevelopingtwonewsetsofitemsunderaparallel

designprocessaswasusedfortheusabilitystudy’sset.Thesetwonewsetsaredesignedto

usetherelevantCCCsinsimilarwaysbutinnewcontexts(hydrologicsystemsandhuman-

bodysystems).Furtherdiscussiononthedevelopmentofthenewitemsappearslaterinthis

chapter.

Implicationb)reflects, inmyestimation,thecurrentstateofthescienceeducation

fieldonStructureandFunctioncomparedtoSystemsandSystemsModels.Itiscommon,in

existing learning progression literature, to see phenomena (and the learning thereof)

portrayedasasetofrelationshipsthatorganizeintoasystematsufficientlysophisticated

levelsofstudentreasoning(Breslynetal.,2016;Gunckeletal.,2012;Jin&Anderson,2012:

Mislevy,2016;Mohanetal.,2009;Songeretal.,2009).Thesephenomenacanspanmany

relevantdisciplines,fromgeoscience(Breslynetal.,2016)toecology(Hokayem,Ma,&Jin,

2015;Jin&Anderson,2012)tohydrology(Gunckeletal.,2012)andbeyond(Mislvey2016),

facilitatingtransferintonewcontextsorcontentareas.Incontrast,StructureandFunction

64

has not been the focus of asmuch science-education literature. The outcomeof the CFA

reveals thatwhile the design framework can design items that are Systems and Systems

ModelsorientedandcandesignitemsthatarenotSystemsandSystemsModelsoriented,more

revisionisnecessarytotransformthenotSystemsandSystemsModelsconstructintoonethat

canmoredirectlyelicitsobservationspertainingtoStructureandFunction.

ResearchQuestion2

Whatdoesstudentdemonstrationsofthecrosscuttingconceptualreasoningaspectoftheirthree-dimensionalscienceunderstandings look likeatdifferent levelsofoverallscienceunderstanding?

OneofthemainaffordancesoftheRaschmodelusedtoanalyzestudentperformance

istheabilitytoplacestudentsalongadimensionofabilitythatisofthesamescaleasthe

dimensionofitemdifficulty(Briggs&Alonzo,2012).Thatallowsmetoestimatestudent’s

position along the hypothesized learning progression (described in Appendix B). In the

followingsubsections Idiscussstudent think-aloudresponses toaselectedStructureand

FunctionitemandaselectedSystemsandSystemModelsitemforstudentswhohavethebest,

theworst,andthemedianabilityscoreinthecorrespondingconstruct.Returningtothefour

rolesofCCCs(Weiseretal.,2017)fromtheliteraturereview,Ianalyzehowtherolesselected

byeachstudentdifferedastheyrespondedtothesameitem.

StudentResponsesatDifferentStructureandFunctionProgressionLevels.

Question5fromtheDAT-CROSSassessment(Figure5.1)wasthefinalclickanddragitem,

in which students amend a prior model of the farm to account for the introduction of

rootwormstotheecosystem. Itwasalsomodeledtohaveadifficultyscoreofabout1.14

logits,makingitoneoftheharderitemsintheStructureandFunctionset.Theinteractive

elementsofthetaskmakeitaproductiveavenueforexaminingthink-aloudresponses.

65

Figure5.1DAT-CROSSQuestion5Note:QuestionKey:Sun->Corn;Corn->Goat;Corn->Rootworm

HighStructureandFunctionability(Usability32).Usability32wasa10thgrader

whohad thehighestmodeledability score at about2.34 logits (corresponding to a75%

chanceofgettingQuestion5correctviatheRaschequation[Equation2.1]).Hereisthem

thinkingaloudwithartifactillustratedinFigure5.2.

Usability32:So,nowthecornrootwormswillaffecttheamountofcornsurviveuntilharvest.Tomakeonefoodwebofcorn(00:07:26)corngoestothesunandthecornrootworms.Thesunpowerthecorn,sothesunisreallyfaraway,soit’sgottobeup(00:07:37).Andthisgoatandthisrootworm,they’rebothnext.Sothecornisgoinginto the goat, but this corn is also going into the rootworm, so, that makes theproblem.In this response,Usability32 can correctlydescribe the teleological functioneach

entityplayed(eitherbeneficiallyordetrimentally)andusetheirunderstandingofadiscrete

setofrelationshipstobuilduptothewholestructure.Fromtheperspectiveofthefourroles

of CCC reasoning (Weiser et al., 2017), this is an example of ‘CCCs as Levers’ employed

successfully.

66

Figure5.2Usability32'sresponsetoDAT-CROSSQuestion5

Interestingly, while Usability 32 had the best performance on the Structure and

Functionitems(Questions2through8),theirperformanceonthelater,SystemsandSystems

Models items (Questions 9 through 13) were middling at best – with all three of their

constructedresponsesscoringa1basedonthescoringrubric.

Median Structure and Function ability (Usability 41). Usability 41 was an 8th

graderwhohadthemedianmodeledabilityscoreatabout0.74logits(correspondingtoa

40%chanceofgettingquestion5correctvia theRaschequation [Equation2.1]).Here is

themreflectingontheirresponsewiththeirartifactillustratedinFigure5.3:

67

Figure5.3:Usability41'sresponsetoDAT-CROSSQuestion5

Usability41:Iknewrootwormwould--likethatmorepeoplewereusingthecorn.Itwasliketakingcornawayfromtheotherpeoplewhowereusingit.Facilitator:Okay.So,youthinkrootwormistakingcornaway?Usability41:Yeah.Itwaslikedecomposingit.Facilitator:Decomposingthat. Okay. So,isthisthemodelthatyoudrew?Orthearrowsaredifferent?Usability41:Ithinkthearrowsarealittledifferent.Facilitator:So,what’syourarrow? Yeah,yourdirection is fromrootwormto thecorn?Usability41:Yeah.Facilitator:Whatdoesthatmean?Usability41:Thattherootwormswerelikeeatingthecorn.Facilitator:Eatingthecorn.Okay.However,here,thecorntogoat,isalsogoateatingthecorn,whythearrowisdifferent?Usability41: I think Idid thatbecauseof thepreviousone. I’dmeantadifferentthing.

Rather than construct a coherent structure inwhich the arrows alwaysmean the

same thing, Usability 41 uses his ‘CCC as a Bridge’ (Weiser et al., 2017) –with discrete

connectionsbetweenrelevantentitiesthataresensible(giventheircontentknowledge)only

68

asasetofpieces.Identificationofdiscreterelationshipsisacommonthemeatlowerstages

ofmanyLPs(Mislevy,2016)includingmyhypothesizedone(AppendixB).

LowStructureandFunctionability(Usability7).Usability7wasa6thgraderwho

hadthelowestmodeledabilityscoreatabout-1.21logits(correspondingtoa9%chanceof

getting question 5 correct via theRasch equation [Equation 2.1]).Here is them thinking

aloudwithartifactillustratedinFigure5.4.

Figure5.4Usability7'sresponsetoDAT-CROSSQuestion5

Usability 7: I think that it reflects your relationships between – the feedingrelationshipsbecauseifyoudon’thavethesun,thenyoucan’thavecornandifyoucan’tcorn,thenthere’snothingtofeedthegoatssotheywoulddie.So,ifoneisifyoudon’thaveone,theotheronewoulddieornotdowell.

In theirmodel,Usability 7 seems to develop an organizing principle, that the sun

needstoexisttosupporttheotherorganisms,butthenstrugglestocoordinatethatprinciple

69

intoacoherentmodel.Interestingly,underthefour-roleframework(Weiseretal.,2017),

thisparticipant’sbehavior isassociatedwithusingthe ‘CCCasaRule,’which(inthetask

model)wasplacedatthehigherendofthelearningprogression.Still,inthecontextofthe

question, itwas inappropriateandledthestudenttocreateamodel inwhichthearrows

meantdifferentthingsacrossdifferententities.

StudentResponsesatDifferentSystemsandSystemModelsProgressionLevels.

Question 10 from the DAT-CROSS assessment (Figure 5.5, below) was the first

constructedresponseitem,inwhichstudentswereaskedtoexplainwhythecontrolstrategy

inwhichpredatorswereintroducedtopreyontherootwormswasnoteffective.

Figure5.5DAT-CROSSQuestion10

Asaconstructedresponse,Question10wasscoredbyarubric(AppendixH)meaning

thateachpotentialscore(from0to3points)isassociatedwithadistinctdifficultymeasure.

Specificallya1-pointscorehadadifficultyof-2.31logits,a2-pointscorehadadifficultyof

0.47logits,andthetopscore(3points)hadadifficultyof1.55logits.Theconstructednature

of students’ responses entails ‘intentionality’ (Jonassen & Rohrer-Murphy, 1999), which

makethemaproductiveavenueforthinkingaboutstudentactivity.

70

HighSystemsandSystemModels ability (Usability20).Usability20wasa10th

graderwhohadthehighestmodeledabilityscoreatabout2.19logits(correspondingtoa

99%,85%,and65%chanceofscoring1,2,or3-points,respectively,viatheRaschequation

[Equation2.1]).Hereistheirconstructedresponse:

Usability20:Thereisthesamenumberofharvestmenthroughyears3to5buttherootwormeggsstillincreasedinnumberoverall.Iftheharvestmencouldonlykeep91 cornstalks harvestable in year 3, they are not going to keep more cornstalksharvestableiftheyhavemorerootwormeggstodealwithinthenextyears.

Usability20’sabilitytodetectatrendinthesystemand,fromthattrend,predictafuture

stateisindicativeofthe‘CCCasaRule”role(Rivetetal.,2016)usedsuccessfully.

MedianSystemsandSystemModelsability(Usability45).Usability45wasa10th

graderwhohadthemedianmodeledabilityscoreatabout0.01logits(correspondingtoa

91%,39%,and18%chanceofscoring1,2,or3-points,respectively,viatheRaschequation

[Equation2.1]).Hereistheirconstructedresponse:

Usability45:AddingtheHarvestmeninthefeildeveryyeardidn'tchangethecornyeildbecasuetheynumberofharvestmenstayedthesameandtherootwormsonlyaddaptedtotheirnewenvirnment.Theharvestmennumberstayedthesamewhilethe rootworms had eggs each year and only continued this cycle. So because theharvestmenwereaddedinsteadofincreasingcornyeild,thecornyeildonlydecresedlessrapidly{sic}.

IncontrasttoUsability20’sabilitytoconstructorganizingprinciplesthatdictatedthestate

ofthefarmecosystem,Usability45makestheCCCactasalensbyidentifyingrelevantparts

thatcouldhelpsupportanexplanation.Whilethatrolewassuccessfulinidentifyingevidence

relevanttotheirclaim,weseethattheirresponsehasnounderlyingreasoninganddescribed

nomechanismbywhichtheharvestmenandtherootwormsarerelatedtoeachother.

71

Low Systems and SystemModels ability (Usability 36). Usability 36 was a 6th

graderwhohadthelowestmodeledabilityscoreatabout-3.75logits(correspondingtoa

19%,1%,and0.5%chanceofscoring1,2,or3-points,respectively,viatheRaschequation

[Equation2.1]).Hereistheirconstructedresponse:

Usability36:BecausethereweretoomanyRootwormsfortheHarvestMen{sic}tofightoff.

These sorts of partial responses are commonplace both in our assessment and in past

assessmentsinvolvingstudentexplanations(Beck,Bookbinder,Lee,&Rivet,2017)inwhich

somerelevantclaimismadewithoutcitinganymeaningfulevidenceorprovidingsufficient

reasoning.

ResearchQuestion3

What challenges do middle-school-aged subjects face in presenting their CCCunderstandingswhileengagingwithan interactivesuitedesignedtotargetSystems-Models and Structure-Function dimensions of their three-dimensional scienceunderstandings?RoleSelectionandDisruptionsinStudentActivity.

FromthediscussionofthefindingsforResearchQuestion2,itseemsthatwhilethe

Roles(Rivetetal.,2016)donotalwaysaligntolearningprogressionlevels(assupposedby

the taskmodel [Appendix A]), students’ selection of Role does influence their ability to

construct explanations for phenomena. This is consistent with my findings from prior

research (Weiser et al., 2017). It stands to reason, given the influenceof ‘Role Selection’

(Weiseretal.2017),toconceptualizethedisruptionsfromtheresultschapterinthelightof

the4Rolesandwhichoneshadbeenselectedwhendisruptionsoccurred.

Disruption1.Indisruption1,Ifoundthatstudents,despiteinteractingwiththefood-

web model during questions 3-6, rarely carried ideas from that set of tasks into their

72

explanationsfortheefficacyoftherootworm-infestation-mitigatingstrategies.Instead,they

wouldfocusonlyonthecurrentmaterialavailableonscreen,identifywhatmatteredfrom

that set, and attempt to construct an explanation from there (often missing out on the

relationships,mechanisms,orreasoningrequired).Here,studentsusedtheirCCCsaslenses

toidentifycomponentstheyfoundrelevanttothetask,butthenstruggledinbridgingthose

components together. This behavior is consistent with prior research regarding student

viewsonbehaviorsandfunctions(Hmelo-Silver,Liu,&Jordan,2009)inwhichstudentscan

make sense of components and the individual behaviors of components before they can

describehowthosebehaviorsimpactotherentities.

Disruption2.Indisruption2,studentsfailednotonlytocarryideasfrompastscreen

intolateritems,theyalsofailedtociteevidencethatwaspresentrightinfrontofthem.The

bestexplanationforthisdisruptionwasthattheydidnothavesufficientpriorknowledge

aboutrelevantsciencepractice,andbyextensioncouldnotdeterminehowthatcontentwas

meant toserveasevidence for theirexplanations.Fromaroleselectionperspective, this

disruption underscores the three-dimensionality of science understanding. Even when

studentsareabletoselectanappropriaterolefortheCCCs(e.g.,tousetheirunderstanding

of thesystemtoconstructhierarchy intherelationshipsofsystemcomponents),without

sufficientunderstandingoftherelevantdisciplinarycoreideasorscienceandengineering

practice,itisnotpossibletoperformtotheexpectationofthestandards.

Disruption3.Disruption3wasacounterpointtodisruption2.Wheredisruption2

occurred when students failed to cite evidence on screen, disruption 3 occurred when

evidence was cited but the explanation constructed followed a lay account of the

phenomenon. This kind of behavior is common when students have developed

73

sophistication in theirCCCsandSEPsbefore theyhavedoneso in theirDCIs(Becketal.,

2017).InUsability26’sresponsetoQuestion10(asdescribedinChapter4),theyareableto

effectivelyusetherightrole,CCCasLever,toconstructasophisticated,albeitlay,accountof

thepredationstrategy.The linguisticsuccessofUsability26,contrastedwith thecontent

failing,revealsthehowimportantlanguageistotheCCCs.Asnotedintheliteraturereview,

the CCCs, while important in unique ways to the scientific domain, also play out in the

everyday senseofpattern, causeandeffect, systems, etc.That is, studentsdevelop these

conceptsbothincoordinationwiththeirlearningexperiencesinthescienceclassroomand

intheireverydayexperienceswhentheymightnotbeattendingtoothersciencecontent.

MakingAmendmentstotheEcosystemTaskStoryboard

Followingmyanalysison theusabilityof theexistingDAT-CROSS items, revisions

wereput forth toaddressusageconcerns fromResearchQuestion3andmakeconstruct

elements(asdescribedinthehypothesizedlearningprogression)moresalient,particularly

intheitemsmeanttotargetStructureandFunction.Additionalhelpertextwasincludedin

Questions 2 and 7 tomake clear that students should click onmore than one option as

appropriate.Language,regardingtheideathatorganismsonthefarmserveafunctioninthe

overallfunctioningofthefarmyardecosystem,totextonscreenbetweenquestions2and3

andbetweenquestions3and4.Thattextprovidestheanswerstopreviousquestionssothat

all students have the necessary content knowledge even if they responded to previous

questionsincorrectly).Laterintheassessment,moreemphasiswasplacedonmakingsure

studentsunderstoodwhythe farmerwas introducingthedifferentcontrolstrategiesand

whatheexpectedeachstrategytodo.

74

ImplicationsfortheDesignofTasksinNewSystemContexts

InadditiontoinformingrevisionstotheexistingitemsofDAT-CROSS,myresearch

sought to inform larger design ideas that could affect future CCC-targeting assessment

developmentwork.InthelightoftheIESgrantandthefindingsfromResearchQuestion1,

newitemsweredesignedtoimproveintra-constructreliabilitywhilealsoaccountingforthe

‘cuttingacrossdisciplines’(NRC,2014,p.37)natureofthecrosscuttingconcepts.Fromthe

findingsofResearchQuestion2, itwas clear that studentsneededgreater in-assessment

supportforthinkingaboutthefunctionsofstructuralcomponentsandhowtheysupportthe

overallgoalofastructure.Thesesupportsnotonlyincludemorescreensbetweenquestions

to introduce content, but also questions that are targeted to the function and behavior

progressionpathwaysfromthelearningprogression(AppendixB).Figure5.6exemplifies

oneofthesenewquestions.

Tominimizeusagedisruptionsacrossthewholesetoftasks,improvedsupporttext

andinstructionalvideosweredevelopedtohelpexplainhowvariousinteractiveelements

worked.

75

Figure5.6ExampleQuestionfromWaterUseSystemStoryline

LimitationstoMyFindings

Myresearchforthisdissertationcamefromonlyasmallsliceofabroaderongoing

project, asusability testing is often the first step in theprocessof assessmentvalidation

(Zaharias & Poylymenakou, 2009). Unlike later stages, which may involve many more

students (and accordingly, less rich data), usability tests involve fewer participants (my

study had 45middle-school-aged students) thatmay be selected on amore intentional,

nonrandomcriteria.ThechoicetousewhatIhavepreviouslycalleda“bestcase”sample,

particularlyinlightoftheframeworkofactivitytheory,impactssomedistinctlimitationson

thegeneralizabilityofmyfindingsregardingtheexactusageproblemsallstudentswillface

wheninteractingwiththeDAT-CROSSSuite.Forexample,oneofthehighlightfindingsfrom

theuseofactivitytheorytoinvestigateresearchquestion3waspreliminaryevidencethat

manystudentshavenaïve,proto-conceptionsaboutcrosscuttingconceptsthathavebuiltup

76

inanevolutionaryreasoningstyle(asdescribedbyOsborneetal.,2018).Theseprimitive

notionsregardingstructuresandtheirfunctionsseempresentinthewordsthatstudents

usetodescribephenomenawheresuchconceptsareusefultowardsmakingmeaningofthe

world,perhapsservingasausefulleverforteachers.However,itisverylikelythatthe“best

case” studentswho participatedwere alsomost likely to attempt to vocalize even their

underdevelopedideaswithoutfearofjudgementbytheirpeers(whowerenotpresentin

the testing setting) or the facilitating adult. Indeed, suchwillingnesswas found in other

investigationsofCCCreasoningwherelikely-affluentpopulationsservedassubjects(Beck

etal.,2016).Incontrast,studentsofcoloroftenstruggletoputforththeirprimitiveideasfor

fear of the teacher-student and student-student interactions that can manifest (Brown,

2004).Itmaybethecaseinfutureuseswithmorevulnerablepopulationsofstudents,that

thesevocalizationsdonotoccur,hinderingeducators’abilitytocapitalizeonthem.

Anotherlimitationisstudents’baselinecontentfamiliarity.Giventherecentnessof

the NGSS (NGSS Lead States, 2013), few students have had any explicit andmeaningful

instruction regarding the crosscutting concepts. Thus, the preliminary evidence for a

learningprogressionseenfromtheresultofresearchquestion2canbebestthoughtofasa

progressionofinformallearninginthatallthelearningexperiencesrelatedtothecontent

and phenomena entailed in the progression have likely only occurred in informal, lived

experiencesettings.Studentsofsimilarlylittleformallearningopportunitiesregardingideas

likesystemsorstructureandfunctionbutofdifferentlivedexperiencesmayprogressalong

averydifferenttrajectory.Itis,therefore,crucialtounderstandmyfindingsonlyasaproof-

of-conceptforassessingCCCs.

77

Chapter6

CONCLUSION

WithintheNGSS, thesalienceofDCIsandSEPsoftenoverwhelmsthevalueof the

CCCs in displaying expected performance. Even among current research in developing

assessment for the NGSS (DeBarger et al., 2015), it is rare to see rubrics that value the

presence of CCCs in student reasoning beyond a binary condition. However, previous

research findings suggest that students’ explanationswere influencednot only bywhich

phenomenatheywereaskedtoexplain,butalsobytheCCCstheywereaskedtouseandthe

wording on the questions they were asked (Weiser et al., 2017). Simultaneously, the

languageoftheNGSSseemstoemphasizestudentsdoingoverstudentsknowing(NGSSLead

States,2013a;NRC,2012).Thescienceeducationcommunityneedsnewassessmentsthat

arebothdoing-orientedandcapableofdrawingevidenceofstudentunderstandingacross

allthreedimensionsofthescience-learningframework(Duschl,2008).Tothisend,activity

theory, when coupled with quantitative metrics, can be a powerful analytic tool in

understanding and improving the kind of work dynamics (Engestrom, 2000) seemingly

desiredbytheNGSS(Duschletal.,2007;NRC,2012).Insettinganovelobservationmodel

forelicitingevidencefromstudents,theATframeworkprovidesteacherandstudentalike

torecognizeopportunitiesforexpressionofthree-dimensionalunderstandingsthatmaynot

beclearlyavailabletothemforpracticalorsocio-culturalreasons(vanEs&Sherin,2002;

Farenga & Joyce, 1997; Jonassen & Rohrer-Murphy, 1999). As I described in previous

sections,activitytheoryfitswellintothedesignofassessments,especiallyonesdesignedto

measuresummativeknowledgeinteractively(Pea,1993)ortoprovideformativelearning

opportunitiestostudents(Black&Wiliam,2009).However,thetaskofaligningsystem-wide

78

assessmentstotheNGSSremains.Ibelievethatactivitytheory,asamethodfordeveloping

evidencemodelsforassessments(Mislevy&Haertel,2007)hasmerit,butworkstillneeds

tobedonetodevelopsuchmodelsatappropriatescaleforthispurpose.

79

References

Abd-El-Khalick,F.,Bell,R.L.,&Lederman,N.G.(1998).Thenatureofscienceandinstructionalpractice:Makingtheunnaturalnatural.ScienceEducation,82(4),417–436.doi:10.1002/(SICI)1098-237X(199807)82:4<417::AID-SCE1>3.0.CO;2-E

Alonzo,A.C.,&Steedle,J.T.(2009).Developingandassessingaforceandmotionlearningprogression.ScienceEducation,93(3),389–421.doi:10.1002/sce.20303

AmericanAssociationfortheAdvancementofScience(AAAS).(1993).Benchmarksforscienceliteracy.AdvancementOfScience.NewYork:OxfordUniversityPress.Retrievedfromhttp://www.project2061.org/publications/bsl/online/index.php

Barab,S.,&Squire,K.(2004).Design-basedresearch:Puttingastakeintheground.JournaloftheLearningSciences,13(1),1–14.doi:10.1207/s15327809jls1301_1

Baxter,G.P.,&Mislevy,R.(2004).TheCaseforanIntegratedDesignFrameworkforAssessingScienceInquiry.LosAngeles:TheNationalCenterforResearchonEvaluation,Standards,StudentTesting(CRESST),CenterforStudiesinEducation,UCLA.

Beck,A.,Bookbinder,A.,Lee,M.J.,&Rivet,A.E.(2017,April).IdentifyingEarlyProductiveSteppingStoneConceptionsofThree-dimensionalEarthScienceUnderstandingbyHighSchoolStudents.PaperpresentatNARST,SanAntonio.

Benson,L.,Elliott,D.,Grant,M.,Holschuh,D.,Kim,B.,Kim,H.,…Reeves,T.C.(2002).Usabilityandinstructionaldesignheuristicsfore-learningevaluation.InP.Barker&S.Rebelsky(Eds.),ED-MEDIA2002--WorldConferenceonEducationalMultimedia,Hypermedia&Telecommunications(pp.1615–1621).Denver,CO:AssociationfortheAdvancementofComputinginEducation(AACE).Retrievedfromhttps://eric.ed.gov/?id=ED477005

Berland,L.K.,&McNeill,K.L.(2010).Alearningprogressionforscientificargumentation:Understandingstudentworkanddesigningsupportiveinstructionalcontexts.ScienceEducation,94(5),765–793.doi:10.1002/sce.20402

Berland,L.K.,&Reiser,B.J.(2009).Makingsenseofargumentationandexplanation.ScienceEducation,93(1),26–55.doi:10.1002/sce.20286

Blumenfeld,P.,Fishman,B.J.,Krajcik,J.,Marx,R.W.,&Soloway,E.(2000).CreatingUsableInnovationsinSystemicReform :ScalingUpTechnology-EmbeddedProject-BasedScienceinUrbanSchools.EducationalPsychologist,35(3),149–164.doi:10.1207/S15326985EP3503

80

Breslyn,W.,McGinnis,J.R.,McDonald,R.C.,&Hestness,E.(2016).Developingalearningprogressionforsealevelrise,amajorimpactofclimatechange.JournalofResearchinScienceTeaching,53(10),1471–1499.doi:10.1002/tea.21333

Bricker,L.A.,&Bell,P.(2008).Conceptualizationsofargumentationfromsciencestudiesandthelearningsciencesandtheirimplicationsforthepracticesofscienceeducation.ScienceEducation,92(3),473–498.doi:10.1002/sce.20278

Briggs,D.C.,&Alonzo,A.C.(2012).Thepsychometricmodelingoforderedmultiple-choiceitemresponsesfordiagnosticassessmentwithalearningprogression.InA.C.Alonzo&A.W.Gotwals(Eds.),LearningProgressionsinScience(pp.293–316).Rotterdam:SensePublishers.doi:10.1007/978-94-6091-824-7_13

Brown,A.L.(1992).DesignExperiments:TheoreticalandMethodologicalChallengesinCreatingComplexInterventionsinClassroomSettings.JournaloftheLearningSciences,2(2),141–178.doi:10.1207/s15327809jls0202_2

Brown,B.A.,Reveles,J.M.,&Kelly,G.J.(2005).Scientificliteracyanddiscursiveidentity:Atheoreticalframeworkforunderstandingsciencelearning.ScienceEducation,89(5),779–802.doi:10.1002/sce.20069

Brown,J.S.,Collins,A.,&Duguid,P.(1989).Situatedcognitionandthecultureoflearning.EducationalResearcher,18(1),32–42.doi:10.3102/0013189X018001032

Clarke,J.,&Dede,C.(2009).DesignforScalability:Acasestudyoftherivercitycurriculum.JournalofScienceEducationandTechnology,18(4),353–365.doi:10.1007/s10956-009-9156-4

Cobb,P.,Zhao,Q.,&Dean,C.(2009).ConductingDesignExperimentstoSupportTeachers’Learning:AReflectionFromtheField.JournaloftheLearningSciences,18(2),165–199.doi:10.1080/10508400902797933

CognitionandTechnologyGroupatVanderbiltUniversity.(1992).Anchoredinstructioninscienceandmathematics:Theoreticalbases,developmentalprojects,andinitialresearchfindings.InR.A.Duschl&R.J.Hamilton(Eds.),Philosophyofscience,Cognitivepsychology,andEducationaltheoryandpractice(pp.244–273).NewYork:SUNYPress.

Cohen,E.(1998).Makingcooperativelearningequitable.EducationalLeadership,56(1),18–21.Retrievedfromhttp://web.b.ebscohost.com.ezproxy.elib10.ub.unimaas.nl/ehost/pdfviewer/pdfviewer?sid=d7594b1a-abc6-4dfd-9d76-a2cf6bc85224@sessionmgr115&vid=1&hid=101

81

Cohen,E.G.,&Lotan,R.A.(1995).ProducingEqual-StatusInteractionintheHeterogeneousClassroom.AmericanEducationalResearchJournal,32(1),99–120.doi:10.3102/00028312032001099

Corcoran,T.B.,Mosher,F.A.,&Rogat,A.(2009).LearningProgressionsinScience:AnEvidence-BasedApproachtoReformLearningProgressionsinScience:AnEvidence-BasedApproachtoReform.CPREResearchReports.Retrievedfromhttp://repository.upenn.edu/cpre_researchreports

Creswell,J.W.(2009).QualitativeInquiryandResearchDesign:ChoosingAmongFiveApproaches.Book(3rded.,Vol.2nded).LosAngeles:SAGEPublications.doi:10.1177/1524839915580941

Cronbach,L.J.(1951).Coefficientalphaandtheinternalstructureoftests.Psychometrika,16,297-334.

deAyala,R.(2010).Itemresponsetheory.InG.R.Hancock,R.O.Mueller,&L.M.Stapleton(Eds.),TheReviewer’sGuidetoQuantitativeMethodsintheSocialSciences(p.155).NewYork,NY:Routledge.

DeBarger,A.H.,Penuel,W.R.,Harris,C.J.,&Kennedy,C.A.(2016).BuildinganassessmentargumenttodesignanduseNextGenerationScienceassessmentsinefficacystudiesofcurriculuminterventions.AmericanJournalofEvaluation,37(2),174–192.doi:10.1177/1098214015581707

DeBoer,G.E.(1991).Ahistoryofideasinscienceeducation:Implicationsforpractice.NewYork:TeachersCollegePress.

DiSessa,A.A.,Cobb,P.,&Andrea,A.(2004).Ontologicalinnovationandtheroleoftheoryindesignexperiments.TheJournaloftheLearningSciences,13(1),77–103.doi:DOI10.1207/s15327809jls1301_4

Duncan,R.G.,&Rivet,A.E.(2013).ScienceLearningProgressions.Science,339(6118),396–397.doi:10.1126/science.1228692

Duncan,R.G.,&Tseng,K.A.(2011).Designingproject-basedinstructiontofostergenerativeandmechanisticunderstandingsingenetics.ScienceEducation,95(1),21–56.doi:10.1002/sce.20407

Duschl,R.(2008).ScienceEducationinThree-PartHarmony:BalancingConceptual,Epistemic,andSocialLearningGoals.ReviewofResearchinEducation,32(1),268–291.doi:10.3102/0091732X07309371

Duschl,R.(2012).Theseconddimension—Crosscuttingconcepts.ScienceandChildren,79(2),10–14.

82

Duschl,R.,Maeng,S.,&Sezen,A.(2011).Learningprogressionsandteachingsequences:areviewandanalysis.StudiesinScienceEducation,47(2),123–182.doi:10.1080/03057267.2011.604476

Duschl,R.,Schweingruber,H.A.,&Shouse,A.W.(2007).Takingsciencetoschool:LearningandteachingScienceingradesK-8.TakingSciencetoSchool.Washington,D.C.:NationalAcademiesPress.doi:10.17226/11625

Edelson,D.C.(2001).Learning-for-Use:Aframeworkforthedesignoftechnology-supportedinquiryactivities.JournalofResearchinScienceTeaching,38(3),355–385.doi:10.1002/1098-2736(200103)38:3<355::AID-TEA1010>3.0.CO;2-M

Eger,M.(1992).Hermeneuticsandscienceeducation:Anintroduction.ScienceandEducation,1(4),337–348.doi:10.1007/BF00430961

Eggert,S.,&Bögeholz,S.(2010).Students’useofdecision-makingstrategieswithregardtosocioscientificIssues:Anapplicationoftheraschpartialcreditmodel.ScienceEducation,94(2),230–258.doi:10.1002/sce.20358

Eisenhart,M.,Finkel,E.,&Marion,S.F.(1996).Creatingtheconditionsforscientificliteracy:Are-examination.AmericanEducationalResearchJournal,33(2),261–295.doi:10.3102/00028312033002261

Elam-Respass,T.(2018).EffectiveInstructionalPracticesthatEngagetheAfricanAmericanMaleinMiddleSchoolScience.UniversityofMaryland,CollegePark.

Engestrom,Y.(2000).Activitytheoryasaframeworkforanalyzingandredesigningwork.Ergonomics,43(7),960–974.doi:10.1080/001401300409143

Farenga,S.J.,&Joyce,B.A.(1997).WhatChildrenBringtotheClassroom:LearningScienceFromExperience.SchoolScienceandMathematics,97(5),248–252.doi:10.1111/j.1949-8594.1997.tb17270.x

Ford,M.(2008).Disciplinaryauthorityandaccountabilityinscientificpracticeandlearning.ScienceEducation,92(3),404–423.doi:10.1002/sce.20263

Freire,P.,&Macedo,D.P.(1995).Adialogue:Culture,langagueandrace.HarvardEducationalReview,65(3),377–403.doi:10.17763/haer.65.3.12g1923330p1xhj8

Furtak,E.M.(2017).Confrontingdilemmasposedbythree-dimensionalclassroomassessment:IntroductiontoavirtualissueofScienceEducation.ScienceEducation.doi:10.1002/sce.21283

Garver,M.andMentzer,J.(1999).Logisticsresearchmethod:Employingstructuralequationmodelingtotestforconstructvalidity.JournalofBusinessLogistics,20(1),33.https://doi.org/10.1002/(ISSN)2158-1592

83

Gorin,J.S.,&Mislevy,R.J.(2013).InherentmeasurementchallengesintheNextGenerationScienceStandardsforbothformativeandsummativeassessment.InInvitationalAssessmentSymposium(pp.2–39).Washington,D.C.

Gotwals,A.W.,&Songer,N.B.(2013).Validityevidenceforlearningprogression-basedassessmentitemsthatfusecoredisciplinaryideasandsciencepractices.JournalofResearchinScienceTeaching,50(5),597–626.doi:10.1002/tea.21083

Griffiths,A.K.,&Grant,B.A.C.(1985).Highschoolstudents’understandingoffoodwebs:Identificationofalearninghierarchyandrelatedmisconceptions.JournalofResearchinScienceTeaching,22(5),421–436.doi:10.1002/tea.3660220505

Gunckel,K.L.,Covitt,B.A.,Salinas,I.,&Anderson,C.W.(2012).Alearningprogressionforwaterinsocio-ecologicalsystems.JournalofResearchinScienceTeaching.doi:10.1002/tea.21024

Hallgren,K.A.(2012).Computinginter-raterreliabilityforobservationaldata:anoverviewandtutorial.TutorialsinQuantitativeMethodsforPsychology,8(1),23–34.https://doi.org/10.1016/j.biotechadv.2011.08.021.Secreted

Hammer,D.(1994).Epistemologicalbeliefsinintroductoryphysics.CognitionandInstruction,12(2),151–183.doi:10.1207/s1532690xci1202_4

Harris,C.J.,Krajcik,J.S.,Pellegrino,J.W.,&McElhaney,K.W.(2016).Constructingassessmenttasksthatblenddisciplinarycoreideas,crosscuttingconcepts,andsciencepracticesforclassroomformativeapplications.MenloPark,CA:SRIInternational.

Hewson,P.W.,&Hewson,M.G.A.B.(1984).Theroleofconceptualconflictinconceptualchangeandthedesignofscienceinstruction.InstructionalScience,13(1),1–13.doi:10.1007/BF00051837

Hmelo-Silver,C.E.,Liu,L.,&Jordan,R.(2009).Visualrepresentationofamultidimensionalcodingschemeforunderstandingtechnology-mediatedlearningaboutcomplexnaturalsystems.ResearchandPracticeinTechnologyEnhancedLearning,04(03),253–280.https://doi.org/10.1142/S1793206809000714

Hoadley,C.M.(2004).MethodologicalAlignmentinDesign-BasedResearch.EducationalPsychologist,39(4),203–212.doi:10.1207/s15326985ep3904

Hobson,A.(1993).OzoneandInterdisciplinaryScienceTeaching-LearningtoAddresstheThingsthatCountMost.JournalofCollegeScienceTeaching,23(1),33–37.

Hogan,K.(2000).Assessingstudents’systemsreasoninginecology.JournalofBiologicalEducation,35(1),22–28.doi:10.1080/00219266.2000.9655731

84

Hogan,K.,&Fisherkeller,J.(1996).Representingstudents’thinkingaboutnutrientcyclinginecosystems:Bidimensionalcodingofacomplextopic.JournalofResearchinScienceTeaching,33(9),941–970.doi:10.1002/(SICI)1098-2736(199611)33:9<941::AID-TEA1>3.0.CO;2-V

Hokayem,H.,Ma,J.,&Jin,H.(2015).Alearningprogressionforfeedbackloopreasoningatlowerelementarylevel.JournalofBiologicalEducation,49(3),246–260.doi:10.1080/00219266.2014.943789

Howell,H.,Phelps,G.,Croft,A.J.,Kirui,D.,&Gitomer,D.(2013).Cognitiveinterviewsasatoolforinvestigatingthevalidityofcontentknowledgeforteachingassessments.ETSResearchReportSeries,2013(1),i-97.https://doi.org/10.1002/j.2333-8504.2013.tb02326.x

Jin,H.,&Anderson,C.W.(2012).Alearningprogressionforenergyinsocio-ecologicalsystems.JournalofResearchinScienceTeaching.doi:10.1002/tea.21051

Joe,J.,Tocci,C.,Holtzman,S.,&Williams,J.(2013).Foundationsofobservation:Considerationsfordevelopingaclassroomobservationsystlemthathelpsdistrictsachieveconsistentandaccuratescores.METProjectPolicyandPracticeBrief:BillandMelindaGatesFoundation.Princeton,NJ.Retrievedfromhttp://www.metproject.org/downloads/MET-ETS_Foundations_of_Observation.pdf

Johnstone,C.J.,Bottsford-Miller,N.a.,&Thompson,S.J.(2006).Usingthethinkaloudmethod(cognitivelabs)toevaluatetestdesignforstudentswithdisabilitiesandenglishlanguagelearners.TechnicalReport44UniversityofMinnesota,NationalCenteronEducationalOutcomes.Retrievedfromhttp://www.cehd.umn.edu/nceo/OnlinePubs/Tech44/%5Cnhttp://education.umn.edu/NCEO/OnlinePubs/Tech44/

Jonassen,D.H.,Rohrer-Murphy,L.,&JonassenH.,D.(1999).Activitytheoryasaframeworkfordesigningconstructivistlearningenvironments.EducationalTechnologyResearchandDevelopment,47(1),61–79.doi:10.1007/BF02299477

Joseph,D.(2004).Thepracticeofdesign-basedresearch:uncoveringtheinterplaybetweendesign,research,andthereal-worldcontext.EducationalPsychologist,39(4),235–242.https://doi.org/10.1207/s15326985ep3904_5

Kanter,D.E.(2010).Doingtheprojectandlearningthecontent:Designingproject-basedsciencecurriculaformeaningfulunderstanding.ScienceEducation,94(3),525–551.doi:10.1002/sce.20381

Kelly,G.J.,McDonald,S.,&Wickman,P.-O.(2012).Sciencelearningandepistemology.InB.J.Fraser,K.Tobin,&C.J.McRobbie(Eds.),SecondInternationalHandbookof

85

ScienceEducation(pp.281–291).Dordrecht:SpringerNetherlands.doi:10.1007/978-1-4020-9041-7

Kerby,D.S.(2014).Thesimpledifferenceformula:anapproachtoteachingnonparametriccorrelation.ComprehensivePsychology,3(1),11.IT.3.1.https://doi.org/10.2466/11.IT.3.1

Kesidou,S.,&Roseman,J.E.(2002).Howwelldomiddleschoolscienceprogramsmeasureup?FindingsfromProject2061’scurriculumreview.JournalofResearchinScienceTeaching,39(6),522–549.doi:10.1002/tea.10035

Kilinc,A.,Demiral,U.,&Kartal,T.(2017).ResistancetodialogicdiscourseinSSIteaching:Theeffectsofanargumentation-basedworkshop,teachingpracticum,andinductiononapreservicescienceteacher.JournalofResearchinScienceTeaching,54(6),764–789.doi:10.1002/tea.21385

Kline,P.(1994).FactorAnalysisinTestConstruction.InAnEasyGuidetoFactorAnalysis(pp.125–139).NewYork:Routledge.

Krajcik,J.(2015).Three-dimensionalinstruction:Usinganewtypeofteachinginthescienceclassroom.TheScienceTeacher,82(8).

Krajcik,J.,&Merritt,J.(2012).EngagingStudentsinScientificPractices:Whatdoesconstructingandrevisingmodelslooklikeinthescienceclassroom?ScienceandChildren,49(7),10.

Krajcik,J.,Codere,S.,Dahsah,C.,Bayer,R.,&Mun,K.(2014).PlanninginstructiontomeettheintentoftheNextGenerationScienceStandards.JournalofScienceTeacherEducation,25(2),157–175.doi:10.1007/s10972-014-9383-2

Krajcik,J.,McNeill,K.L.,&Reiser,B.J.(2008).Learning-goals-drivendesignmodel:Developingcurriculummaterialsthatalignwithnationalstandardsandincorporateproject-basedpedagogy.ScienceEducation,92(1),1–32.doi:10.1002/sce.20240

Lakoff,G.,&Johnson,M.(1997).MetaphorsWeLiveBy.Theproductionofreality:essaysandreadingonsocialinteraction(2nded.).Chicago:UniversityofChicagoPress.

Lave,J.,&Wenger,E.(1991).Situatedlearning:Legitimateperipheralparticipation.NewYork:CambridgeUniversityPress.Retrievedfromhttps://books.google.co.uk/books?id=ZVogAwAAQBAJ&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false

Lederman,N.G.,&Lederman,J.S.(2014).Isnatureofsciencegoing,going,going,gone?JournalofScienceTeacherEducation,25(3),235–238.doi:10.1007/s10972-014-9386-z

86

Lederman,N.G.,&Zeidler,D.L.(1987).ScienceTeachers’ConceptionsoftheNatureofScience:DoTheyReallyInfluenceTeachingBehavior?ScienceEducation,71(5),721–734.doi:10.1002/sce.3730710509

Lee,H.S.,&Liu,O.L.(2010).Assessinglearningprogressionofenergyconceptsacrossmiddleschoolgrades:Theknowledgeintegrationperspective.ScienceEducation,94(4),665–688.doi:10.1002/sce.20382

Lee,O.(1997).Guesteditorial:Scientificliteracyforall:Whatisit,andhowcanweachieveit?JournalofResearchinScienceTeaching,34(3),219–222.doi:10.1002/(SICI)1098-2736(199703)34:3<219::AID-TEA1>3.0.CO;2-V

Lee,O.,Quinn,H.,&Valdes,G.(2013).ScienceandlanguageforEnglishlanguagelearnersinrelationtoNextGenerationScienceStandardsandwithimplicationsforCommonCorestatestandardsforEnglishlanguageartsandmathematics.EducationalResearcher,42(4),223–233.doi:10.3102/0013189X13480524

Lehrer,R.,&Schauble,L.(2006).CultivatingmodelBasedreasoninginscienceeducation.InR.K.Sawyer(Ed.),TheCambridgehandbookofthelearningsciences(pp.371–387).Cambridge,UK:CambridgeUniversityPress.

Lin,C.,&Hu,R.(2003).Students’understandingofenergyflowandmattercyclinginthecontextofthefoodchain,photosynthesis,andrespiration.InternationalJournalofScienceEducation,25(1loving2),1529–1544.doi:10.1080/0950069032000052045

Liu,L.,Rogat,A.,&Bertling,M.(2013).ACBALTMScienceModelofCognition:DevelopingaCompetencyModelandLearningProgressionstoSupportAssessmentDevelopment.

Maclure,M.,&Willett,W.C.(1987).Misinterpretationandmisuseofthekappastatistic.AmericanJournalofEpidemiology,126(2),161–169.https://doi.org/10.1093/aje/126.2.161

McComas,W.F.,Clough,M.P.,&Almazroa,H.(2002).Theroleandcharacterofthenatureofscienceinscienceeducation.InW.F.McComas(Ed.),Thenatureofscienceinscienceeducation:Rationaleandstrategies(pp.3–39).Dordrecht:KluwerAcademicPublishers.doi:10.1007/0-306-47215-5_1

Mislevy,R.J.(2016).Howdevelopmentsinpsychologyandtechnologychallengevalidityargumentation.JournalofEducationalMeasurement,53(3),265–292.https://doi.org/10.1111/jedm.12117

Mislevy,R.J.(2017).Resolvingtheparadoxofrichperformancetasks.InTestFairnessintheNewGenerationofLarge-ScaleAssessment(pp.1–46).Charlotte,N.C.:InformationAgePublishers.

87

Mislevy,R.J.,&Haertel,G.D.(2006).Implicationsofevidence-centereddesignforeducationaltesting.EducationalMeasurement:IssuesandPractice,25(4),6–20.doi:10.1111/j.1745-3992.2006.00075.x

Mislevy,R.J.,Haertel,G.,Riconscente,M.,WiseRutstein,D.,&Ziker,C.(2017).Assessingmodel-basedreasoningusingevidence-centereddesign:Asuiteofresearch-baseddesignpatterns.Cham,Switzerland:SpringerInternationalPublishing.doi:10.1007/978-3-319-52246-3

Mislevy,R.J.,Riconscente,M.M.,&Rutstein,D.W.(2009).Designpatternsforassessingmodel-basedreasoning(Large-ScaleAssessmentTechnicalReport6).MenloPark,CA:SRIInternational.

Mohan,L.,Chen,J.,&Anderson,C.W.(2009).Developingamulti-yearlearningprogressionforcarboncyclinginsocio-ecologicalsystems.JournalofResearchinScienceTeaching,46(6),675–698.doi:10.1002/tea.20314

Morin,O.,Simonneaux,L.,&Tytler,R.(2017).Engagingwithsociallyacutequestions:Developmentandvalidationofaninteractionalreasoningframework.JournalofResearchinScienceTeaching,54(7),825–851.doi:10.1002/tea.21386

Mueller,R.O.,&Hancock,G.R.(2015).Factoranalysisandlatentstructureanalysis:confirmatoryfactoranalysis.InJ.D.Wright(Ed.),InternationalEncyclopediaoftheSocial&BehavioralSciences(SecondEdi,pp.686–690).Elsevier.https://doi.org/10.1016/B978-0-08-097086-8.25009-5

Mutegi,J.(2011).Theinadequaciesofscience-for-allandthenecessityandnatureofasociallytransformativecurriculumapproachforAfricanAmericanscienceeducation.JournalofResearchinScienceTeaching,248(3),301–316.

Nardi,B.A.(1996).Studyingcontext:AcomparisonofActivityTheory,SituatedActionModels,andDistributedCognition.InB.A.Nardi(Ed.),ContextandConciousness:ActivityTheoryandHumanComputerInteraction(pp.69–102).Cambridge,MA:TheMITPress.doi:citeulike-article-id:1292510

NationalResearchCouncil.(1996).NationalScienceEducationStandards.Washington,D.C.:NationalAcademiesPress.doi:10.17226/4962

NationalResearchCouncil.(2011).SuccessfulK-12STEMEducation.Washington,D.C.:NationalAcademiesPress.

NationalResearchCouncil.(2012).AframeworkforK-12scienceeducation:Practices,crosscuttingconcepts,andcoreideas.Washington,D.C.:NationalAcademiesPress.Retrievedfromhttp://www.nap.edu/catalog/13165/a-framework-for-k12-science-education-practices-crosscutting-concepts-and

88

NationalResearchCouncil.(2013).MonitoringprogresstowardsuccessfulK-12STEMeducation:Anationadvancing?NationalAcademiesPress.Washington,D.C.:NationalAcademiesPress.doi:10.17226/13509

NationalResearchCouncil.(2014).DevelopingAssessmentsfortheNextGenerationScienceStandards.Washington,D.C.:NationalAcademiesPress.doi:10.17226/18409

Nersessian,N.J.(2002).Thecognitivebasisofmodel-basedreasoninginscience.InP.Carruthers,S.Stich,&M.Siegal(Eds.),TheCognitiveBasisofScience(pp.133–153).NewYork:CambridgeUniversityPress.doi:10.1017/cbo9780511613517.008

Neumann,K.,Viering,T.,Boone,W.J.,&Fischer,H.E.(2013).Towardsalearningprogressionofenergy.JournalofResearchinScienceTeaching,50(2),162–188.doi:10.1002/tea.21061

NGSSLeadStates.(2013a).NextGenerationScienceStandards:ForStates,ByStates.Washington,DC:TheNationalAcademiesPress.

NGSSLeadStates.(2013b).NextGenerationScienceStandards:ForStates,ByStates(AppendixE).Washington,DC:TheNationalAcademiesPress.

NGSSLeadStates.(2013c).NextGenerationScienceStandards:ForStates,ByStates(AppendixG).Washington,DC:TheNationalAcademiesPress.

Noble,T.,Suarez,C.,Rosebery,A.,O’Connor,M.C.,Warren,B.,&Hudicourt-Barnes,J.(2012).“Ineverthoughtofitasfreezing”:Howstudentsanswerquestionsonlarge-scalesciencetestsandwhattheyknowaboutscience.JournalofResearchinScienceTeaching,49(6),778–803.doi:10.1002/tea.21026

Nussbaumer,D.(2012).Anoverviewofculturalhistoricalactivitytheory(chat)useinclassroomresearch2000to2009.EducationalReview,64(1),37–55.https://doi.org/10.1080/00131911.2011.553947

O’Brien,H.L.,&Toms,E.G.(2008).Whatisuserengagement?Aconceptualframeworkfordefininguserengagementwithtechnology.JournaloftheAmericanSocietyforInformationScienceandTechnology,59(6),938–955.https://doi.org/10.1002/asi.20801.1

Osborne,J.(2014).Teachingscientificpractices:Meetingthechallengeofchange.JournalofScienceTeacherEducation,25(2),177–196.doi:10.1007/s10972-014-9384-1

Osborne,J.F.,Henderson,J.B.,MacPherson,A.,Szu,E.,Wild,A.,&Yao,S.Y.(2016).Thedevelopmentandvalidationofalearningprogressionforargumentationin

89

science.JournalofResearchinScienceTeaching,53(6),821–846.doi:10.1002/tea.21316

Osborne,J.,Rafanelli,S.,&Kind,P.(2018).Towardamorecoherentmodelforscienceeducationthanthecrosscuttingconceptsofthenextgenerationsciencestandards:theaffordancesofstylesofreasoning.JournalofResearchinScienceTeaching,55(7),962–981.https://doi.org/10.1002/tea.21460

Pellegrino,J.W.(2013).ProficiencyinScience:AssessmentChallengesandOpportunities.Science,340(6130),320–323.doi:10.1126/science.1232065

Pellegrino,W.,Chudowsky,N.,&Glaser,R.(2001).Thenatureofassessmentandreasoningfromevidence.InKnowingWhatStudentsKnow:TheScienceandDesignofEducationalAssessment(pp.37–54).Washington,D.C.:NationalAcademiesPress.Retrievedfromhttp://books.google.com/books?hl=en&lr=&id=t5mcAgAAQBAJ&pgis=1

Peterson,C.H.,Peterson,N.A.,&Powell,K.G.(2017).Cognitiveinterviewingforitemdevelopment:validityevidencebasedoncontentandresponseprocesses.MeasurementandEvaluationinCounselingandDevelopment,50(4),217–223.https://doi.org/10.1080/07481756.2017.1339564

Plummer,J.D.,&Krajcik,J.(2010).Buildingalearningprogressionforcelestialmotion:Elementarylevelsfromanearth-basedperspective.JournalofResearchinScienceTeaching,47(7),768–787.doi:10.1002/tea.20355

Plummer,J.D.,Palma,C.,Flarend,A.,Rubin,K.,Ong,Y.S.,Botzer,B.,…Furman,T.(2015).Developmentofalearningprogressionfortheformationofthesolarsystem.InternationalJournalofScienceEducation,37(9),1381–1401.doi:10.1080/09500693.2015.1036386

Pluta,W.J.,Chinn,C.A.,&Duncan,R.G.(2011).Learners’epistemiccriteriaforgoodscientificmodels.JournalofResearchinScienceTeaching,48(5),486–511.doi:10.1002/tea.20415

Quellmalz,E.S.,&Pellegrino,J.W.(2009).TechnologyandTesting.Science,323(5910),75–79.doi:10.1126/science.1168046

Ríordáin,M.N.,Johnston,J.,&Walshe,G.(2015).Makingmathematicsandscienceintegrationhappen:keyaspectsofpractice.InternationalJournalofMathematicalEducationinScienceandTechnology,47(September),1–23.doi:10.1080/0020739X.2015.1078001

Rivet,A.E.,&Kastens,K.A.(2012).Developingaconstruct-basedassessmenttoexaminestudents’analogicalreasoningaroundphysicalmodelsinEarthScience.JournalofResearchinScienceTeaching,49(6),713–743.doi:10.1002/tea.21029

90

Rivet,A.E.,&Krajcik,J.S.(2004).AchievingstandardsinUrbansystemicreform:Anexampleofasixthgradeproject-basedsciencecurriculum.JournalofResearchinScienceTeaching,41(7),669–692.doi:10.1002/tea.20021

Rivet,A.E.,&Krajcik,J.S.(2008).Contextualizinginstruction:Leveragingstudents’priorknowledgeandexperiencestofosterunderstandingofmiddleschoolscience.JournalofResearchinScienceTeaching,45(1),79–100.doi:10.1002/tea.20203

Rivet,A.E.,Weiser,G.,Lyu,X.,Li,Y.,&Rojas-Perilla,D.(2016).Whatarecrosscuttingconceptsinscience?Fourmetaphoricalperspectives.InC.K.Looi,J.L.Polman,U.Cress,&P.Reimann(Eds.),ProceedingsofInternationalConferenceoftheLearningSciences,ICLS(Vol.2).Signapore:InternationalSocietyoftheLearningSciences.doi:10.22318/icls2016.149

Rodriguez, A. J. (2015).What about a dimension of engagement, equity, and diversitypractices?Acritiqueofthenextgenerationsciencestandards.JournalofResearchinScienceTeaching,52(7),1031–1051.https://doi.org/10.1002/tea.21232

Roseman,J.,Kesidou,S.,&Stern,L.(1996).Identifyingcurriculummaterialsforscienceliteracy:AProject2061evaluationtool.RetrievedNovember27,2014,fromhttp://www.project2061.org/publications/articles/roseman/roseman2.htm

Roth,W.-M.,&Tobin,K.(2002).Redesigningan“Urban”TeacherEducationProgram:AnActivityTheoryPerspective.Mind,Culture,andActivity,9(2),108–131.doi:10.1207/S15327884MCA0902

Rutherford,F.J.,&Ahlgren,A.(1990).ScienceforAllAmericans.NewYork:OxfordUniversityPress.

Sandoval,W.A.(2005).Understandingstudents’practicalepistemologiesandtheirinfluenceonlearningthroughinquiry.ScienceEducation,89(4),634–656.doi:10.1002/sce.20065

Schwarz,C.(2009).Developingpreserviceelementaryteachers’knowledgeandpracticesthroughmodeling-centeredscientificinquiry.ScienceEducation,93(4),720–744.doi:10.1002/sce.20324

Schwarz,C.V.,&White,B.Y.(2005).MetamodelingKnowledge:DevelopingStudents’UnderstandingofScientificModeling.CognitionandInstruction,23(2),165–205.doi:10.1207/s1532690xci2302_1

Schwarz,C.V.,Reiser,B.J.,Davis,E.A.,Kenyon,L.,Achér,A.,Fortus,D.,...&Krajcik,J.(2009).Developingalearningprogressionforscientificmodeling:Makingscientificmodelingaccessibleandmeaningfulforlearners.JournalofResearchinScienceTeaching,46(6),632-654.

91

Shavelson,R.J.,Phillips,D.C.,Towne,L.,&Feuer,M.J.(2003).OntheScienceofEducationDesignStudies.EducationalResearcher,32(1),25–28.doi:10.3102/0013189X032001025

Snow,E.,&Katz,I.R.(2009).UsingcognitiveinterviewstovalidateaninterpretiveargumentfortheETSiSkillsTMassessment.CommunicationsinInformationLiteracy,3(2),99–127.

Songer,N.B.,Kelcey,B.,&Gotwals,A.W.(2009).Howandwhendoescomplexreasoningoccur?Empiricallydrivendevelopmentofalearningprogressionfocusedoncomplexreasoningaboutbiodiversity.JournalofResearchinScienceTeaching,46(6),610–631.doi:10.1002/tea.20313

Songer,N.B.,Lee,H.-S.,&McDonald,S.(2003).Researchtowardsanexpandedunderstandingofinquirysciencebeyondoneidealizedstandard.ScienceEducation,87(4),490–516.doi:10.1002/sce.10085

Sparks,J.R.,&Deane,P.(2015).CognitivelyBasedAssessmentofResearchandInquirySkills:DefiningaKeyPracticeintheEnglishLanguageArts.ETSResearchReportSeries,2015(2),1–55.doi:10.1002/ets2.12082

Stevens,S.Y.,Delgado,C.,&Krajcik,J.S.(2009).Developingahypotheticalmulti-dimensionallearningprogressionforthenatureofmatter.JournalofResearchinScienceTeaching,47(6),687–715.doi:10.1002/tea.20324

Stroupe,D.(2015).Describing“SciencePractice”inlearningsettings.ScienceEducation,99(6),1033–1040.doi:10.1002/sce.21191

Svoboda,J.,&Passmore,C.(2011).TheStrategiesofModelinginBiologyEducation.Science&Education,22(1),119–142.doi:10.1007/s11191-011-9425-5

vanEs,E.A.,&Sherin,M.G.(2002).LearningtoNotice:ScaffoldingNewTeachers’InterpretationsofClassroomInteractions.JournalofTechnologyandTeacherEducation,10(4),571–596.

Weiser,G.,&Liu,L.(2018,March).Adesignframeworkforthedevelopmentofscenario-basedassessmentsforsummativeassessmentsettings.PosterpaperpresentatNARST,Atlanta.

Weiser,G.,Lyu,X.,&Rojas-Perilla,D.F.(2017,April).WhatAreCrosscuttingConceptsinScience?FourMetaphoricalPerspectives.PaperpresentatNARST,SanAntonio.

Wiliam,D.(2010).Whatcountsasevidenceofeducationalachievement?Theroleofconstructsinthepursuitofequityinassessment,.ReviewofResearchinEducation,34(1),254–284.doi:10.3102/0091732X09351544

92

Wilson,M.,&Draney,K.(2005).Somelinksbetweenlarge-scaleandclassroomassessments:ThecaseoftheBEARassessmentsystem.YearbookoftheNationalSocietyfortheStudyofEducation,103(2),132–154.doi:10.1111/j.1744-7984.2004.tb00051.x

Yilmaz,S.,Eryilmaz,A.,&Geban,O.(2006).Assessingtheimpactofbridginganalogiesinmechanics.SchoolScienceandMathematics,106(6),220–230.doi:10.1111/j.1949-8594.2006.tb17911.x

Yoon,S.A.(2008).Anevolutionaryapproachtoharnessingcomplexsystemsthinkinginthescienceandtechnologyclassroom.InternationalJournalofScienceEducation,30(1),1–32.doi:10.1080/09500690601101672

Yoon,S.A.,Goh,S.-E.,&Park,M.(2018).TeachingandlearningaboutcomplexsystemsinK–12scienceeducation:Areviewofempiricalstudies1995–2015.ReviewofEducationalResearch,88(2),285–325.doi:10.3102/0034654317746090

Yore,L.,Bisanz,G.L.,&Hand,B.M.(2003).Examiningtheliteracycomponentofscienceliteracy:25yearsoflanguageartsandscienceresearch.InternationalJournalofScienceEducation,25(6),689–725.doi:10.1080/09500690305018

Zaharias,P.,&Poylymenakou,A.(2009).Developingausabilityevaluationmethodfore-learningapplications:Beyondfunctionalusability.InternationalJournalofHuman-ComputerInteraction,25(1),75–98.doi:10.1080/10447310802546716

93

APPENDICES

AppendixA:TaskModelforAssessmentofEcologicalSystems,Structures,andFunctions–AdaptedfromMS-LS2-3,MS-LS2-4,&MS-LS2-5

PerformanceExpectationsandTheirDimensions

Studentswhodemonstrateunderstandingcan:MS-LS2-3(Original) Developamodel todescribe the cyclingofmatter and flowof

energy among living and nonliving parts of an ecosystem.[Clarification Statement: Emphasis is on describing theconservationofmatterandflowofenergyintoandoutofvariousecosystems, and on defining the boundaries of the system.][AssessmentBoundary:Assessmentdoesnotincludetheuseofchemicalreactionstodescribetheprocesses.]

MS-LS2-3(Adapted) Develop a model to describe relationships among living andnonlivingpartsofanecosystemthat iscapableof tracking thecycling of matter and flow of energy among ecosystemcomponents.

MS-LS2-4(Original) Construct an argument supported by empirical evidence thatchanges tophysical or biological components of an ecosystemaffect populations. [Clarification Statement: Emphasis is onrecognizingpatterns indataandmakingwarranted inferencesabout changes in populations, and on evaluating empiricalevidencesupportingargumentsaboutchangestoecosystems.]

MS-LS-4(Adapted) Construct an argument supported by empirical evidence thatchangesthefunctioningofanecosystem(includingitsabilitytosupport population growth) can be driven by changes to thephysicalorbiologicalcomponentsoftheecosystem.

MS-LS2-5(Original) Evaluate competing design solutions for maintainingbiodiversity and ecosystem services. [Clarification Statement:Examplesofecosystemservicescouldincludewaterpurification,nutrient recycling, andpreventionof soil erosion.Examplesofdesign solution constraints could include scientific, economic,andsocialconsiderations.]

MS-LS2-5(Adapted) Evaluatecompetingdesignsolutionsforachievingsomehumanwant based on criteria that considermaintaining biodiversityandecosystemservices.

ScienceandEngineeringPractices

DisciplinaryCoreIdeas CrosscuttingConcepts

94

Developing and UsingModelsModelingin6–8buildsonK–5experiencesandprogressestodeveloping,using,andrevisingmodels to describe, test, andpredict more abstractphenomena and designsystems.

• Develop a model todescribe phenomena.(MS-LS2-3)

Engaging in Argument fromEvidenceEngaging in argument fromevidencein6–8buildsonK–5experiencesandprogressestoconstructing a convincingargument that supports orrefutes claims for eitherexplanations or solutionsaboutthenaturalanddesignedworld(s).

• Construct an oral andwritten argumentsupportedbyempiricalevidenceandscientificreasoning to supportor refute anexplanationoramodelforaphenomenonorasolution to a problem.(MS-LS2-4)

• Evaluate competingdesign solutionsbasedon jointly developedand agreed upondesign criteria. (MS-LS2-5)

LS2.B: Cycle of Matter andEnergy Transfer inEcosystems

• Foodwebsaremodelsthat demonstrate howmatter and energy istransferred betweenproducers, consumers,and decomposers asthe three groupsinteract within anecosystem. Transfersofmatter into and outof the physicalenvironment occur atevery level.Decomposers recyclenutrients from deadplantoranimalmatterback to the soil interrestrialenvironmentsortothewater in aquaticenvironments. Theatomsthatmakeuptheorganisms in anecosystem are cycledrepeatedly betweenthelivingandnonlivingpartsoftheecosystem.

LS2.C: Ecosystem Dynamics,Functioning,andResilience

• Ecosystems aredynamic in nature;their characteristicscan vary over time.Disruptions to anyphysical or biologicalcomponent of anecosystem can lead toshifts in all itspopulations.

SystemsandSystemsModels(Adaptation):

• Representing flows ofmatter and energywithin a larger,complexsystemcanbeaccomplished bymodeling flows withinand among relevantsub-systems. (MS-LS2-3)

• Models canbeused torepresent the impactsof changes in a sub-system on the largersystem particularlywhen evidence canonly be collected fromlimited aspects of thebroader phenomenon.(Might be adaptableintoStructure/Function ifwe add ESS2.A orESS2.CDCIs) (MS-LS2-4)

Structure and Function(Adaptation):

• The structure ofengineering solutionsare designed to serveparticular functionsthrough the physicalproperties of thematerialsusedandtheenvironment thesolution inhabits (MS-LS2-5)

EnergyandMatter(Original)• The transfer of energy

can be tracked asenergyflowsthroughanaturalsystem.

95

• Biodiversity describesthe variety of speciesfound in Earth’sterrestrial and oceanicecosystems. Thecompleteness orintegrity of anecosystem’sbiodiversity is oftenusedasameasureofitshealth.

LS4.D: Biodiversity andHumans

• Changesinbiodiversitycan influence humans’resources,suchasfood,energy,andmedicines,as well as ecosystemservices that humansrely on—for example,water purification andrecycling.(secondary)

ETS1.B:DevelopingPossibleSolutions

• There are systematicprocesses forevaluating solutionswith respect to howwell they meet thecriteriaandconstraintsof a problem.(secondary)

Stability and Change(Original)

• Small changes in onepartofasystemmightcause large changes inanotherpart.

Practice-orientedFocal KSAs (Note:“ability” here meanscapabilitytoreasonasdescribedaboutgivenDCIs and CCCs. Noclaim is made for“abilities” asdecoupled from DCIsandCCCs.)

Abilitytodevelopmodel-basedarguments:• Abilitytodevelopmodelsthatrepresentnaturaleventssystems,

aspectsofatheoryandevidence,ordesignsolutions.• Abilitytodeterminethecomponentsaswellasconnectionsand

relationshipsamongmultiplecomponentsof theevent,system,ordesignsolutiontoincludeinthemodelandthosetoomit.

• Abilitytodeterminescope,scale,andgrain-sizeofthemodel,asappropriatetoitsintendeduse.

96

Thissectionadaptedfrom Model-basedArgumentationDesign PatternDocument.

• Abilitytorepresentmechanisms,relationships,andconnectionsto explain the event, system, or design solution with multipletypesofmodels.

Abilitytoevaluatecompetingmodels:• Ability to evaluate the explanatory or predictive power of

competing models taking into account evidence and empiricaldataassociatedwithaphenomenonunderinvestigation.

• Abilitytoevaluatehowcompetingmodelsdescribemechanismsandprocessesrelatedtothetargeteventorsystem.

• Ability to collect evidence to reason qualitatively orquantitatively about concepts and relationships represented inmodels.

• Abilitytouseevidenceorempiricaldatatogenerateexplanationsandpredictionsaboutthebehaviorofascientificphenomenon.

• Abilitytoapplyscienceconceptsorprinciplestoreasonwhythedatasupportorrefuteanargument.

• Abilitytoselectappropriatemodelrepresentationsthataremostusefulforsupportingorrefutinganargument.

Abilitytorevisemodel-basedarguments:• Abilitytorevisemodelsinlightofempiricalevidencetoimprove

theirexplanatoryandpredictivepower.• Ability to apply alternative science concepts/principles to

supportanargument.• Ability to refine the data collection approach to improve the

appropriateness,accuracy,orsufficiencyoftheempiricaldata.Viable Indicators ofCross-DimensionalProgress (Note:“cross-dimensional”heremeansreasoningthat manifests in theprocess of engagingwith a practice in aparticularwayaboutaparticularphenomenon. Noclaim is made for“abilities” asdecoupledfromDCIs.)Thissectionadaptedfrom Design

ProgressinSystemsandSystemModelsSM.1. Lowest performing students – Ability to identify relevant

features but not provide linkages connecting severalfeaturesnorprovideexplicitreasoningwhyfeaturesareorarenotrelevant.SM.1.a. Fragmented identification of system components and

relationshipswithinandacrossscalesorscopesthatdonotmaptoeachother.

SM.1.b. Systemcomponentsmayberepresentedas“boxedin”bysystemboundaryandimmutabletochange.

SM.1.c. Limitedabilitytorepresenttheboundariesofthesystemandsystemmaybeinappropriatelycategorizedasopenorclosedtoexternalinputs.

SM.1.d. Identification of salient features may be based onfamiliarity (or similarity to familiarfeatures/phenomena)ratherthanonwhatbestsuitsthegoal.

97

Patterns for CCC xSEP

SM.1.e. Goal of model development is visuospatial illustrationphenomenon with overemphasis on representing “thelook” of the system rather than relationships amongentities.

SM.2. Lower Intermediate performing students – Able to createsingle entity-entity relationship connections. Able to findseveralinstancesinwhichthesameconnectionappears.SM.2.a. In model development, may identify multiple system

entities and relationships atmultiple system levels, butnot represent interdependence among those systemlevels.

SM.2.b. Inmodeluse,maybeabletodescribethatsystemlevelsareinterdependentwithoutbeingabletoprovidedrivingmechanism.

SM.2.c. Able to identify movement of energy or matter acrosssystem boundaries (though inciting act will rely onanthropomorphicagents).

SM.2.d. May be inconsistently able to create a chain of singlecause-singleeffectrelationshipswithinthesystemlinkedoneafterthenext.

i. Represented chains will always reinforce theeffect(positivefeedbackloopsonly).

SM.2.e. Reasoning for represented relationshipsmaybe just-soor overly emphasize anthropomorphic agents. Goal ofmodel development is to describe “theway things are”rather than to have some broaderargumentative/predictivegoal

SM.3. HigherIntermediateperformingstudents–Abletoidentifyhow micro-level causal relationships can produce higher-levelsystembehavior.Abletocontextualizeamodelintermsofsomeexplanatoryorargumentativegoal.SM.3.a. In models and explanations, indicate relationship

mappinginwhichmanycausescombinetoproduceanyoneeffect.

i. At the higher end of this benchmark, mappingmaybegintobecomemany-to-many

SM.3.b. In models and explanations, interdependentrelationships across system/subsystem levels arerepresentedinlimitedform.

ii. Student can identify/represent evidence at thesubsystem level for relationships at the systemlevel.

98

iii. Studentcanidentifyevidenceatthesystemlevelfor relationships that primarily exist at thesubsystemlevel.

SM.3.c. Can describe/represent both positive and negativefeedback loops in matter/energy (or generalizedinput/output)flows.

SM.3.d. Continues to explain emergent phenomena in terms ofcentral control or adherence to a coherent, rigidframework.

SM.3.e. Can explain how features of model support a broaderexplanatory goal and make choices regarding modelfeatureselectionpursuanttothatgoal.

SM.4. Highestperformingstudents–AbletorecognizeconstraintsandlimitationsofamodelSM.4.a. Able to identify feature mediators (e.g. temperature,

topography, available inputs) that may alter existingsystemrelationships.

SM.4.b. Able to identify subsystem relationships mediate (orcreatedynamism)amongsystemrelationships

SM.4.c. –MayalreadybeatthehighestlevelbySM.3.cSM.4.d. In models and explanations, students represent how

emergent properties at higher system levels manifestfromtherulesgoverninglower-levelsysteminteractionswithoutrequiringcentralizedagents/actors.

SM.4.e. Goal of the model is to support critique of providedarguments, suggestnew sourcesof evidence, andmakecomplex predictions about how relationships fit into abroadersystemcontext

ProgressinStructureandFunctionSF.1. Lowest performing students – Ability to identify relevant

features but not provide linkages connecting severalfeaturesnorprovideexplicitreasoningwhyfeaturesareorarenotrelevant.SF.3.a. Fragmented identification of structural features and

entityfunctionswithinandacrossscalesorscopesthatdonotmaptoeachother.

SF.3.b. In models and model use, students primarily identifymacro-scalestructuresorfunctionsandonlysomemicro-scale functions (micro-scale structure may not beexpressed).

SF.3.c. Identification of salient features may be based onfamiliarity (or similarity to familiar

99

features/phenomena)ratherthanonwhatbestsuitsthegoal.

SF.3.d. Goal of model development is visuospatial illustrationphenomenon

SF.2. Lower Intermediateperformingstudents (Bridges?)–Abletocreatesinglestructure-functionconnections(orchainofthatconnection).Abletofindseveralinstancesinwhichthesameconnectionappears.SF.2.a. In models and explanations, indicate structures and

functionsaremapped1:1SF.2.b. In models and explanations may identify multiple

structuresandfunctionsatmultiplesystemlevelsSF.2.c. Inmodelsandexplanations,failtoidentifyhowstructures

andinterrelationshipsamongstructuresaffordfunction.SF.2.d. Reasoning for structure may be just-so or overly

emphasizeanthropomorphicagents.SF.3. HigherIntermediateperformingstudents–Abletoidentify

howmacro-levelfunctionalpropertiesinteractwithmicro-levelstructuresandhowmodelcanbeusedtosupportgoalSF.3.a. Inmodels and explanations, indicate structure-function

mapping(maybemanytoone)SF.3.b. Inmodelsandexplanations,identifythatstructuresand

functionsareinter-relatedacrosssystemlevelsSF.3.c. Canusethemodeltomakeanargumentinsupportofa

drivingmechanismthatmakesstructuresaffordfunction.SF.3.d. Can explain how features of model support a broader

argumentative goal andmake choices regardingmodelfeatureselectionpursuanttothatgoal.

SF.4. Highestperformingstudents–AbletorecognizeconstraintsandlimitationsofamodelSF.4.a. Inmodels and explanations, identify structure-function

relationshipsaredynamicSF.4.b. Abletoidentifymediators(e.g.temperature,topography,

available inputs) that may alter existing structure-functionrelationships.

SF.4.c. Able to use model of structure and function to makepredictions about how relationships fit into a broadersystemcontext.

SF.4.d. Abletocritiqueamodeloraskaquestionofamodelthathighlights failure to reach anexplanatory/predictive/argumentativegoal.

100

DCI AssessmentTargets

Studentscan:LS2.B.a. Differentiatebetweenorganismsbasedonwhethertheyare

producers,consumers,ordecomposers.LS2.B.b. Describetherolenonlivingpartsofanecosystem(likewater,

air,andminerals)playinmediatingtheinteractionsbetweenproducers, consumers, and decomposers within anecosystem.

LS2.B.c. Tracksandquantifythecyclingofmatterandenergythroughvarious reservoirs (both living and nonliving) within anecosystem.

LS2.B.d. Describe relationships between the consumption ofmatterand energy by both consumer and producer organisms intermsofmatter/energyconservationprinciples.

LS2.C.a. Identify evidence that changes are occurring among thephysical or biological components of an ecosystem anddescribetherateatwhichsuchchangesareoccurring.

LS2.C.b. Identify evidence that changes are occurring to thefunctioningofanecosystem(particularlyitsabilitytosupportpopulation growth) and describe the rate at which suchchangesareoccurring.

LS2.C.c. Identifykeystoneecosystemcomponents/relationshipsthat,if changed, have disproportionately large effects on thecontinuedfunctioningofecosystem.

LS2.C.d. Describe mechanisms by which various ecosystemcomponentsact tosupportpopulationgrowth,quantify thestrength of correlations between those mechanisms andpopulation sizes, and the describe the resilience of thosemechanismstochange.

LS2.C.e. Usebiodiversityasameasureofthehealthofanecosystemandtheabilityofanecosystemtosupportpopulationgrowth.

LS4.D.a. Identifyvarious‘Ecosystemservices’inwhichhumansacttosupportthestability/resilienceofanecosystem.

LS4.D.b. Describe the tradeoffs associated with various ecosystemservicesinlightofalocalcontext(bothlocaldesiresandlocalconstraints).

LS4.D.c. Identify potential unintended consequences of a givenecosystem service endeavor and describe ways that canmitigatetheriskofunintendedconsequencesoccurring.

ETS1.B.a. Identify relevant constraints and criteria that apply todesigning an engineering solution to a problem faced by alocalcommunity.

ETS1.B.b. Describetradeoffsbetweensolvingonlyalimitednumberofthetotalsetofproblem-pieces.

101

ETS1.B.c. Evaluate the effectiveness of multiple (three or more)potentialdesignsolutions toanengineeringproblembasedonprovideddata.

ETS1.B.d. Identifyappropriateevidencesourcesthatcouldbeusedtoevaluatepotentialdesignsolutionsontheirabilitytomeetthecriteria of the problem definition while staying withinrelevantconstraints.

PossiblePhenomenaorContexts*

Potential criteria/constrains that may govern the viability of designsolutionsinclude:

• Relevantphysicalprinciples• Costofdevelopment• Featuresofthelocalgeography• Differentialfeaturesoflocalclimate• Trendsinpopulationgrowthorper-capitaresourceconsumption• Localandglobaltrendsinhumanbehavior

Examples ofIntegration ofAssessment TargetsandEvidence

1. Systems and System Models – Task provides a modelrepresentingfoodwebrelationshipsamongorganismswithinan ecosystem. Students are asked to identify consumers,producers, and decomposers as well as identify keystoneorganisms (whereby changes to their populations havedramaticeffectsontheotherpopulationsintheecosystem).

a. SuccessfulPerformance:Studentsareabletoidentifyand categorize organisms appropriately in light of aprovidedcontext(SM.1.a).

b. Unsuccessful Performance: Students suggest thatirrelevantsystemcomponentsaremostsalient(i.e.fisharekeystoneorganismsbecausehumanswillfeelsadiftheydieout)(SM.1.d).

2. SystemsandSystemModels–Taskexpandsonthefoodwebmodeltoincludethetrackingoftransferofenergyandmatterviatheproductionandconsumptionoforganisms’biomasses(thismaybeasimulatedtask).Studentsareaskedtoquantifymatter/energystoredindifferentreservoirs.

a. Successful Performance: Students are able to trackthe transfer of energy among ecosystem componentsvia a chain of solar source -> producer viaphotosynthesis -> primary consumer -> secondaryconsumer->etc(SM.2.c).Theymayexhibitconfusionabout the role of decomposers as part of a nutrienttransportcycle(SM.2.b&SM.2.d).

b. Unsuccessful Performance: Student fails to identifyenergy/matter transport pathways or fails to amendmodeltorepresenttransport(SM.1.c).

102

3. SystemsandSystemModels–Taskexpandson thecontextgroundingthe foodwebmodel toprovidedetailsofachangethat may be affecting the physical or biological subsystemcomponents of the ecosystem. Students are asked to reasonabout the higher-level effects these changes might have onexistingenergy/mattertransportpathways.(Notethat,atthislevel, students will still focus on centralized controlmechanisms).

a. SuccessfulPerformance:Studentsareabletoexpressbi-directional, interdependent relationships betweensubsystemandsystemcomponents(SM.3.b).Theyareabletousetheserelationshipstoexplainwhetherornottheecosystemisresilienttothechangeprovidedbythetask(SM.3.c).

b. UnsuccessfulPerformance:Student fails todescribeways in which changes to system components canresult in changes to existing food-web relationshipsand/orfailstodescribewaysinwhichchangestooneormore food-web relationshipsmay affect the entireweb(SM.2.a&SM.2.b).

c. Potentialmidpointperformance:Studentsisabletodescribe interdependentrelationships(SM.3.b)but isonlyabletorepresentpositivefeedbackloops(therebyalwaysconcludingthatthereisalackofresilienceintheecosystem)(SM.2.d.i).

4. Systems and System Models – Task re-contextualizes theecosystem changes in terms of risks to change (i.e. anunintendedconsequenceofanecosystemservice).Studentsareaskedtomakeanargumentwhichcontrastspotentialbenefitsof the ecosystem service against potential negativeconsequences(bothofactionandinaction).

a. Successful Performance:Student is able to describemediating factors that contribute or mitigate risk,including feedback loops, entity mediators, andrelational mediators in order to predict outcomes(SM.4.a & SM.4.b). Argument for/against ecosystemserviceshouldbegroundedinthatprediction(SM.4.e).

i. Alternative: Student is able to critique aprovided argument/prediction on the basis ofrelationalmediators(SM.4.e).

b. UnsuccessfulPerformance:Student fails todescribehow system features or entities may act to mediateother system relationships (i.e. fails to describe how

103

populationgrowthaffectsoverallresourceavailability)(SM.3.d). They make inaccurate or inappropriatepredictionsorfailtoarguehowmodelelementsexisttosupportthatprediction(SM.2.e).

5. Structure and Function – Task provides a description ofseveral potential engineering solutions each designed toprovide an ecosystem service in light of the needs of a localcommunity. Students are asked to model the criteria andconstraints of the problem definition, highlighting relevantaspectsofthelocalenvironment.(Modelsmaytaketheformofillustrations,conceptmaps,linguisticdescriptions,etc.)

a. Successful Performance: Student is able to identifyrelevant ecosystem structures and describe how theengineering solution fits into those pre-existingstructures.(SF.1.a&SF.1.d)

b. UnsuccessfulPerformance:Studentoveremphasizesfamiliar structures or functions (SF.1.c). Studentrepresentsonlystructuresoronlyfunction(SF.1.b).

6. Structure and Function – Task provides further detailsregardingtheproposedengineeringsolutionforanecosystemservice.Studentsareaskedtodrawconnectionsbetweenthestructure/function relationships that are common to severalproposeddesigns.

a. SuccessfulPerformance:Studentsuccessfullymapsaparticularengineeredstructuretothefunctionalgoalofthe design (SF.2.a).They are able to identify severalinstances in which a given structure-function pairingappearsinmanydifferentproposedsolutions.(SF.2.b)

b. Unsuccessful Performance: Student fails to form astructure-function pairing or fails to see that samerelationshipinotherdesigns(SF.1.a).

7. StructureandFunction–Taskprovidesadditionalcontextinwhicha localcommunity isseeking toevaluate theproposedsolutionsinordertoimplementonethatbestsuitstheirneeds.Studentsareaskedtobuildontheconnectionsfromthebridgetasktoidentifysourcesofevidencethatcouldbeusedtoassessandselectthebestdesign.

a. Successful Performance: Student is able to usecommonalities in structure-function relationshipsacrossdesignstoidentifypotentialsourcesofevidence(SF.3.c). Student is able to identify evidence thatmanifests at a different level than the structure-

104

functionrelationshiporacrosslevelsofthestructure-functionrelationship(SF.3.b).

b. Unsuccessful Performance: Student fails to developan argument for how evidence collected supports anassessment of the proposed design (SF.2.c). Studentfails to contextualize relationships in terms of thecriteriaandconstraintsboundingtheproblem(SF.2.d).

8. StructureandFunction–Taskprovidesanargument fororagainst the implementationof aparticular ecosystemservicevia the construction of some human-engineered structure.Students are asked to use a model of the functioning of thesolutiontocritiquetheproposedimplementation/constructionstrategy.

a. SuccessfulPerformance:Studenteffectivelyusesthemodel to identify mediating relationships that runcounter to the functioning of the engineering design(SF.4.b) and proposes amendments to theimplementation that may act to mitigate unintendedconsequences(SF.4.c).

b. UnsuccessfulPerformance:Studentfailstopresentanaccount of the functioning of the design solutionwhereby mediators transform a many-to-onerelationshipintoamany-to-manyrelationship(SF.3.a).

CommonMisconceptions*

AdditionalAssessmentBoundaries

*Notanexhaustivelist

105

AppendixB:HypothesizedLearningProgressionsSystemsandSystemsModels

Level 1 à Level 2 à Level 3 à Level 4 à Level 5 SM1. Aggregate

level phenomenon

SM1.1.A. Inaccurately identify phenomena of interest (i.e., perturbed state, equilibrium)

SM1.1.B. Inaccurately describe phenomenon using observable features and personal experience.

SM1.2.A. Accurately identify relevant phenomenon of interest. May inaccurately identify other phenomena.

SM1.2.B. Accurately describe the phenomenon using observable features and personal experience only. May be inconsistent in describing phenomenon.

SM1.3.A. Accurately identify relevant phenomenon of interest only.

SM1.3.B. Accurately describe the phenomenon using observable features and one hidden mechanism. May be inconsistent in describing phenomenon.

SM1.4.A. Already at highest

SM1.4.B. Accurately describe the phenomenon using observable features and several hidden mechanisms. May be inconsistent in describing the mechanisms.

SM1.5.A. Already at highest

SM1.5.B. Accurately describe the phenomenon using observable features and hidden mechanisms.

106

SM2. Components

SM2.1.A. Identify irrelevant and/or inaccurate components related to the phenomenon

SM2.1.B. No identification of sub-components

SM2.1.C. No description of components as being static and/or changing.

SM2.2.A. Accurately identify some components relevant to the phenomenon. May identify irrelevant components.

SM2.2.B. Identify sub-components but may be irrelevant and/or inaccurate.

SM2.2.C. Describe components as being static or unchanging (initial conditions).

SM2.3.A. Accurately identify all components relevant to the phenomenon.

SM2.3.B. Accurately identify all sub-component

SM2.4.A. Already at highest

SM2.4.B. Already at highest

SM2.4.C. Describe components (or initial conditions) as changing due to external and internal factors. Description include short-term and long-term changes and may be inaccurate.

SM2.5.A. Already at highest

SM2.5.B. Already at highest

SM2.5.C. Accurately describe components (or initial conditions) as changing due to external and internal factors. Description include short-term and long-term changes.

107

SM2.3.C. Describe

components (or initial conditions) as changing due to external factors. Description is limited to short-term change and may be inaccurate.

SM3. Relationships between components

SM3.3.A. Describe causal relationships between components

SM3.4.A. Accurately describe causal relationships between

SM3.5.A. Already at highest

SM3.5.B. Accurately mention bidirectional

108

and sub- components but may be irrelevant and/or inaccurate.

SM3.3.B. Identify unidirectional between components and sub- components but may be irrelevant and/or inaccurate

SM3.3.C. Identify and one to one relationships but may be irrelevant and/or inaccurate

SM3.3.D. Identify linear relationship between components and sub- components but may be irrelevant

components and sub- components.

SM3.4.B. Accurately identify unidirectional between components and sub- components.

SM3.4.C. Identify one to one relationships and one-to-many relationships but may be irrelevant and/or inaccurate.

SM3.4.D. Accurately identify linear relationships between components and sub- components

and/or cyclical relationship between components and sub- components.

SM3.5.C. Accurately identify one to one relationships and one-to-many relationships

SM3.5.D. Accurately identify linear and non-linear relationships between components and sub- components

109

and/or inaccurate

SM4. Mechanisms (processes/ relationship)

(e.g., emergence, feedback loops, adaptation, iteration, randomness).

SM4.3.A. Identify one central mechanism that explains observed phenomena. May be inaccurate.

SM4.3.B. Identify mechanisms at one level only (e.g., local only and no aggregate). May be inaccurate.

SM4.3.C. Does not identify random nature of mechanisms at the local or individual level

SM4.3.D. Describe mechanisms as static, even as the components change.

SM4.4.A. Identify multiple mechanisms that explains observed phenomena. May be inaccurate and/or continue to identify one central mechanism.

SM4.4.B. Identify mechanisms at multiple levels (e.g., local and aggregate). May be inaccurate.

SM4.4.C. Identify random nature of mechanisms at the local or individual level. May be inaccurate.

SM4.5.A. Accurately identify multiple mechanisms that interact together to give rise to observed phenomena.

SM4.5.B. Accurately identify mechanisms at local and aggregate levels.

SM4.5.C. Accurately identify random individual/local mechanisms that interact across multiple levels in the system.

SM4.5.D. Accurately describe mechanisms and components as changing dynamically in response to one another.

110

SM4.3.E. Does not make any predictions about system level behaviors based on mechanisms

SM4.4.D. Describe mechanisms and components as changing dynamically in response to one another. May be inaccurate.

SM4.4.E. Predictions about system level behaviors from mechanisms are linear in scope (e.g., single mechanisms as impacting system)

SM4.5.E. Predictions about system level behaviors from mechanisms are dynamic. (e.g., multiple interacting behaviors as impacting system)

SM5. Properties of System

SM5.3.A. Limited or unclear description of relevant boundaries

SM5.3.B. System may be inappropriately categorized as open or closed

SM5.4.A. Detailed description of the boundaries of the system. May be inaccurate.

SM5.4.B. System categorized as open or closed to external inputs. May

SM5.5.A. Accurate description of the boundaries of the system

SM5.5.B. System accurately categorized as open or closed to external inputs.

SM5.5.C. Understanding that

111

to external inputs.

SM5.3.C. Inaccurate understanding of equilibrium as simply meaning positive balance

include inaccuracies

SM5.4.C. Limited understanding that mathematical equilibrium includes stable & positive population only

mathematical equilibrium includes both unstable and stable, positive and extinction of populations (i.e., chaos is the norm)

StructureandFunction

Level 1 à Level 2 à Level 3 à Level 4 à Level 5 SF1. Structure

(Components)

SF1.1.A. Identify a single structure but not all of its components. Description is based on familiarity or visual similarity. May be inaccurate.

SF1.1.B. Describe structures as consisting of smaller components. May be inaccurate.

SF1.2.A. Accurately identify a single structure and its sub-structures

SF1.2.B. Accurately describe structures as consisting of smaller components.

SF1.3.A. Accurately identify multiple structures and including sub-structures

SF1.3.B. Already at highest

SF1.4.A. Already at highest

SF1.4.B. Already at highest

SF1.5.A. Already at highest

SF1.5.B. Already at highest

112

SF2. Function: Purpose of the structure

SF2.1.A. Describe function by inferring from observable conditions or surface details, but details are limited or significantly inaccurate.

SF2.2.A. Describe function by inferring from observable conditions or surface details, but details are limited.

SF2.3.A. Describe individual functioning of structures and sub-level structures but does not describe overall function as a product of sub-level conditions.

SF2.4.A. Describes the multiple functions of structures and sub-level structures. Infers function as a product of sub-level conditions. May be inaccurate.

SF2.5.A. Accurately describes the multiple functions of structures and sub-level structures. Accurately infers function as a product of sub-level conditions.

SF3. Behavior (Mechanisms): Attributes or specific states derived from the structure Examples: cell wall or membrane

SF3.2.A. Does not describe properties/behavior of structure

SF3.2.B. Does not describe conditions under which the properties of the structure support its function

SF3.2.C. Does not describe relationships between structural

SF3.3.A. Limited description of properties or behavior that are inferred from macro level structures only.

SF3.3.B. Limited description of conditions under which the properties of the structure supports its function

SF3.3.C. Limited or no description

SF3.4.A. Infer properties or behavior from articulating relationship among micro-macro structures and their function. May be inaccurate

SF3.4.B. Causal description of the conditions under which the properties of the structure supports its function. May be inaccurate

SF3.5.A. Accurately infer properties or behavior from articulating relationship between (sub)-structures and their function.

SF3.5.B. Accurate causal description of the conditions under which the properties supports its function.

SF3.5.C. Describe causal relationships between structural

113

components and overall function.

of causal relationships between structural components and overall function on one level only. May be inaccurate.

SF3.4.C. Describe causal relationships between structural components and overall function that are parallel and occur at multiple levels. May be inaccurate.

components and overall function that are parallel and integrated across multiple levels in relation to overall function of system

114

AppendixC:EcosystemTaskItemswithDesignRationales

QUESTION1:Whyworryabouttherootworminvasion?

Key:Unkeyed.ThisitemwasredesignedtoaskstudentstoranktheconcernsaccordingtotheirsalienceforthepeopleofFarmville.Question1:DesignRationale• What is the assessment item asking? This item introduces students to the relevantentities that will be at play over the course of the whole activity while probing theirunderstanding that disturbances to one element in a system can affect other systemcomponents.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilityto identify likelyoutcomesof therootworminvasion,particularlythatmultiplenegativeoutcomesmayoccur.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfordistractorresponses.

1. Thegoatswillnothaveenoughcorn toeat. –The corn rootworms (CRWs)directlydecreasestheamountofcornavailable.Goatseatthecorn,soareductionincorndecreasestheamounttheycaneat.

2. Thepeoplewillnothaveenoughcornasfood.–TheCRWsdirectlydecreasestheamountofcornavailable.PeopleinTownsvilleeatthecorn,soareductionincorndecreasestheamounttheycaneat.

3. Thecornrootwormswillspreadtootherfarms.–TheCRWsaremobileandcanmove fromfarmto farm(Note:checkrespondent’s thoughtsaboutcentralmechanismdescribedinthestory).

115

4. Jonahwill not have enough corn for earningmoney. –The CRWs directlydecreases the amount of corn available. Jonah sells excess corn to support hisfamily,soareductionincorndecreasestheamounthecansell.

5. Thecornplantswillnotsurviveandeventuallydieoff.–TheCRWsconsumethe corn, if left unchecked, the CRWs will consume all the corn (Note: checkrespondent’sthoughtsabouthumanabilitytopreventcollapse).

QUESTION2:Differentiatingtheroleoforganismsonthefarm

Key: Corn–Producer,Goat–PrimaryConsumer,Cornrootworm–PrimaryConsumer

Human–PrimaryConsumerandSecondaryConsumer

ItemDesignRationale• What is the assessment item asking? This item presents a variety of trophic rolesorganismscantakeonwithinanecosystemandasksrespondentstocorrectlyassociatestoryelements(corn,goats,people,andcornrootworms[CRWs])withthoseroles.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytodifferentiateprimaryandsecondaryconsumption,identifywhichkindoforganismsareconsidered producers, and to see CRWs as invasive consumers rather than keydecomposers.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexist for some select distractor responses. Further rationales may need to beinvestigatedwithinterviewprobes.

7. Thecornareproducers–Cornproducesediblebiomassbyconvertingmatter(carbonintheairandwaterfromair/soil)byusingenergyfromthesun.

8. Thegoatsareprimaryconsumers.–Thegoatdirectlyconsumesthecorn.

116

9. The people of Townsville are both primary consumers and secondaryconsumers. –Humans consume corn both by directly eating it and by eatingorganisms(thegoats)thateatthecorn.

10. ThepeopleofTownsvilleareoneofeitherprimaryorsecondaryconsumers-Humansconsumecorneitherbydirectlyeatingitorbyeatingorganisms(thegoats) that eat the corn. (Note: check respondent’s reading of the story andwhethertheynotedthatbothconsumptionsoccur).

11. Thecornrootwormsareprimaryconsumers.–TheCRWsdirectlyconsumethecorn.

12. The corn rootworms are decomposers. – The CRWs cause the corn to die(decompose).

QUESTION3:Creatingthebaselineenergy-transfermodel

Key:Sun->Corn;Corn->GoatItemDesignRationale• What is theassessment itemasking? This itemprovides a selectionof elements thatoughttobeincorporatedviatheSageModelerinterface.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytoincorporatetheappropriateentities,drawarrowsinappropriatedirection,theorderofelementincorporation,andanyerrorsmadealongtheway.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexist for some select distractor responses. Further rationales may need to beinvestigatedwithinterviewprobes.

117

1. Sun->Corn->Goat–Thesunproducestheenergythatsupportstheproductionofplant(corn)matterthatisultimatelyconsumedbythegoat.

2. Goat->Corn->Sun–Thegoatsderiveenergyfromthecornwhichderiveenergyfrom the sun (Note: check on respondent’s understanding on themeaning ofarrowsandtheirdirectionality).

3. Sunnotinmodelornotconnectedtoothermodelelements–Foodwebsonlyshow the flow of matter, sunlight is not matter and, therefore, should not beincludedinthemodel(Note:checkonrespondentsreadingofthetaskprompt).

4. Goatconnectedviathesun–Thegoatsliveduringtheday(whenthesunisout);they would die if the sun was gone (reasoning may vary); the sun providesnecessarythermalenergytothegoats.

QUESTION4:Whatdomodelelementsrepresent?

Key:2ItemDesignRationale• What is theassessment itemasking? This item levels students from thepastmodel-creation taskandasks them to reflecton the informationabout theecosystem that themodelisattemptingtoconvey.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytoidentifythemechanismsbywhichmodelelementsservearolewithintheecosystem(asrepresentedbythearrowsinthemodel).• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfordistractorresponses.

118

1. Transferofmatter in thesametrophic level.–All theorganismsare in thesamelevel(horizontally)thereforethematterisflowingwithinthatsinglelevel(Note:checkrespondent’sunderstandingofwhat“trophiclevels”are).

2. Transfer of energy in the ecosystem. –Energy in the ecosystem originatingfromtheSunmovesfromelementatthebackofthearrowtotheelementattheheadofthearrow.

3. Feedingrelationshipsbetweenorganisms.–ThegoatsonJonah’sfarmsurvivebyfeedingonthecorn.

4. Thepopulationsizeofeachorganism–Withintheecosystemthereisasun,acorn,andagoat.(Note:checkhowstudentswouldmodelecosystemifthefarmcontainedmanycornstalksormanygoats).

QUESTION5:Integratingthecornrootwormintothemodel

Key: Sun->Corn;Corn->Goat;Corn->Rootworm ItemDesignRationale• Whatistheassessmentitemasking?This itemprovidesthecornrootwormasanewelementtobeincorporatedintothepreviousmodelviatheSageModelerinterface.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytoincorporatetheappropriateentities,drawarrowsinappropriatedirection,theorderofelementincorporation,andanyerrorsmadealongtheway.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexist for some select distractor responses. Further rationales may need to beinvestigatedwithinterviewprobes.

1. Corn->CornRootworm–Cornisafood/energysourcefortherootworms.

119

2. CornRootworm->Corn–Therootwormsderiveenergy fromthecornwhichderiveenergyfromthesun(Note:checkonrespondent’sunderstandingonthemeaningofarrowsandtheirdirectionality).

3. CornRootwormconnectedtothegoats–TheCRWscompetewiththegoatsandcausethegoatpopulationtochange(Note:checkonrespondentsreadingofthetaskprompt).

QUESTION6:Whatdomodelelementsrepresent?

Key: 1-newfoodchain;foodweb

2-producerstoconsumers;consumerstoproducersItemDesignRationale• What is theassessment itemasking? This item levels students from thepastmodel-creationtaskandasksthemtoreflectonthechangestotheinformationcontainedwithinthemodelasafunctionoftheincorporationoftheCRW.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilityto relate thenewroleof theCRWwithin theecosystem toa change inwhereavailableenergyinthesystemsgoes.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfordistractorresponses.

1. Newfoodchain.–Previouslythemodelshowed1foodchain(sun->corn->goat)andnowitalsoshowsanotherfoodchain(sun->corn->CRW)aswell(Note:checkstudentreasoningaboutthedifferencebetweenmultiplefoodchainsandasinglefoodweb).

2. Foodweb.–TheintegrationoftheCRWsmeanswehavecreatedamodelinwhichmanyorganismsserveastheproducerfororconsumerofmanyotherorganisms.

120

3. Producers to consumers – The CRWs consume the corn and derive energyproducedfromitsgrowthprocess(photosynthesis).

4. Consumerstoproducers.–WhentheCRWsdie,theirbodiesprovidenutrientstothesoilthatcornneedstogrow(i.e.“circleoflife”naïveconception;Note:checkon respondent’s understanding on the meaning of arrows and theirdirectionality).

QUESTION7:Inferringtheconsequencesoftheinvasion

Key: Corn–decrease

Goat–decrease(ifcornistheonlyfoodavailableinthefarm)People–remainthesame

ItemDesignRationale• Whatistheassessmentitemasking?This itemasksrespondentstomakepredictionsaboutchangestopopulations(bothseenandunseen)livingintheecosystemasaresultoftheCRWinvasion.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilityto identify likelyoutcomesof therootworminvasion,particularlythatmultiplenegativeoutcomesmayoccur.Theyshouldbeabletobackthesepredictionswithastatementaboutevidenceavailableinthemodel.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexist for some select distractor responses. Further rationales may need to beinvestigatedwithinterviewprobes.

1. Cornpopulationdecreases.–Thecornrootworms(CRWs)consumethecorn,directlyreducingcornpopulationnumbers.

121

2. Cornpopulationincreases–WhentheCRWsdie,theirbodiesprovidenutrientstothesoilthatcornneedstogrow(i.e.“circleoflife”naïveconception;Note:checkonstudentconceptionsabouttimelinedisplayedinthemodel–thatthefoodwebisasnapshotintime).

3. Cornpopulationstaysthesame–Thecorndrawsitsenergyfromthesun,sonewconsumersdon’taffecthowmuchfoodthecorngets(orbyextensionthecornpopulation supportedby the ecosystem;Note: check respondent’s viewof theconnectionbetweenenergyacquisitionandmatterconsumptiondepictedinthemodel).

4. Goatpopulationdecreases–The invasionof theCRWsmeansthere isanewcompetitorforthecorn,decreasingtheamountofenergyavailabletosupportthecurrentgoatpopulation,andresultinginagoatpopulationdecrease.

5. Goat population stays the same –While the CRWs consume corn, goats aremobileandcanconsumeotherplantswiththeaidoffarmerJonah(Note:checkrespondent’sthoughtsaboutcentralmechanismdescribedinthestory).

6. Populationofpeopledecreases.–TheCRWsdirectlydecreasestheamountofcornavailable.Jonahandhisfamilyeatthecorn,soareductionincorndecreasestheamountheandhisfamilycanconsume(Note:checkrespondent’sthoughtsabouthumanabilitytopreventcollapse).

7. Population of people increases – Human populations increase because, onaverage,more children are born than die (Note: check respondent’s ability toconnectclaimmadeaboutpopulationininformationcontainedwithinthemodel)

8. Populationofpeoplestaysthesame.–Unlike thegoats,humanscanreadilyacquireenergyfromnewsourcesandarenotreliantoneatingcorn(hencewhytheywerenotnecessarytoincludewithinthefoodchain/web).

QUESTION8:Understandingtheconsequencesoftheinvasion

122

Key:2&4ItemDesignRationale• Whatistheassessmentitemasking?Thisitemprovidesavarietyofdatadisplaysfromthe results of the CRW-invasion simulation and asks students to reason about theconsequencesofnotimplementinganycontrolstrategies.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytolinkclaimsmadeabouttheimpactsoftheCRWinvasionwithevidencefromthedatadisplaysandreasoningaboutthemechanismsatplayduringtheinvasion.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfordistractorresponses.

1. Thenumberofcornplanted increased.–Since thecornrootworms(CRWs)directlydecreasestheamountofcornthatsurvives,thefarmerJonahshouldplantmore so that enough survives the season (Note: check respondent’s ability toconnectclaimmadeaboutpopulationinevidencecontainedinthedatadisplay)

2. Thenumberofcornharvesteddecreased.–Eachyearthenumberofcornthatsurvivesuntilharvestdecreasesperthedatainthegreencolumn.

3. Thecornrootwormsremainedthesame.–ThecorncanonlysupportafixednumberofCRWs, so theirpopulationwill be the sameover time (Note: checkrespondent’s ability to connect claim made about population in evidencecontainedinthedatadisplay).

4. Thenumberofcornrootwormeggsincreased.–AsCRWssurvivetheseason,theyareabletolaymorethan1eggperrootworm.ThiscausestheamountofCRWeggstorapidlyincrease.

123

QUESTION9:Determiningtheefficacyofharvestmenpredatorspt.1

Key:3,4,&5ItemDesignRationale• Whatistheassessmentitemasking?ThisitemprovidesavarietyofdatadisplaysfromtheresultsoftheCRW-invasionsimulationwhentheharvestmen-basedcontrolstrategywas implementedandasksstudents to reasonabout thechanges thatoccurred to farmpopulationsovertime.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytolinkclaimsmadeabouttheimpactsoftheCRWinvasionwithevidencefromthedatadisplaysandreasoningaboutthemechanismsatplayduringtheinvasion.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfordistractorresponses.

1. The%cornyieldremainedthesameeveryyear.–Implementingthecontrolstrategydecreases the growth rateof the corn rootworms (CRWs)population,improving corn yield percentage (Note: check respondent’s ability to connectclaimmadeaboutpopulationinevidencecontainedinthedatadisplay).

2. Thenumberofcornplantedincreasedeveryyear–Sincethecornrootworms(CRWs) directly decreases the amount of corn that survives, the farmer Jonahshouldplantmoresothatenoughsurvivestheseason(Note:checkrespondent’sabilitytoconnectclaimmadeaboutpopulationinevidencecontainedinthedatadisplay).

3. Thenumberofcornrootwormsincreasedeveryyear.–Despiteimplementingthe control strategy, the CRW population (as depicted in the pink column)continuedtoincrease.

4. The number of corn harvested decreased every year. – – Each year thenumberofcorn thatsurvivesuntilharvestdecreasesper thedata in thegreencolumn.

124

5. Thenumberofharvestmenremainedthesameeveryyear.–Theharvestmenare not native organisms to the farm and can only survive while there arerootwormsavailabletoeat.Duringthewinter,theydieoffandJonahmustreleaseanewsetof10eachspring.

QUESTION10:Determiningtheefficacyofharvestmenpredatorspt.2

ItemDesignRationale• Whatistheassessmentitemasking?Thisitemlevelsstudentssotheyrecognizerelevantpatterns inpopulationnumbers following the implementationof theharvestmen-basedcontrol strategy and asks them to provide some reasoning for why the strategy wasineffective.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilityto the criteria students generate for determining what counts as effective solutions(students might note that the harvestmen strategy is still better than nothing). Whatelementsofthesystem(andthesimulation)dostudentsfocuson?Dotheytiebacktheirresponsestothenot-simulatedgoatandhumanpopulations?• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfor some select naïve responses. Further rationales may need to be investigated withinterviewprobes.

1. Answerfocusesoninsufficientnumberofharvestmenaddedeachyear.2. Answerfocusesonthestrategystillbeingmoreeffectivethandoingnothingatall.3. AnswerfocusesontheavailableamountofenergytosupporttheCRWs.4. Answer focusesonvisual elements from thevideo (i.e. “theharvestmendidn’t

interactwiththebugsenough”)

125

QUESTION11:Determiningtheefficacyofthetrapcroppt.1

Key:1&3ItemDesignRationale• Whatistheassessmentitemasking?ThisitemprovidesavarietyofdatadisplaysfromtheresultsoftheCRW-invasionsimulationwhenthetrap-crop-basedcontrolstrategywasimplemented and asks students to reason about the changes that occurred to farmpopulationsovertime.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytolinkclaimsmadeabouttheimpactsoftheCRWinvasionwithevidencefromthedatadisplaysandreasoningaboutthemechanismsatplayduringtheinvasion.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfordistractorresponses.

1. The%cornyieldremainedthesameeveryyear.–ImplementingthetrapcropdistractstheCRWswithlessnutritiousnon-cornalternatives,allowingthecornto grow tomaturitybeforedestructionby theCRWs (asdepicted in the greencolumn)whilealsolimitingtheCRWpopulation.

2. Thenumberofalfalfa-trap-cropplantedincreasedeveryyear.–Byplantinglotsofalfalfa,butnotharvesting it for thegoatsorhumans, itspopulationcanincrease(Note:checkthatstudentcanidentifyrelevantdatacolumns).

3. Thenumberofcornrootwormsdecreasedfromyear3toyear4.–TheCRWsdonotreceiveenoughnutritionfromthetrapcrop,inhibitingthemfromlayingenougheggstoreplacetheincidentCRWpopulation.

4. Thenumberofcornharvesteddecreasedfromyear3toyear4.–TheCRWsstillcontinuetoconsumethecorn,eveniftheyalsoconsumethealfalfa.Coupledwiththedecreaseincornplantedbyreplacingsomelandwiththenon-corntrap

126

crop,means that corn to harvestmay decrease (Note: check that student canidentifyrelevantdatacolumnsinrelevantyears).

QUESTION12:Determiningtheefficacyoftrapcroppt.2

ItemDesignRationale• Whatistheassessmentitemasking?Thisasksstudentstogenerateanargumentaboutthe efficacy of the trap-crop-based control strategy and asks them to provide somereasoningforwhythestrategywasorwasnoteffective.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilityto the criteria students generate for determining what counts as effective solutions(studentsmight note that the trap crop strategy entails an immediate reduction in theamountofcornplanted/harvested).Whatelementsofthesystem(andthesimulation)dostudentsfocuson?Dotheytiebacktheirresponsestothenot-simulatedgoatandhumanpopulations?• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistforsomeselectnaïveresponses.Furtherrationalesmayneedtobeinvestigatedwithinterviewprobes.

1. Answer focuses on short term reduction, concluding the strategy wasineffective

2. Answerconcededshorttermreductionbutalsolong-termstabilization.3. Answerfocusesonthestrategystillbeingmoreeffectivethandoingnothing

atall.4. AnswerfocusesontheavailableamountofenergytosupporttheCRWs.5. Answerfocusesonvisualelementsfromthevideo(i.e.“thealfalfatakesup

toomuchspace”)

127

QUESTION13:Drawingaconclusionaboutthestrategies

ItemDesignRationale• Whatistheassessmentitemasking?Whatistheassessmentitemasking?Thisitemlevelsstudentssotheyrecognizerelevantpatternsinpopulationnumbersfollowingtheimplementationof the trap-crop-basedcontrol strategyandasks them toprovide somereasoning for why the strategy was effective (particularly several years afterimplementationbegan).• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytoextendthereasoningandargumentfromQ12.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistforsomeselectnaïveresponses.Furtherrationalesmayneedtobeinvestigatedwithinterviewprobes.

6. Answer focuses on short term reduction, concluding the strategy wasineffective

7. Answerconcededshorttermreductionbutalsolong-termstabilization.8. Answerfocusesonthestrategystillbeingmoreeffectivethandoingnothing

atall.9. AnswerfocusesontheavailableamountofenergytosupporttheCRWs.10. Answerfocusesonvisualelementsfromthevideo(i.e.“thealfalfatakesup

toomuchspace”)

128

AppendixD:Post-TaskInterviewProtocol

Item2(optionalquestions-askonlyiftimeallows)

• HowdoesidentifyingtheroleofCORN,GOATSandCORNROOTWORMhelpscientiststhinkabouttheimpactsofthecornrootworm?

Item3(Required)

• WhydidyouaddelementX(sun,corn,goat)intothemodel?HowdidyouconnectittoelementY?

• WhydidyouleaveelementX(sun,corn,goat)outofthemodel?HowdidyouconnectittoelementY?

• Whatdothearrowsinthemodelrepresent?Howdidyoudecidetoplacetheminthedirectionyoudid(indicatedirectionbasedonnotes)?

• RelevantSageModelerusabilityquestions.o Howeasywasusingwiththemodelingtool?o Wasthetutorialvideohelpfultoteachyouwhattodo?o Wasitclearhowtofixamistake(ifmade)?o Anysuggestionsforimprovement?

ITEM5(Required)

• HowdidyoudecidetoconnecttherootwormstoelementY?(Alternative:Whydidyouchosenottoconnecttherootwormstoanyoftheothermodelelements?)

• Howdidaddingtherootwormtothemodelhelpthescientiststhinkabouttheimpactsoftheinvasiononthefarmecosystem?

ITEM7(Required)

• Howmighttheinvasionofthecornrootwormshavethestatedeffect?(Interviewershouldremindthestudentsoftheresponsestheyselected)

• Didyouusethemodeltomakeyourhypothesis?Ifso,howwasthemodelhelpful?VIDEO:ExaminingtheImpactoftheRootwormInvasion(Required)

• Wasthevideousefultohelpyouunderstandtherelationshipsinthesystem?• Basedonthevideo,whatwasthepurposeofthesimulation?Whatdidyoulearn

fromthevideo? ITEM8(Required)

• Usabilityquestionsregardingthedatatableandgraphs.o Whatdiddifferentcolumnsinthedatatablerepresent?o Foreachgraph,whatpatternsdidyousee?Whatistherelationshipbetween

thegraphandthedatatable?o Whatdidwemeanbyyieldpercentage?Whyisyieldpercentageauseful

measureofJonah’sFarm?

129

VIDEO:EnactingtheHarvestmenControlStrategy(Required)

• WasthevideousefultohelpyouunderstandhowHarvestmenControlStrategyworks?

• Basedonthevideo,whatwasthepurposeofthesimulation?Whatdidyoulearnfromthevideo?

ITEM10(Required)

• Whywereonly10harvestmenreleasedperyearonthefarm?• Whatotherdatamighthavebeenhelpfulindrawingaconclusionabouttheefficacy

oftheharvestmen-basedcontrolstrategy?VIDEO:EnactingtheAlfalfaTrapCropControlStrategy(Required)

• WasthevideousefultohelpyouunderstandhowtheAlfalfaTrapCropControlStrategyworks?

• Basedonthevideo,whatwasthepurposeofthesimulation?Whatdidyoulearnfromthevideo?

ITEM12(Required)

• WhydidplantingtheAlfalfadecreasetheamountofcornplanted?• Whatotherdatamighthavebeenhelpfulindrawingaconclusionabouttheefficacy

ofthetrap-crop-basedcontrolstrategy?ITEM13(Required)

• Whatinformationabouttheefficacyofthestrategywasaddedbylettingthesimulationrunforseveralmoreyears?

• Howdidyouusethevideosofthesimulatedcontrolstrategiestohelpyourespondtorelevantquestions?

OverallTaskFeedbackQuestions

• Didyoufeellikeyouhadalltheinformationneededtoengagewiththetask?Whatotherinformationmighthavebeenhelpful?

• Weretheexperiencesofthistasksimilartotasksyouhadalreadyengagedinwithinyourscienceclassroom?

• Doyouthinkthefarmscenariowasmeaningfultoyou?Ifnot,whattypesofscenarioswillbemoreengaging?

• Wereyouhopingthatyoucouldhavedirectinteractionswiththesimulationshowninthevideo?Ifyouwereabletointeractwiththesimulationshowninthevideo,whattestswouldyouliketorun?

[EndofSemi-StructuredProtocol][Iftime,interviewasksfortaskfeedbackoraboutinterestingobservationsmadeduring

thinkaloud]

130

AppendixE:DAT-CROSS:BackgroundQuestionnaire

Beforewegetstarted,pleaseanswerthefollowingquestions.

1. What grade are you currently in? ○ 6th grade ○ 8th grade ○ 10th grade

2. How old are you? _(text entry)_

3. What is your gender?

o Male o Female o Other__________ o Prefer not to answer

4. Which of the following best describes you? (Select all that apply.)

African American

White Asian Hispanic Pacific Islander Native American

Mixed Race Other_______________

Prefer not to answer

5. Have you received any instruction about systems? Explain and use examples if necessary

131

6. Have you received any instruction about system models? Explain and use examples if necessary

7. Have you received any instruction about structure and function? 8. Do you have any experience using online or virtual simulations in school?

132

AppendixF:DataCollectionMatrix

DataCollection AnalysisMethods

• Background information questionnaire (See

AppendixE)

• Assessment items that probe students to be

moreexplicitintheirCCCreasoningbutarenot

expectedtofitintoanactualteachingsetting.

o Mixture of Multiple Choice/Multiple

Select and Free Response Items that

examineaCCCacrossDCIsfromseveral

domains.(SeeAppendixC)

• In-situthinkaloudwithstudentsthatprovide

them greater opportunity to vocalize their

reasoning. This will allow us to form some

validity in our assessment rubric. (See

AppendixH)

• Post-task reflective interviews that aloud

students to discuss challenges working with

the task, amend their prior thinking, and

recommendchanges.(SeeAppendixD).

• Rubricforquestionnairethatcan

distinguish between differing

levels of sophistication in CCC

reasoning.

• Coded interviews to ensure that

the rubric aligns to the ways

students are thinking about the

assessmenttask.

• UsingRtogenerateRaschmodel

to confirm construct leveling in

studentsCCCreasoning.

• Structuralequationtodetermine

if different constructs are

divergent.

133

AppendixG:EticCodingSchemeforInterviewData

TheeticschemeforcodingdrawsfrompriorliteratureintothenatureofCCCs(Rivetetal.,

2016;Weiseretal.,2017)aswellasintoknownavenuesofprogressioninsystemthinking

(Breslynetal.,2016;Gunckeletal.,2012;Jin&Anderson,2012:Mohanetal.,2009;Songer

etal.,2009).

• CCCsasLenses–TheroleoftheCCCistohighlightsalientfeaturesthatmaynotbe

immediatelyobviousduetoscale,scope,orsize.

• CCCsasBridges–TheroleoftheCCCistodrawconnectionsbetweentwoentities,

facilitatingtransferofunderstanding.

• CCCsasLevers–TheroleoftheCCCistocombineseveral,relatedideas,entities,or

representations in order to make understanding take on a new form towards a

particulargoal.

• CCCs as Rules – The role of the CCC is to validate a representation’s utility in

explainingorpredictingthenaturalworld.

• System Phenomena – As students build sophistication in their thinking around

systems, they are better able to describe phenomena as a system of many

simultaneousinteractions.

• System Components – As students build sophistication in their thinking around

systems,theyarebetterabletobreakdownthecomponentsofasystemintotheir

constituent,dynamicparts.

• System Relationships – As students build sophistication in their thinking around

systems, they are better able to describe the relationships between previously

identifiedcomponentsofthesystem.

• System Boundaries – As students build sophistication in their thinking around

systems,theyarebetterabletodefinetheboundariesofthesystemandtrackthe

flowsofinputsandoutputsacrossthoseboundaries.

134

AppendixH:RubricforAssessingConstructedResponsesOverallGuidanceonUsingRubric

• EachCRitemshouldbescoredfrom0to3holistically.Thatis,whileguidanceisprovided

to assess the items in terms of the claim, evidence, and reasoning of their argument,

scoresshouldcorrespondtotheoverallqualityoftheperformance.

• Ateachscorelevelforeachiteminthisrubricyouwillseeaconceptualdefinition(in

black),guidanceforwhatelementsmustappearinordertobescoredatleastashighas

thecurrentlevel(inred),andanexampleasprovidedbyastudent(inblue).

• Duetolimitationsonthesophisticationofthetask,itisnotreasonabletoexpectstudents

todisplayabroadrangeofclaimswithintheirarguments.Scorelevels2and3mayhave

similarcriteriaforthequalityofargumentativeclaimsmadebystudentsintheiroverall

response.Thedistinctionbetweenascoreof2orof3restsonthesophisticationofthe

evidenceandreasoningprovidedaswellasthelackincorrectorinaccuratestatements.

Question10:Determiningtheefficacyofharvestmenpredatorspt.2

ScoreLevel ScoringCriteria3

(Thislevelbroadlyalignstolevel4

elementsonthelearningprogression)

Student response demonstrates complex understanding of systems and systemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited,andtheirreasoning.

• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentcanaccuratelydescribeefficacyofthecontrolmechanismusingatleastoneunseen/hidden/underlyingmechanism(SM1.3.B).

o Studentresponsesatthislevelshouldclaimthatthefailureoftheharvestmenwasaresultofatleastoneofthefollowingaspectsofthesystem:

§ Therewasafinitenumberofharvestmenreleased,whichwas insufficient to consume the pre-existing number ofrootworms.

§ Therewasnotenoughtimeinaseasonfortheharvestmentoconsumeenoughrootworms.

o Additionalclaimscomparingtheefficacyoftheharvestmentothebaseline(inwhichnostrategyisimplemented)arepermittedeventhoughtheyarenotpartofthequestionprompt.

• Evidence(fromsystemcomponentsorrelationships,SM2/3):responsecitefrom changes to two or more system components that are accuratelyrelatedtotheuseoftherelevantcontrolstrategyandthedegreetowhichthosecomponentschanged(SM2.4.C+SM3.4.A).

o Studentmust cite evidence related to all three of the number ofharvestmen,thenumberofrootworms,andtheamountofcornthatsurvivestheseason.

135

o Response should include a statement of rate (i.e. how fast therootwormsareprocreatingand/orconsumingcorn).

o Studentmay also include a statement about rootworm survival–thatsomerootwormssurvivetheseasonwithoutbeingeatenbytheharvestmen.

o Additional evidence from the baseline (in which no strategy isimplemented)canbecitedwithoutpenalty.

• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovides a relevant mechanism related to a causal chain between theharvestmen,rootworms,andtheamountofcornharvested(SM4.5.D).Allthree components must be addressed. Reasoning provided must berelevanttotheclaimsstudentsmakeandtheevidencetheyhavecited.

o Student response must make reference to the followingrelationships:

§ Theharvestmenconsumesomeoftherootworms,butsomerootworms survive the season (student may but do nothavetoreferencelayingeggs).

§ If sufficientlymanyrootwormssurvive theseason(to layeggs fornextyear), thentheirpopulationcancontinue togrowevenastheharvestmenattempttoconsumethem.

§ Astherootwormpopulationgrows,theamountofcornthatsurvivesuntiltheharvestdecreases.

o Response should include a statement of rate attribution –attributing the rate (or changes to the rate) of one change inrelevant components (the corn yield and/or the rootwormpopulation)tothebehaviorofothersystemcomponents.

FromDatusability40:Theremightnothavebeenenoughharvestmentokeepthepopulationofcornrootwormsundercontrol,soeventhoughthepopulationofcornrootworms did not go up as rapidly as before, as shown in the graph data, thepopulation went up nevertheless, which also means that the number of cornharvestedstillwentdown.

2(Thislevelbroadlyalignstolevel3

elementsonthelearningprogression)

Studentresponsedemonstratesacceptableunderstandingofsystemsandsystemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited, and their reasoning. The student response includes some incorrect orinappropriateelementsbutisstillsophisticated.

• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentcanaccuratelydescribeefficacyofthecontrolmechanismusingatleastoneunseen/hidden/underlyingmechanism(SM1.3.B).

o Studentresponsesatthislevelshouldclaimthatthefailureoftheharvestmenwasaresultofatleastoneofthefollowingaspectsofthesystem:

§ Therewasafinitenumberofharvestmenreleased,whichwas insufficient to consume the pre-existing number ofrootworms.

§ Therewasnotenoughtimeinaseasonfortheharvestmentoconsumeenoughrootworms.

136

o Additionalclaimscomparingtheefficacyoftheharvestmentothebaseline(inwhichnostrategyisimplemented)arepermittedeventhoughtheyarenotpartofthequestionprompt.

• Evidence (from system components or relationships, SM2/3): responsecitefromchangestotwoormoresystemcomponentsbutdoesnotprovidearelationshipbetweenthoseevidencesandarelevantcontrolmechanism(SM2.3.C).

o Studentmustciteevidencerelatedtoatleasttwoofthenumberofharvestmen,thenumberofrootworms,andtheamountofcornthatsurvivestheseason.

o Studentmay also include a statement about rootworm survival–thatsomerootwormssurvivetheseasonwithoutbeingeatenbytheharvestmen.

o Additional evidence from the baseline (in which no strategy isimplemented)canbecitedwithoutpenalty.

o Somestatedevidencecanbeinaccurateorinappropriate,providedthatsufficientlymuchaccurate/appropriateevidenceisalsocitedinsupportoftheclaim.

• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovides a relevant mechanism related to a causal chain between theharvestmen, rootworms, and the amount of corn harvested. All threecomponents must be addressed. Response includes a mixture of bothappropriateandinappropriatesystemproperties(SM3.4.A+SM4.4.D).

o Student response must make some reference to the followingrelationships:

§ Theharvestmenconsumesomeoftherootworms,butsomerootworms survive the season (student may but do nothavetoreferencelayingeggs).

§ If sufficientlymanyrootwormssurvive theseason(to layeggs fornextyear), thentheirpopulationcancontinuetogrowevenastheharvestmenattempttoconsumethem.

§ Astherootwormpopulationgrows,theamountofcornthatsurvivesuntiltheharvestdecreases.

o Responseincludesreferencestoappropriaterelationshipsbutfailsto accurately attribute changes in the number of rootworms oramountofcornharvestedtothoserelationships.

FromDatusability42:Ithinkaddingtheharvestmentothefieldfor5yearsdidnt{sic}helpbecause{sic}ifyoulooktheharvestmenpopulationonlyhit10inyear3whiletherootwormpopulationwasstartingatyear2withaninitialpopulationof18andafinalpopulationof53.Thisshowsthatthepopulationofrootwormswasalwaysgreaterthanthepopulationofharvestmenfromthestartmakingitharderfortheharvestmentokilloffanyrootworms.

1(Thislevelbroadlyalignstolevel2

elementson

Studentresponsedemonstratesinsufficientunderstandingofsystemsandsystemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited,andtheirreasoning.Responsemaybebroadlyinaccurate,butstillarticulatetheexistenceofmultiplesystemcomponents(rootworms,rootwormeggs,corn,harvestmen,etc.)thatmayhavebeenrelevantduringthesimulation.

137

thelearningprogression)

• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentinaccuratelydescribesefficacyofthecontrolmechanism(SM1.2.B).

o Studentresponsedirectlyanswersquestionbutdoesnotrefertoanyaspects/features/behavioroftheharvestmen.

§ May include statements like “Adding harvestmen to thefielddidnothelpincreasethecornyieldpercentage”

• Evidence (from system components or relationships, SM2/3): responsecitesfromchangestoonlyasinglesystemcomponent(SM2.2.A).

o Responseonlycitesfromchangesintheamountofcornharvestedas indicativeof theefficacyof theharvestmenORresponseonlycitesfromchangesinthepopulationofrootwormsasindicativeoftheefficacyoftheharvestmen.

o Studentmayrefertorootwormsurvivalbutonlyasevidenceoflackofefficacy,notasevidenceinsupportofarelationship(i.e.doesnotconnectsurvivaltofuturegrowthofrootwormpopulation).

• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovidesarelevantmechanismrelatedtoasinglerelationshipbetweenanytwooftheharvestmen,rootworms,andtheamountofcornharvestedbutdoesn’taddressallcomponents(SM3.3.B).

o Response only describes a relationship between the harvestmenand the rootworms OR response only describes a relationshipbetweentherootwormsandthecorn.

FromDatusability9:Thenumberofcorndidnotincreasewhentheharvestmenwereputintothefield.Thisisbecuse{sic}therewerestillmanyrootwormeggsintheendoftheyear,andtheycan'tstoptherootwormscompletaly{sic}.

0(Thislevelbroadlyalignstolevel1

elementsonthelearningprogression)

Student response demonstrates little or no understanding of systems and theirresponse is missing argumentative elements. They may provide a simpledescriptionof the videoor restate information thatwaspreviouslyprovided tothemOrStudentresponseisofftopicorinappropriate.This score may also be assigned for any response that fails to meet minimumexpectationsforalevel1score.

• Student response indirectly answers question but does not include a“because”statement

FromDatusability7:Inthevideosimulater{sic}thehavestmen{sic}ateboththerootwormsanf{sic}thecornsothecornharvestedstilldecreased.

Question12:Determiningtheefficacyoftrapcroppt.2

ScoreLevel ScoringCriteria3

(Thislevelbroadly

Student response demonstrates complex understanding of systems and systemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited,andtheirreasoning.

138

alignstolevel4

elementsonthelearningprogression)

• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentcanaccuratelydescribeefficacyofthecontrolmechanismusingatleastoneunseen/hidden/underlyingmechanism(SM1.3.B).

o Studentresponsesatthislevelshouldclaimthatthesuccessofthealfalfawasa resultofat leastoneof the followingaspectsof thesystem:

§ Therootwormsatealfalfainsteadofeatingcorn.§ Eatingalfalfainsomewaypreventedtherootwormsfrom

layingeggso Additional claims comparing the efficacy of the alfalfa to the

baseline (in which no strategy is implemented) or to theharvestmenmethodarepermittedeventhoughtheyarenotpartofthequestionprompt.

• Evidence(fromsystemcomponentsorrelationships,SM2/3):responsecitefrom changes to two or more system components that are accuratelyrelatedtotheuseoftherelevantcontrolstrategyandthedegreetowhichthosecomponentschanged(SM2.4.C+SM3.4.A).

o Studentmustciteevidencerelatedtoallthreeofthepresence,thenumberof rootworms, and the amountof corn that survives theseason.

o Response should include a statement of rate (i.e. how fast therootwormsareprocreatingand/orconsumingcorn).

o Studentmayalsoincludeastatementaboutrootwormeggs.o Additional evidence from the baseline (in which no strategy is

implemented)canbecitedwithoutpenalty.• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):response

provides a relevant mechanism related to a causal chain between thealfalfa, rootworms, and the amount of corn harvested. All threecomponentsmustbeaddressed.Reasoningprovidedmustberelevanttothe claims students make and the evidence they have cited(SM4.4.B+SM2.5C).

o Studentresponsemustmakereferencetoallthreeofthefollowingrelationships:

§ Therootwormsconsumealfalfa inaddition to consumingcorn.

§ Consumingalfalfa in somewayprevents corn rootwormsfromsurvivingorprocreating.

§ Astherootwormsconsumethealfalfainsteadofthecorn,thecornyieldimproveseitherbecausethealfalfaisbeingeateninplaceofthecornorbecauseconsumingthealfalfadecreasestherootwormpopulation(andtherebydecreasesthelossofcorn).

o Response should include a statement of rate attribution –attributing the rate (or changes to the rate) of one change inrelevant components (the corn yield and/or the rootwormpopulation)tothebehaviorofothersystemcomponents.

FromDatusability12:Ithinkthatitdoes,anditismoreeffectivethanmethodone,thisisbecauseeventhoughthestartwasrough,the%didstarttogoup.Alsotherewas a higher harvest when you look at the # of corn planted to the # of corn

139

harvested.Therootwormeggsalsostartedtodecrease.Ithinkthiswasbecausetheydidn'thaveenough time toplanteggsbecause theywereeating thealfalfainsteadofthecorn,whichishowtheygrow.Inthesimulation,therootwormswerebecomingadults in the fall innstead{sic}of thesummer,whichmeant that theycouldn'talllaytheireggs,makingtheproblemlessofanissue.

2(Thislevelbroadlyalignstolevel3

elementsonthelearningprogression)

Studentresponsedemonstratesacceptableunderstandingofsystemsandsystemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited, and their reasoning. The student response includes some incorrect orinappropriateelementsbutisstillsophisticated.

• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentcanaccuratelydescribeefficacyofthecontrolmechanismusingatleastoneunseen/hidden/underlyingmechanism(SM1.3.B).

o Studentresponsesatthislevelshouldclaimthatthesuccessofthealfalfawasa resultofat leastoneof the followingaspectsof thesystem:

§ Therootwormsatealfalfainsteadofeatingcorn.§ Eatingalfalfainsomewaypreventedtherootwormsfrom

layingeggso Additional claims comparing the efficacy of the alfalfa to the

baseline (in which no strategy is implemented) or to theharvestmenmethodarepermittedeventhoughtheyarenotpartofthequestionprompt.

• Evidence (from system components or relationships, SM2/3): responsecitefromchangestotwoormoresystemcomponentsbutdoesnotprovidearelationshipbetweenthoseevidencesandarelevantcontrolmechanism(SM2.3.C).

o Studentmustciteevidencerelatedtoallthreeofthepresence,thenumberof rootworms, and the amountof corn that survives theseason.

o Response should include a statement of rate (i.e. how fast therootwormsareprocreatingand/orconsumingcorn).

o Studentmayalsoincludeastatementaboutrootwormeggs.o Additional evidence from the baseline (in which no strategy is

implemented)ortotheharvestmenmethodcanbecitedwithoutpenalty.

o Somestatedevidencecanbeinaccurateorinappropriate,providedthatsufficientlymuchaccurate/appropriateevidenceisalsocitedinsupportoftheclaim.

• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovides a relevant mechanism related to a causal chain between thealfalfa,rootworms,andtheamountofcornharvested.Allthreecomponentsmustbeaddressed.Responseincludesamixtureofbothappropriateandinappropriatesystemproperties(SM3.4.A+SM4.4.D).

o Studentresponsemustmakesomereferencetoatleastoneofthefollowingrelationships:

§ Therootwormsconsumealfalfa inaddition to consumingcorn.

140

§ Consumingalfalfa in somewayprevents corn rootwormsfromsurvivingorprocreating.

§ Astherootwormsconsumethealfalfainsteadofthecorn,thecornyieldimproveseitherbecausethealfalfaisbeingeateninplaceofthecornorbecauseconsumingthealfalfadecreasestherootwormpopulation(andtherebydecreasesthelossofcorn).

o Responseincludesreferencestoappropriaterelationshipsbutfailsto accurately attribute changes in the number of rootworms oramountofcornharvestedtothoserelationships.

FromDatusability34:Growingalfalfahelpedkeepthe%ofthecornyieldashighaspossiblebecausewhenthealfalfawasneartotherootworms,therootwormsdied,whichdecreasedtheamountofcorntheycouldeat,becausetheirwaslessofthem.

1(Thislevelbroadlyalignstolevel2

elementsonthelearningprogression)

Studentresponsedemonstratesinsufficientunderstandingofsystemsandsystemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited,andtheirreasoning.Responsemaybebroadlyinaccurate,butstillarticulatetheexistenceofmultiplesystemcomponents(rootworms,rootwormeggs,corn,alfalfa,etc.)thatmayhavebeenrelevantduringthesimulation.

• Claim(aboutsystemphenomena,SM1):responseaddressesonlyoneofthe“reducethecornrootworm”or“keepthecornyieldhigh”componentsofthe question. Response shows that the student inaccurately describesefficacyofthecontrolmechanism(SM1.2.B).

o Studentresponsedirectlyanswersquestionbutdoesnotrefertoanyaspects/features/behaviorofthealfalfa.

§ Mayincludestatementslike“Plantingalfalfainthefielddidnothelpincreasethecornyieldpercentage”

• Evidence (from system components or relationships, SM2/3): responsecitesfromchangestoonlyasinglesystemcomponent(SM2.2.A).

o Responseonlycitesfromchangesintheamountofcornharvestedas indicative of the efficacy of the alfalfaORresponse only citesfromchangesinthepopulationofrootwormsasindicativeoftheefficacyofthealfalfa.

o Studentmayrefertothetrade-offbetweenlandusedtoplantcornandlandusedtoplantalfalfa,butonlyasevidenceoflackofefficacy,notasevidenceinsupportofarelationship(i.e.doesnotconnectsurvivaltofuturegrowthofrootwormpopulation).

• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovidesarelevantmechanismrelatedtoasinglerelationshipbetweenanytwoofthealfalfa,rootworms,andtheamountofcornharvestedbutdoesn’taddressallcomponents(SM3.3.B).

o ResponseonlydescribesarelationshipbetweenthealfalfaandtherootwormsORresponseonlydescribesarelationshipbetweentherootwormsandthecorn.

FromDatusability41:Thealfalfaplantedinthecornfielddoeshelpthecornyieldashighasitcanbe.Andeveninyear3-4therewasanincreaseofcornharvested.

141

Howeverthereisthedownsideofnotplantingasmuchcornaspossiblebutthismethodseemedtoworkbetterthanthefirstone.

0(Thislevelbroadlyalignstolevel1

elementsonthelearningprogression)

Student response demonstrates little or no understanding of systems and theirresponse is missing argumentative elements. They may provide a simpledescriptionof the videoor restate information thatwaspreviouslyprovided tothemOrStudentresponseisofftopicorinappropriate.This score may also be assigned for any response that fails to meet minimumexpectationsforalevel1score.From Datusability9: Yes, becuse {sic} the results produce a more suggestingnumberthatthecornisharvestingmorewhenthereisAlfalfainit.Thisisbecuse{sic}theAlfafa{sic}isstrongerandcangetridofmorerootworms.

Question13:Drawingaconclusionaboutthestrategies

ScoreLevel ScoringCriteria3

(Thislevelbroadlyalignstolevel4

elementsonthelearningprogression)

Student response demonstrates complex understanding of systems and systemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited,andtheirreasoning.

• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentcanaccuratelydescribeefficacyofthecontrolmechanismusingatleastoneunseen/hidden/underlyingmechanism(SM1.3.B).

o Studentresponsesatthislevelshouldclaimthatthesuccessofthealfalfawillorwillnotcontinueintofutureyearsbecauseofatleastoneofthefollowingaspectsofthesystem:

§ The alfalfa was successful in reducing the rootwormpopulationinyears3-5

§ Thecornrootwormpopulationincreasedwhenalfalfawasnotplanted

o Response should include a statement of prediction in which thestudent asserts that the strategy will/will not continue to beeffectiveinthefuture.

o Additional claims comparing the efficacy of the alfalfa to thebaseline (in which no strategy is implemented) or to theharvestmenmethodarepermittedeventhoughtheyarenotpartofthequestionprompt.

• Evidence(fromsystemcomponentsorrelationships,SM2/3):responsecitefrom changes to two or more system components that are accuratelyrelatedtotheuseoftherelevantcontrolstrategyandthedegreetowhichthosecomponentschanged(SM2.4.C).

o Student must cite evidence related to all three of the presencealfalfa, the number of rootworms (or rootworm eggs), and theamountofcornthatsurvivestheseason.

142

o Response may also include a statement regarding balance orequilibrium(i.e.“heldtherootwormsatbay”).

o Studentmayalsoincludeastatementaboutrootwormeggs.o Additional evidence from the baseline (in which no strategy is

implemented)ortotheharvestmenmethodcanbecitedwithoutpenalty.

• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovides a relevant dynamism (ability to change in the future) to theconsequencesofacausalrelationshipbetweenthealfalfa,rootworms,andtheamountof cornharvested.All threecomponentsmustbeaddressed.Reasoningprovidedmustberelevanttotheclaimsstudentsmakeandtheevidencetheyhavecited(SM4.5.D+SM4.5.E).

o Student response must make reference to both of the followingrelationships:

§ The rootworms population grows in the absence of acontrolstrategy

§ Consumingalfalfa in somewayprevents corn rootwormsfromsurvivingorprocreating.

o Responsemust includea statementofpredictability inwhich thestudent asserts that evidence from prior years can be used toestimatethefuturestateofthesystem.

o Response may include a statement of dynamism in which thestudent asserts that aspects of the system (like number ofrootwormsoramountofcornsurvivingtoharvest)aresubjecttochangeinresponsetochangestothesystem(i.e.theintroductionofacontrolstrategy).

FromDatusability6:Jonahcancontinuetoplantalfalfainyear7.Hecancontinuethatbecauseithelpskeepingthecornhealthyandkeepingthecornrootwormsaslowaspossible. In thedata table it shows that fromyear1 toyear2 thecorndecreasedbyalotwhenhedidnt{sic}usethealfalfa.Butwhenheusedit,itactuallyincreased.Intheotherdatatableoftherootwormeggsitshowsthatfromyear1toyear2theeggsincreaseed{sic}whenhedidnt{sic}plantalfalfa.Butfromyear3toyear5therootwormsdecreased.Inconclusion.Jonahshouldusethealfalfainyear7.

2(Thislevelbroadlyalignstolevel3

elementsonthelearningprogression)

Student response demonstrates complex understanding of systems and systemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited,andtheirreasoning.

• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentcanaccuratelydescribeefficacyofthecontrolmechanismusingatleastoneunseen/hidden/underlyingmechanism(SM1.3.B).

o Studentresponsesatthislevelshouldclaimthatthesuccessofthealfalfawillorwillnotcontinueintofutureyearsbecauseofatleastoneofthefollowingaspectsofthesystem:

§ The alfalfa was successful in reducing the rootwormpopulationinyears3-5

§ Thecornrootwormpopulationincreasedwhenalfalfawasnotplanted

143

o Response should include a statement of prediction in which thestudent asserts that the strategy will/will not continue to beeffectiveinthefuture.

o Additional claims comparing the efficacy of the alfalfa to thebaseline (in which no strategy is implemented) or to theharvestmenmethodarepermittedeventhoughtheyarenotpartofthequestionprompt.

• Evidence (from system components or relationships, SM2/3): responsecitefromchangestotwoormoresystemcomponentsbutdoesnotprovidearelationshipbetweenthoseevidencesandarelevantcontrolmechanism(SM2.3.C).

o Studentmustciteevidencerelatedtoall threeof thepresenceofalfalfa, the number of rootworms (or rootworm eggs), and theamountofcornthatsurvivestheseason.

o Response may also include a statement regarding balance orequilibrium(i.e.“heldtherootwormsatbay”).

o Studentmayalsoincludeastatementaboutrootwormeggs.o Additional evidence from the baseline (in which no strategy is

implemented)ortotheharvestmenmethodcanbecitedwithoutpenalty.

o Somestatedevidencecanbeinaccurateorinappropriate,providedthatsufficientlymuchaccurate/appropriateevidenceisalsocitedinsupportoftheclaim.

• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovides a relevant dynamism (ability to change in the future) to theconsequencesofacausalrelationshipbetweenthealfalfa,rootworms,andtheamountof cornharvested.All threecomponentsmustbeaddressed.Response includes a mixture of both appropriate and inappropriatepotentialrelationshipdynamics(SM3.4.B+SM4.4.E).

o Student response makes reference to only one of the followingrelationships:

§ The rootworms population grows in the absence of acontrolstrategy

§ Consumingalfalfa in somewayprevents corn rootwormsfromsurvivingorprocreating.

o Response may include a statement of dynamism in which thestudent asserts that aspects of the system (like number ofrootwormsoramountofcornsurvivingtoharvest)aresubjecttochangeinresponsetochangestothesystem(i.e.theintroductionofacontrolstrategy).

FromDatusability13:Therootwormsdecreasedintheamountoftimethealfalfawasbeingused.Thismethodwouldprobablycontinuetoworkbecause theeggcounthasalreadygonedown,andwhentheeggcountgoesdown,morecorncanbeharvested.Therewillbelessrootwormstoeatthecorn.

1(Thislevelbroadlyalignsto

Studentresponsedemonstratesinsufficientunderstandingofsystemsandsystemmodeling(?) within the three main parts of an argument: claim, evidence, andreasoning.

144

level2elementsonthelearningprogression)

• Claim(aboutsystemphenomena,SM1):responseincludesaclaimthatonlyaddresses the state of the alfalfa. Response shows that the studentinaccuratelydescribesefficacyofthecontrolmechanism(SM1.2.B).

o Response should include a statement of prediction in which thestudent asserts that the strategy will/will not continue to beeffective in the future but does not refer to anyaspects/features/behaviorofthealfalfa.

§ Mayincludestatementslike“Plantingalfalfainthefielddidnothelpincreasethecornyieldpercentage”

• Evidence (from system components or relationships, SM2/3): responsecitesfromchangestoonlyasinglesystemcomponent(SM2.2.A).

o Responseonlycitesfromchangesintheamountofcornharvestedas indicative of the efficacy of the alfalfaORresponse only citesfromchangesinthepopulationofrootwormsasindicativeoftheefficacyofthealfalfa.

o Studentmayrefertothetrade-offbetweenlandusedtoplantcornandlandusedtoplantalfalfa,butonlyasevidenceoflackofefficacy,notasevidenceinsupportofarelationship(i.e.doesnotconnectsurvivaltofuturegrowthofrootwormpopulation).

o Responsemaycitedirectlyfromthegraphswithoutconnectiontoacausalrelationshipormechanism.

• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovidesarelevantmechanismrelatedtoasinglerelationshipbetweenanytwoofthealfalfa,rootworms,andtheamountofcornharvestedbutdoesn’taddressallcomponents(SM3.3.B).

o ResponseonlydescribesarelationshipbetweenthealfalfaandtherootwormsORresponseonlydescribesarelationshipbetweentherootwormsandthecorn.

o Student overemphasizes slight variation in years 4, 5, and 6 toassertthatthecontrolstrategyisnolongereffective.

o Response may include an inaccurate statement of dynamism inwhichthestudentassertsthataspectsofthesystem(likenumberof rootworms or amount of corn surviving to harvest) will notchange despite changes to other aspects of the system (like thecessationofcontrolstrategyimplementation).

FromDatusability40: Ithinkthatgrowingalfalfainyear7tohelpcontrolcornrootwormswill not help, because although the number of corn rootworms diddecreasewhen the alfalfawas planted in years 3 and 4, the numbers began toincreaseagain inyear5,and if it continuesat thatsamerate,plantingalfalfa inyears6and7willnotcontinuetohelp.

0(Thislevelbroadlyalignstolevel1

elementson

Student response demonstrates little or no understanding of systems and theirresponse is missing argumentative elements. They may provide a simpledescriptionof the videoor restate information thatwaspreviouslyprovided tothemOrStudentresponseisofftopicorinappropriate.

145

thelearningprogression)

This score may also be assigned for any response that fails to meet minimumexpectationsforalevel1score.FromDatusability7: On the second chart rootworms goes up and down so hecannotpredictwhatwillhappennextyearandforthatreasonhecannotcontroltheamountofrootworms

146

AppendixI:RScriptforStructuralEquationandRaschModellibrary(psych) library(GPArotation) library(eRm) library(mixRasch) library(dplyr) library(haven) library(lavaan) library(QuantPsyc) library(MASS) library(Hmisc) library(readxl) library(semPlot) #Read in data DATXecosystemUsabilityData <- read_excel("Dissertation Stuff/DATX Score Analysis.xlsx") ItemScoresOnly<- dplyr::select(DATXecosystemUsabilityData,Q2_SUM:Q5,Q6_SUM,Q7_SUM,Q8:Q13) #checking for unidimensionality scree(ItemScoresOnly) pca(ItemScoresOnly, nfactors = 2,residuals=TRUE) #sem cfa model <- ' #measurement model StructureFunction =~ Q8 + Q7_SUM + Q6_SUM + Q5 + Q4 + Q3 + Q2_SUM SystemModels =~ Q13 + Q12 + Q11 + Q10 + Q9 #regressions StructureFunction ~~ SystemModels StructureFunction ~ Grade SystemModels ~ Grade ' fit<-lavaan::sem(model, data = DATXecosystemUsabilityData) summary(fit, standardized=TRUE) semPlot::semPaths(fit) semPlot::semPaths(fit,what="path",whatLabels = "par") #Cronbach Alpha psych::alpha(ItemScoresOnly) psych::alpha(dplyr::select(DATXecosystemUsabilityData,Q2_SUM:Q5,Q6_SUM,Q7_SUM,-Q8)) psych::alpha(dplyr::select(DATXecosystemUsabilityData,Q9:Q13)) #Rasch modeling-Combined RaschCombined<-eRm::PCM(dplyr::select(DATXecosystemUsabilityData,Q2_SUM:Q5,Q6_SUM,Q7_SUM,Q8:Q13)) #Average Difficulty and Threshold position thresholds(RaschCombined) #Item Category Curves plotICC(RaschCombined, mplot = TRUE, legpos = FALSE, ask = FALSE) #Person Item Map plotPImap(RaschCombined,sorted=TRUE,main="Person-Item Map of Combined Behavior Items")

147

#item-fit statistics itemfit(person.parameter(RaschCombined)) ######Examining Fit Statistics RaschCombinedPersons<-person.parameter(RaschCombined) RaschCombinedPersonResid<-residuals(RaschCombinedPersons) scree(RaschCombinedPersonResid) #some indication that unidimensionality assumption fails CombinedTheta<-RaschCombinedPersons$theta.table ####Repeat with SF Separated RaschSF<-eRm::PCM(dplyr::select(DATXecosystemUsabilityData,Q2_SUM:Q5,Q6_SUM,Q7_SUM,Q8)) #Average Difficulty and Threshold position thresholds(RaschSF) #Item Category Curves plotICC(RaschSF, mplot = TRUE, legpos = FALSE, ask = FALSE) #Person Item Map plotPImap(RaschSF,sorted=TRUE,main="Person-Item Map of Structure Function Items") #item-fit statistics itemfit(person.parameter(RaschSF)) ######Examining Fit Statistics RaschSFPersons<-person.parameter(RaschSF) RaschSFPersonResid<-residuals(RaschSFPersons) scree(RaschSFPersonResid) #some indication that unidimensionality assumption fails SFTheta<-RaschSFPersons$theta.table ####Repeat with SSM Separated RaschSSM<-eRm::PCM(dplyr::select(DATXecosystemUsabilityData,Q9:Q13)) #Average Difficulty and Threshold position thresholds(RaschSSM) #Item Category Curves plotICC(RaschSSM, mplot = TRUE, legpos = FALSE, ask = FALSE) #Person Item Map plotPImap(RaschSSM,sorted=TRUE,main="Person-Item Map of Systems and System Model Items") #item-fit statistics itemfit(person.parameter(RaschSSM)) ######Examining Fit Statistics RaschSSMPersons<-person.parameter(RaschSSM) RaschSSMPersonResid<-residuals(RaschSSMPersons) scree(RaschSSMPersonResid) #some indication that unidimensionality assumption fails SSMTheta<-RaschSSMPersons$theta.table #Wilcoxon Signed Rank Test for difference between Thetas wilcox.test(SFTheta$`Person Parameter`,SSMTheta$`PersonParameter`,paired = FALSE)

148

AppendixJ:DefinitionsofSomeKeyTerms

• CognitiveLabs(orCogLabsorUsabilityTests)–Inassessmentdesign,acriticalstepinthedevelopmentofacomplexinstrument(suchasthoseusedintheDAT-CROSSprojects) are guided interviews (typically think-alouds), surveys, and item-responseanalysesthatexaminethedegreetowhichtheinstrumentiseffectivelyengagingrespondentsinthekindsofcognitionrelevanttothemeasuredconstructwithoutposingunnecessarychallengesthatmaysystematicallyprohibitthesubjectfrom acting toward their true ability. Drawing on the focus of diminishing non-constructrelevantbarrierstoparticipants’useoftheinstrument,theprocessmayalsobereferredtoas‘usabilitytesting’(Zaharias&Poylymenakou,2009).

• Models– In teachingparlance,modelling isan instructionalbehavior (i.e. the teachermodels thebehaviorshewishes thestudent toenact).However,models in thispaperrefer to scientific models – means of representing science knowledge by mappingabstractscienceconceptsontopictures,words,orotherstructures.Modelsservemanypurposes in science and some philosophers argue that all science knowledge can bethoughtofintermsofthevariousrepresentationsscientistsusetoexplain,describe,andpredict the naturalworld.More onmodels in the NGSS can be found in Kracjik andMerritt(2012).

• Performance Expectations – The standards in the NGSS are not a mere listing ofimportantideasinscience(thoughthisdimensiondoesappearasDCIs)becauseacentralthemeoftheNGSSisthatwhatstudentsdoordonotknowandonlybeevaluatedintermsoftheoperationalizationofthatknowledge.PerformanceexpectationsintheNGSSdetailwaysstudentscanoperationalizetheirscienceknowledgeviasomeactivity.

• Practices – In this paper, “practices” refer to the suite of activities authentic to thescientificcommunityoutlinedintheScienceandEngineeringPractices(SEPs)dimensionoftheFramework(NRC,2012).Theseactivitiesrepresentthevariouswaysthatscientistsengagewiththeirscienceknowledge,andfeatureasadominantpartofthelanguageofeachperformanceexpectationoftheNGSS.

• Scripts–ThefociofinvestigationinEngestrom’s(2000)activitytheoryanalyticalframeworkarethedevelopedschemaforhowpeopleengageinvariousactivitiesthatmakeupbotheverydayandprofessional life.Engestromcalls these schemascripts.Muchlikeascriptforanactor,activityscriptsareconstructedpriortoanyspecific instanceof anactivity inorder to informhow toproceedas theactivityoccurs.Justastheproductionofaplaycanbedisruptedinsuchawayastopushanactor off-script, so too candisruptions in the learning settingpush a student offscript. While there can be value in disrupting a script for the purpose ofconstructivist learning, such disruptions are an impediment to effectivemeasurementofstudents’currentstatesofunderstanding.

149

• Storyboards – The development of rich performance assessments entails themovement through a design process that starts at a very general level (domainmodeling),movestoincludespecificsofwhatperformancesmightbepossible(taskmodels), and culminates as a sequence of stimuli (including written prompts,videos,andimages)andresponsecollections.Thisseriesofstimuliandresponse,thelastdesignstagebeforeataskbecomeoperational,comprisesadocumentcalledthe “storyboard” (NRC, 2014) which details the specifics of what students willinteractwithastheyengageinthetask.