A COMPENDIUM OF SCALES for use in the

SCHOLARSHIP OF TEACHING AND LEARNING

EDITED BY: Rajiv S. Jhangiani

Jordan D. Troisi Bethany Fleck

Angela M. Legg Heather D. Hussey

2

AcknowledgementsWefirstthankJenniferStiegler-Balfourforhercontributionsduringtheconceptualizationofthise-book.Wealsothankthefollowingresearchassistantsfortheirassistance:TamakiSasa,MayReinert,JonathanVargas,NatalieClark,NadyaHall,MisaelMartinez,andMelissaAdolphefromthePaceUniversitySHARPPLab,andRebeccaDeutschmannandEmilieLangfromtheKPUSocial&PoliticalCognitionLab.Coverimage:"Measureathousandtimes,cutonce"bySonnyAbesamis(CC-BY2.0)

3

TableofContents

Chapter1:Introduction..................................................................................................................5RajivS.Jhangiani,JordanD.Troisi,BethanyFleck,AngelaM.Legg,andHeatherD.Hussey

Section1:Choosing,Using,Developing,andValidatingScales

Chapter2:BestPracticesinScaleUseinSoTL..............................................................................11ReganA.R.Gurung

Chapter3:SelectingtheRightScale:AnEditor’sPerspective......................................................20AndrewN.Christopher

Chapter4:APrimeronScaleDevelopment..................................................................................29HeatherD.HusseyandTaraJ.Lehan

Chapter5:TheStateofScaleValidationinSoTLResearchinPsychology....................................44GeorjeannaWilson-Doenges

Section2:ScalesforUseintheScholarshipofTeachingandLearningChapter6:MeasuringLearningandSelf-Efficacy.........................................................................55PamMarek,AdrienneWilliamson,andLaurenTaglialatela

Chapter7:MeasuringCriticalThinkingSkills...............................................................................74R.EricLandrumandMaureenA.McCarthy

Chapter8:StudentEngagementTowardCoursework:Measures,Considerations,andFutureDirections.....................................................................................................................................87KevinL.ZabelandAmyHeger

Chapter9:MeasuringService-LearningandCivicEngagement.................................................102LoriSimons

Chapter10:MeasuringIndividualDifferencesinEpistemologicalBeliefs..................................123KellyL.Y.Ku

Chapter11:MeasuringWell-BeingintheScholarshipofTeachingandLearning......................133KristinLayous,S.KatherineNelson,andAngelaM.Legg

4

Chapter12:AssessingProfessor-StudentRelationshipsUsingSelf-ReportScales.....................149JennaM.MeyerbergandAngelaM.Legg

Chapter13:EffectiveToolsforAssessingCharacteristicsofExcellentTeaching:TheTeacherBehaviorsChecklistasExemplar................................................................................................161ClaireKirk,JessicaBusler,JaredKeeley,andWilliamBuskist

Chapter14:SoTLScales:TheCaseofMissingLinks...................................................................174AaronS.Richmond

5

Chapter1:IntroductionRajivS.Jhangiani1,JordanD.Troisi2,BethanyFleck3,AngelaM.Legg4,andHeatherD.Hussey51KwantlenPolytechnicUniversity,2Sewanee:TheUniversityoftheSouth,3MetropolitanStateUniversityofDenver,4PaceUniversity,5NorthcentralUniversity

Thescholarshipofteachingandlearning(SoTL)hasincreasedinbothprevalenceandprofileduringthepastdecade(Bishop-Clark&Dietz-Uhler,2012;Gurung&Landrum,2015;Gurung&Wilson,2013).Overthistime,SoTLworkhasbecomemoremethodologicallyrigorousandmoreacceptedbyuniversityadministratorsasvalidandvaluableproductsofscholarship.Givenitsstrongempiricalfoundationandlonghistoryofbasicresearchsuchascognitive,learning,behavioral,andsocial,psychologyasadisciplineisespeciallywell-positionedtoleadinvestigationsintopracticesthatenhancetheeffectivenessofteachingandlearning.Withastatedmissionto“promoteexcellenceintheteachingandlearningofpsychology,”theSocietyfortheTeachingofPsychology(STP)hasbeenattheforefrontofthismovementwithinourdiscipline.STPhassupportedSoTLbyawardinggrants(e.g.,theSoTLgrant),developingdemonstrablyeffectiveteachingresources(e.g.,instructionalresourceawards),organizingconferencesandmeetings(e.g.,theAnnualConferenceonTeaching),andeffectivelydisseminatingresearchfindings(e.g.,publicationinitsflagshipjournalTeachingofPsychology).Thise-bookisintendedtofurthersupporttheseeffortsbyprovidingavaluableresourcethatfacilitatesthelocation,evaluation,selection,and(wherenecessary)developmentofpsychometricallysoundscalesforthemanytraditionalareasoffocuswithinSoTL.Indoingso,thiscompendiumwillachievethebroadergoalofraisingthescientificstandardsofevidence-basedteachingandlearning.Aseditorsofthise-book,we—themembersoftheSocietyfortheTeachingofPsychology’sEarlyCareerPsychologists(ECP)committee—identifiedtherelevanttopicareasandinvitedwell-establishedSoTLresearcherswithinthoseareastocontributechapters.AsECPs,werecognizedtheneedtoservenewfacultymembersand,inparticular,theimportanceoffocusingonpedagogywhilefacilitatingSoTL.However,althoughthise-bookisclearlyhelpfultothosejustgettingstartedinthisareaofresearch,itwillbeequallyvaluabletoseasonedresearchers.SoTLresearchcoversabroadrangeoftopicsincludingcriticalthinking,metacognition,professor-studentrelationships,andstudentperceptionsoflearningandteaching.Thiscompendiumcoverseachofthesetopics,alongwithmanyothersthatareattheforefrontofSoTLresearch.Whereasaveteranresearchermightbefamiliarwithsomeoftheseareas,theywillstillbenefitfromlearningmoreaboutothers,aswellaspotentiallynewSoTLtools.OrganizationofthisE-BookOrganizedbytopic,thiscompendiumcontainsscaledescriptions,validationinformation(ifavailable),andreferencessoscholarscanexaminepastresearchthatusedeachscale.Inaddition,theauthors—eachwellestablishedwithintheirareaoffocus—provideadviceon

6

choosingappropriatescales,developingscales,andthetypesofscalestheSoTLliteraturestillneeds.Thefirstsectionofthise-bookfocusesontheselection,use,development,andvalidationofscales.InChapter2,ReganGurungdiscussesseveralbestpracticesconcerningscaleuseandchoice,includingidentifyingrelevantpsychologicalvariablesthatinfluencelearningandusingpublishedscalesinsteadofcobblingtogetheranon-validatedmeasure.Thechapterconcludeswithareviewofcriteriaforselectinganappropriatescaleand(wherenopublishedscaleisavailable)writingyourownitems.InChapter3,AndrewChristophercomplementsthebestpracticeschapterwithadviceonhowtoselecttherightscale,fromhisperspectiveasthecurrenteditorofTeachingofPsychology.Thenexttwochaptersfocusspecificallyonscaledevelopmentandvalidation.InChapter4,authorsHeatherHusseyandTaraLehanprovideabrief,accessibleguidetothescaledevelopmentprocess.Theysuccinctlydescribetheearlystagesofscaledevelopmentsuchasconductingliteraturereviews,creatingitems,pilottesting,andrevisingitems.Theyalsoprovideanexcellentsummaryofcommonreliabilityandvaliditytests,whichwillproveparticularlyusefultoanyonenewtothescalevalidationprocess(orevenifyoujustneedarefresher).Section1concludeswithacommentaryonthestateofscalevalidationinSoTLresearch(Chapter5),inwhichGeorjeannaWilson-DoengesprovidessomeexemplarsofwaysinwhichSoTLresearchershaveadoptedthebestpracticesforscalevalidationwhileoperatingwithinthecommonconstraintsofsamplesize,classcomposition,andsemesterlength;allofwhichareperennialissuesamongSoTLresearchers.ThechaptersinSection2makeupthebulkofthee-bookandpresentatopicalselectionofscaleinformation:InChapter6,PamMarek,AdrienneWilliamson,andLaurenTaglialateladiscussthemeasurementofstudentlearningandself-efficacy.Theauthorsdescribebothformativeassessments(e.g.,classroomassessmenttechniques)andsummativeassessments(e.g.,writingassignmentsandstandardizedtests),beforeconcludingwithareviewofmeasuresofperceivedlearningandperceivedself-efficacy.ThemeasurementofcriticalthinkingskillsisaddressedinChapter7byEricLandrumandMaureenMcCarthy,whoreviewmainstreammeasuresofcriticalthinkingthatarespecifictopsychologyaswellasbroad-basedgeneralmeasures.ThischapterconcludeswithasetofrecommendationsforhowSoTLresearchersmightbalancethedesiresforefficiencyandvalidityinthemeasurementofthiscomplexconstruct.Measuresofstudentengagementtowardcourseworkatbothmacro-andmicro-levelsarereviewedinChapter8byKevinZabelandAmyHeger.Thelatterincludesdescriptionsofmeasuresofstudentinterest,studentengagement,aswellasancillarymeasuressuchasgrit

7

andboredom;whereastheformerincludesmoregeneralmeasure(e.g.,nationalsurveyofstudentengagement).LoriSimonssystematicallyexamines21quantitativemeasuresofservicelearningandcivicengagementinChapter9,includingmeasuresofserviceimpactsoncommunity,facultyperceptionsofservice,andserviceimpactsonstudents.Alongwithdistinguishingbetweenscaleswithandwithoutpsychometricevidence,Simonsprovidesadviceforeducatorsinterestedinmeasuringcivicengagementasastudentlearningoutcome.Students’epistemologicalbeliefsarethefocusofChapter10,writtenbyKellyKu.Byprovidingatheoreticalbackgroundaswellascommonmeasurestoassesshowstudentsconceptualizethenatureofknowledge,Kuaddressesquestionssuchas:Dostudentsbelieveknowledgeisrelativeandchangeableordotheyviewitinabsoluteterms?Howdotheyviewtheperspectivesandopinionsorauthoritiesandwhenaretheyabletoseeambiguityinknowledgerepresentations?InChapter11,KristinLayous,S.KatherineNelson,andAngelaM.Leggproposethatstudents’psychologicalwell-beingisanessentialfactorinunderstandingtheirexperienceintheclassroom.Theauthorsprovideanoverviewofscalesthatassessdifferentaspectsofwell-being,includingbothpositive(e.g.,lifesatisfaction,subjectivewell-being,meaninginlife)andnegative(e.g.,generalanddomain-specificstressandanxiety)aspects.Professor-studentrelationshipsarethefocusofChapter12,contributedbyJennaMeyerbergandAngelaM.Legg.Theseauthorsbeginbyprovidingaconceptualframeworkforunderstandingthepositiveimpactonstudentoutcomesofimmediacybehaviors,professor-studentrapport,andthelearningalliance,beforereviewingmeasuresofallthreeconstructs.InChapter13,ClaireKirk,JessicaBusler,JaredKeeley,andWilliamBuskistreviewseveralapproachestoassessingprofessorefficacy,includingbycollectingstudentandpeerfeedbackandevaluatingteachingmaterials,beforeprovidinganin-depthexaminationoftheTeacherBehaviorsChecklistasanexemplartool.Thee-bookconcludeswithathoughtful(andinourmindshumorous)chapterbyAaronRichmond,whohighlightsthetypesofscalestheliteraturestillneeds(Chapter14),includingmeasuresofmetacognitionandlearningstrategies,scalestoassesssyllabiandmodelteachingcharacteristics,aswellasvalidbehavioralmeasurestoaccompanythemanyself-reportSoTLscales.RichmondconcludeswithaseriesoffiverecommendationsthatrepresentacalltoactionforSoTLresearch.Byhighlightingthemostprominentscalesonarangeoftopics,thise-bookservesasanimportantreferenceguidewithinthescholarshipofteachingandlearning.Furthermore,thepracticaladvicethateachexpertauthorhasprovidedwillsurelyenhancetherigorofscholarlyworktofollow.Wehopethatthiscompendiumprovidesausefultoolforearlycareerpsychologistandseasonedresearchersalike.

8

SuggestionsforReadingthisE-BookTheorganizationofthise-bookallowsreaderstopicktheirownstartingpoint.WhereastheChaptersinSection1provideexcellentandpracticaladviceonaspectsofthescaleselectionanddevelopmentprocess,eachchapterinSection2providesastandaloneaccountofthattopic’svalidatedscales,permittingreaderstouseeachchapterasitsownreferenceguide.Intrackingdowntheseexistingscales,readersmaybepleasedtodiscoverthatwithinthereferencessectionofeachchapter,anasteriskisplacednexttothereferenceforeachscalediscussedwithinthatchapter.Ourhopeisthatemergingandwell-establishedSoTLresearchersalikewillfindgreatvalueinthegeneralandtopic-specificguidancewithinthise-bookregardingthebestpracticesinscaledevelopment,validation,anduse.Foreducators,thisbookcanalsoserveasanexcellentsupplementarytextforcoursessuchastestsandmeasurements,researchmethods,andeducationalassessment.Manyofthechaptersprovideaccessibledescriptionsthatgraduateandevenundergraduateaudienceswillappreciate.Veryfewresourcesexistthatprovidea“compareandcontrast”presentationofavarietyofmeasurementtools,especiallywithintheSoTLliterature.Thechaptersinthise-bookprovidestudentswithexemplarsofthescaledevelopmentandvalidationprocesswhileofferingacurrentaccountofhowSoTLresearchersassessdifferentconstructs.Asanadditionaltool,manyofourauthorsalsoprovidepedagogicalsuggestionsforteachingstudentsaboutthescalestheydiscuss.Finally,asourauthorsnoteintheirindividualchapters,manygapsstillexistinthedevelopmentofvalidatedscalesforSoTLuse.Thus,ourfinalsuggestionforreadersistotakeinspirationfromtheextantliteratureandtakeupthechallengeofaddingtothefield’sincreasingrelianceonvalidatedmeasures.Itisourhopethat,upondevelopingandvalidatingthesemuch-neededscales,thise-bookwillrequireasecondeditiontoupdatechapters,addnewchapters,andreconsiderthestateofscalevalidationanduseinSoTLwork.

9

ReferencesBishop-Clark,C.&Dietz-Uhler,B.(2012).Engaginginthescholarshipofteachingandlearning:

Aguidetotheprocess,andhowtodevelopaprojectfromstarttofinish.Hemdon,VA:StylusPublishing.

Gurung,R.A.R.,&Landrum,R.E.(2015).Editorial.ScholarshipofTeachingandLearninginPsychology,1(1),1-6.

Gurung,R.A.R.,&Wilson,J.H.(Eds.).(2013).Doingthescholarshipofteachingandlearning:Measuringsystematicchangestoteachingandimprovementsinlearning.SanFrancisco:Jossey-Bass.

Section1:Choosing,Using,Developing,andValidatingScales

Chapter2:BestPracticesinScaleUseinSoTLReganA.R.Gurung

UniversityofWisconsin-GreenBay

"Measurewhatismeasurable,andmakemeasurablewhatisnotso"(GalileoGalilee)Measurementisatthecoreofrobustscholarshipofteachingandlearning(SoTL,Gurung&Landrum,2012).Ismyteachingeffective?Aremystudentslearning?Whatpredictsstudentlearning?Mostempiricalprojectsandquantitativeapproachestopedagogicalresearchinvolvemeasurementandifthereisanexistingscaletomeasurewhatyouwantto,whyreinventthewheel?InthischapterIwillreviewsomekeyissuestokeepinmindwhenpickingandusingscales.IwilloverviewconstructvaliditysopedagogicalresearchersrememberwhattolookforinascaleandIwilltouchonbestpracticesinwritingyourownitems.Beforedivingin,itisimportanttoclarifysometerminologyusage.Indoingdescriptivestudiesresearchersmeasuremanydifferentvariablesanduseanumberofdifferentresearchdesigns(Bartch,2013;Schwartz&Gurung,2012).Indescriptivestudies,wewanttogetapictureofwhatisgoingoninclassroomorteaching(e.g.,Howmanystudentscanlistthemajorapproachestopsychology?).Incorrelationalstudieswewanttomeasureassociationsbetweenvariables(e.g.,Arethestudentswhotookmorepracticequizzesonthechapterbetteratlistingthemajorapproachestopsychology?).Inexperimentaldesignswewanttocollectevidencetoseeifthechangesweimplemented,toassignments,lectures,ordesign,resultedinincreasesinlearning(e.g.DidthegroupofstudentswhowatchedmyspecialIntrotoPsychvideolistmoreofthemajorapproachestopsychology?).Foreachofthesemajordesigns,descriptive,correlational,andexperimental,thereareanumberofwaystomeasurevariables.Asidefromobservationorconductingfocusgroups,whichrequiredifferentformsofmeasurement,themajorityofpedagogicalresearchinpsychologyinvolvesassessingstudents’attitudesandbehaviorswithsurveysandmeasuringtheirlearningwithexams,quizzes,andotherperformanceoutcomes.Surveysorquestionnairesaregeneraltermsandeachsurveycancontainmanyitemsorquestions.Sometimestheresearchergeneratesthequestionsthemselves.Oftentheresearcherusesapreexistingpublishedscale.Asurveycanhencecompriseofmanyscales,manyquestions(i.e.,neverbeforebeenusedorpublished),oracombinationofboth.Therearebestpracticestoboththeselectionanduseofscales,andthecreationofyourownitemsorquestions.Iwillfirstcoversomebestpracticesregardingscaleuseandchoiceandthenbrieflyreviewpointersforitemconstruction. MeasuretheUsualSuspects:KeyVariablesinLearningThereisalonghistoryofresearchonteachingandlearning(Gurung&Schwartz,2012).Awiderangeofdisciplinesstudythefactorsinfluencinglearning,withpsychologyandeducationplayingamajorrole,althoughtherearealsosignificantstudiesoftheneuroscienceoflearning(Doyle&Zakrajsek,2013).Theaveragepedagogicalresearcherwillnotbepoppingstudents

12

intofMRImachinestoobservewhichnervecellsorareasofthebrainfireduringthinking,soareviewofthebiologicalformsofmeasurementcanbecircumvented.Incontrast,mostpsychologicalSoTLstudiesusesurveysofsomeformortheother.Althoughyoumayneedtowritesomespecificquestionstogetatuniqueresearchquestionsyouhave,thereareawidevarietyofexistingscalesthathelpmeasuresomecommonlearningvariables.Alargebodyofacademicliteratureidentifieskeyfactorsinfluencinglearning(Credé&Kuncel,2008;NationalResearchCouncil,2001;Robbinsetal.,2004)andwhatlearningtechniquesworkwell(e.g.,Dunlosky,Rawson,Marsh,Nathan,&Willingham,2013).Studytechniquesinparticularareimportantbecausetheypredictacademicperformanceoverandabovestandardizedtestsandpreviousgrades(Gurung,2015).Thereareahostofkeypsychologicalvariablesthatareimportantaswell.Forexample,itisclearthatachievementmotivation,goals,socialinvolvement,andperceivedsocialsupportareallpositivelyassociatedwithstudents’academicperformance(Hattie,2015;Robbinsetal.,2004).Inparticular,factorssuchaseffort,ability,habits,andself-efficacyarestronglyrelatedtoacademicperformance(Credé&Kuncel,2008;Komarraju&Nadler,2013).Notsurprisingly,currentcollegeGPAandothercognitiveindicatorssuchasACTscoresandhighschoolGPAalsopredictlearninginauniversitysetting(Komarraju,Ramsey,&Rinella,2013).MeasuringACTscoresandGPAiseasy,oratleastalreadydoneforyou.Itisgettingattheothervariablesthatisthechallenge.Thegoodnewsisthattherearescalestomeasureeachofthedifferentvariablesreviewedabove.Abestpracticethenistobesureyouidentifykeypsychologicalvariablesthatarerelevanttoyourstudyandthenusetheassociatedscale.SomescalesrepresentingcommoncontrolvariablesinSoTLresearcharelistedbelow,withsomeofthemincludingsampleitemsfromthescale:

- AcademicSelfefficacy(Gredler&Schwartz,1997;Zajacova,Lynch,&Ependshade,2005)o Withtheprompt“pleaseanswerhowstressfulthesetasksareforyou,”

participantsrespondtoitemssuchas“keepingupwiththerequiredreadings,”“doingwellonexams,”“participatinginclassdiscussions,”and“understandingcollegeregulations.”Participantsalsorespondthesameitemswiththeprompt“howconfidentareyouthatyoucansuccessfullycompletethesetasks.”

- Selfregulation(Brown,Miller,&Lawendowski,1999)- Criticalthinkingandmotivation(Valenzuela,Nieto,&Saiz,2011)- Academiclocusofcontrol(Curtis&Trice,2013)

o ParticipantscompleteaTrueorFalseresponsetoitemssuchas“Ihavelargelydeterminedmyowncareergoals,”“TherearesomesubjectsinwhichIcouldneverdowellin,”“Studyingeverydayisimportant,”and“Iameasilydistracted.”

- Metacognition(Schraw,&Dennison,1994;Tuncer&Kaysi,2013;Wells&Cartright-Hatton,2004)

o Participantsrespondtoitemssuchas“Ihaveapoormemory,”“Ineedtoworryinordertodoworkwell,”“Iamconstantlyawareofmythinking,”and“itisbadtothinkcertainthoughts.”

- Motivatedstrategiesforlearning(Pintrich,Smith,Garcia,&McKeachie,1993)

13

o Participantsrespondtoitemssuchas“IpreferclassworkthatischallengingsoIcanlearnnewthings,”“IthinkIwillbeabletolearnwhatIlearninthisclassinotherclasses,”“Iexpecttodoverywellinthisclass,”and“Iworryagreatdealabouttests.”

- Depthofprocessing(Enwistle,2009)- LifelongLearningScale(Wielkiewicz,&Meuwissen,2014)- Procrastination(Tuckman,1991)

o Participantsrespondtoitemssuchas“IpostponestartinginonthingsIdon’tliketodo,”“Idelaymakingtoughdecisions,”“Igetrighttowork,evenonlife’sunpleasantchores,”and“whensomethingisnotworththetrouble,Istop.”

- StudyBehaviors/Process(Fox,McManus,&Winder,2001;Gurung,Weidert,&Jeske,2012)

- TextbookAssessmentandUtilityScale(Gurung&Martin,2011)

WhyShouldYouUsePublishedScales?Theeasyansweristhatitsavesyoualotofwork.Measurementisnotsomethingdonecasuallyorquickly.Developingvalidandreliablemeasuresisacomplexandinvolvingprocess(Noar,2003;Sosu,2013),sothesimplereasontousepublishedscalesisthatthehardworkhasbeendoneforyou.Robustscaledevelopmentinvolvesmultiplestudiesanditerations.Apublishedscalehasbeenthroughthepeerreviewprocessandtheassociatedchecksandbalances.Furthermore,otherresearcherswillhavealsousedthatpublishedscaleprovidingyouwithadditionalinformationabouttheconstruct.Correspondingly,youhavetheuseofascalethatshouldsatisfytwoimportantcriteriaforagoodscale:validityandreliability.Validityandreliabilityareessentialconceptsinmeasurement.Howwellhaveyoumeasuredyouroutcomesandpredictors?Howlikelyareyourmeasurestoprovidethesameresultswhenusedagain?Validityingeneralreferstotheextenttowhichthescalemeasureswhatitissupposedtomeasure(Anastasi,1988).Therearemanydifferentformsofvalidity(e.g.,external,statistical,internal)butwhenusingscaleswecaremostaboutconstructvalidity.Constructvalidityreferstotheideathatascaleismeasuringwhatwethinkitis(Morling,2015).Evenwhenyouusepublishedscales,itisprudenttobeawareandcomfortablewiththemainformsofconstructvaliditysoyoucanassessthequalityofthescale.Whereassomeformsofconstructvalidityaresubjectiveinnature,themajorityofthemareobjectiveinnatureandeasilyassessedbystatisticalrubrics.Subjectiveformsofconstructvalidityincludefacevalidityandcontentvalidity.Ascalewithgoodfacevaliditylookslikeitismeasuringwhatitissupposedto(youcanseehowsubjectivethisis).Aretheitemsplausiblewaystogetattheunderlyingconcept?Contentvaliditygetsatwhetherameasureencompassesallpartsoftheunderlyingconcept.Arethedifferentitemsgettingatallthedifferentpartsofconcept?Tohaveadequatecontentvalidity,ascaleshouldhaveitemsatallthedifferentpartsofaconcept.Often,scaleshavedifferentsub-componentsorsubscalesinordertofullycaptureconcepts.Objectiveformsofconstructvalidityincludecriterion(concurrentandpredictive),divergent,andconvergentvalidity.Criterionvalidityassessesifthescaleisrelatedtotheoutcomeitis

14

measuring.Ifyouuseagoodmeasureofprocrastination(Tuckman,1991)youwanttoseeitmeasuringabehavioralmeasureoftheoutcome(i.e.,thecriterion)suchastimetoturninanassignment.Therelationshipofthescaletoanoutcomecanbemeasuredatonetestingtime(concurrentvalidity)ortoanoutcomeatalatertime(predictivevalidity).Inconvergentvalidity,thescalecorrelateswithsimilarscalesormeasures.Forexample,themeasureofprocrastinationshouldcorrelatewithmeasuresofconscientiousness(inanegativeway).Indivergentvalidity,thescaleshouldnotcorrelatewithdissimilarscalesormeasures.Procrastinationneednotshowahighrelationshipwithextraversion.Whenyouuseapublishedscale,alltheseformsofvalidityaresuitablyestablished.Goodscalesarealsoreliable.Reliabilityreferstoconsistencyofmeasurement(Anastasi,1988).Inthecontextofscaledevelopmentanduse,twomajorformsofreliabilityareimportant.Test-retestreliabilityassessesifyouwillgetconsistentscoreseverytimeyouusethemeasure.IfImeasureself-efficacytoday,willthatsameself-efficacyscaleprovideasimilarresultinaweekoramonth(assumingnointerventiontochangeselfefficacy)?Internalreliabilityassessestheextenttowhichyourparticipantprovidesaconsistentpatternofanswers.Ifascalehas10items,doparticipantsanswerinaconsistentpatternacrossallitemseveniftheitemsarewordeddifferently?Goodscaleshavehightest-retestandinternalreliabilityandthesetwoformsarewhatyouneedtolookforwhenselectingmeasures.Bothformsofreliabilityaremeasuredusingformsofthecorrelationcoefficient(r).Correlationscloserto1suggesthighreliability.InternalreliabilityiscalculatedusingacorrelationbasedstatisticcalledCronbach’salpha(easilygeneratedbystatisticalpackagessuchasSPSS).Therelatedbestpracticethenistoselectmeasureswheretest-testreliabilityandinternalreliabilityarebothhigh,over.75and.65,respectively.HowDoYouPickScales?BeforeyoutakethetroubletowriteyourownitemsitiswellworthyourtimetousePsycINFOorotherdatabasesandseeifascaleexistsforwhatyouwanttomeasure.Thise-bookcontainsthemajorscalesyouneedandmanymoreexist.Somekeypointstokeepinmindwhenyouselectscales:

- Rememberthatsingleitemscanbeusedtomeasureconceptsbutmultipleitemscalesarebetter(DeVellis,1991).Youcannotcalculateinternalreliabilitywithasingleitem.

- Lookforscaleswithhighinternalreliability(Cronbach’salpha>.65),thathaveshowncriterion,convergent,anddivergentvaliditywhenpossible,andpreferablyhavehighfaceandcontentvalidity.YouwillfindthisinformationinaMethodsection.

- Besuretouseallitemsfromapublishedscale.Donotpickandchosetheitemsyouthinkbestfityourstudyasscalevalidityandreliabilityisbasedonyouusingalltheitems.Andifascaledoeshavesubscales,makesureyoudonotpresentallsubscaleitemsinonegroupeventhoughtheymayappeartogetherintheoriginalpublication.Youshouldhoweverfeelfreetomixuptheorderoftheitems.

- Somescales,however,suchasformeasurementoftheBigFivepersonalitytraits,haveshort-formversionsthatprovidegoodalternativestofullversions.

15

- Beawareofwhetherthescaleisunidimensional(givingyouonetotalscore)ormultidimensional(givingyoudifferentsubscalescores).Youcannotuseatotalscoreiftherearesubscales.

- Whenscoringascale,besureyouhavereversecodeditemsasneeded.- Becognizantofhowlongthescaleis,asmanylongscalesinasinglestudycancause

participantfatigueandthreatentheinternalvalidityofyourstudy(i.e.,aretherepossiblealternativeexplanationsfortheresults/changesinoutcomevariables?).

- Ordereffectsoccurwhenparticipants’responsestoscaleslaterinthestudymaynotbeasreliableasresponsesearlyinthestudy.

- Althoughusingscalesexactlyaspublished(sameinstructions,sameresponsescales)isoptimal,youmaysometimeshaveaneedtomodifyscalesslightlytosuityourpurposes.Notethatthepublishedvalidityandreliabilitiesmaynolongerhold.

BestPracticesinWritingYourOwnItemsIfthereisnopublishedscaleforyourpurposesyouthenhavetowriteyourownitemstogettheinformationyouneed.Althoughafullexpositionofscaleoritemconstructionisbeyondthescopeofthischapter(seeBerk,2006foragreattreatiseonthesame),therearesomeeasykeypointstokeepinmind.Itemscanbeofthreemainforms.Youcanaskopen-endedquestions(Describeyourlearningexperienceinthisclass?),forced-choiceitems(Whichofthefollowingassignmentsdidyoubestlearnfrom?),orLikertscaleitems(Howmuchdoyouagreewiththefollowingonascalerangingfromstronglyagree,agree,neitheragreenordisagree,disagree,stronglydisagree?).Sometimesparticipantscanbeaskedtorespondtoaquestionusinganumericscaleanchoredbyadjectives(Ratetheeasinessofthiscourseonascalerangingfrom1=Easyto5=Hard).Thislasttypeiscalledasemanticdifferenceformat.Cullingtogetherbestpracticesfromavarietyofsources(Berk,2006;DeVellis,1991;Morling,2015;Noar,2003)thefollowingaresuggestionsofstrong,concreteguidelinestokeepinmind.Youritemsshouldbe:

- Clearandconcise:Eachstatementshouldconveyaspecificideathatiseasilyinterpretablebytherespondent.Trytolimityourselftoabout10wordsorless.

- Devoidofslangorjargon,ordoublenegatives:Avoidlanguage,words,ortermsthatarespecifictoyourfieldorexpertiseandthatmaynotbemeaningful(orknown)totheaveragereader.

- Unambiguous:Yourmeaningshouldbeclear.Itisagoodideatohavefriendsreadyouritemsandtellyouwhattheythinkitmeans.Thisenablesyoutoconfirmyourintent.

- Bothpositiveandnegative(toavoidresponsessets):Writesomequestionssothattheaccurateanswerusesthelowerendofthescalewhileotherquestionsrequireanswersattheotherendofthescale.Forexampleboththefollowingitemsmeasurehighself-esteem,butthesecondisnegativelyworded:“Ifeelgoodaboutmyself”“AttimesIfeellikeafailure.”

- Genderandculturallysensitive:Itemsshouldbeapplicabletoallrespondents.Tryandwritegenericstatementsasitpertainstosex,ethnicity,andcultureingeneral.

16

- Ataneasyreadinglevel:Ifyoucanusesimplewords,doit.Thisisnotaplacetodisplayyourimpressivevocabulary.Asaruleofthumb,useahighschoolreadinglevel.

- Notleadingornudgingtheparticipanttoacertainanswer:Befairandbalanced.Someonereadingtheitemshouldnotbeabletoguesswhattheresearcher’shypothesismaybe.

- Single-barreled:Itemshouldonlyaskonequestion;eachstatementshouldhaveonlyonecompletebehavior,thought,orconcept.

- Haveresponsescalesthat:o Donotexceed7pointsonthescale.o Haveeachpointlabeled(e.g.,1=stronglyagree;2=agree).o Areevennumbered(toavoidfencesitting).o Donotusenumbersonthescalebutonlythewordlabelsorletterabbreviation

(e.g.,“SA”for“StronglyAgree”).o Avoidthe“NotApplicable”option.

ConclusionPedagogicalresearchersoftodaycanbegratefulforawiderangeofjournalsthatshowcaseusefulscales.TogetherwithScholarshipofTeachingandLearninginPsychology,TeachingofPsychology,andPsychologyofLearningandTeaching,anumberofjournalsacrossdisciplines,suchasTheInternationalJournaloftheScholarshipofTeachingandLearning,andtheJournalofScholarshipofTeachingandLearning,providescalesforuse.Therearesomebasicbestpracticesforusingscalesasdescribedabove.BeingcomfortablewiththesepracticeswillensurerobustSoTL.Whereasyoucanmakegreatcontributionstotheliteratureandthestudyofteachingbydevelopingscalestofillinthegapsthatexist,thatisawholeotherballofwax.Thereareasignificantnumberofwell-validatedscalesinthepublishedliterature.Itagoodideaforyoutoseekoutascalemeasuringthepreciseconceptualvariableyouareinterestedin,andonlywriteitemsiftheyareneeded.

17

ReferencesReferencesmarkedwithanasteriskindicateascale.Anastasi,A.(1988).Psychologicaltesting(6thed.).NewYork:Macmillan.Bartsch,R.A.(2013).DesigningSoTLstudies-PartI:Validity.InR.A.R.Gurung&J.Wilson(Eds.)

NewDirectionsForTeaching&Learning(pp.17-33).doi:10.1002/tl.20073Berk,R.A.(2006).Thirteenstrategiestomeasurecollegeteaching:Aconsumer’sguidetorating

scaleconstruction,assessment,anddecisionmakingforfaculty,administrators,andclinicians.Sterling,VA:Stylus.

*Brown,J.M.,Miller,W.R.,&Lawendowski,L.A.(1999).Theself-regulationquestionnaire.InL.VandeCreek,&T.L.Jackson(Eds.),Innovationsinclinicalpractice:Asourcebook(pp.281–292).Sarasota,FL:ProfessionalResourcePress/ProfessionalResourceExchange.

Credé,M.,&Kuncel,N.R.(2008).Studyhabits,skills,andattitudes:Thethirdpillarsupportingcollegiateacademicperformance.PerspectivesonPsychologicalScience,3,425-453.doi:10.1111/j.1745-6924.2008.00089.

*Curtis,N.A.,&Trice,A.D.(2013).Arevisionoftheacademiclocusofcontrolscaleforcollegestudents.Perceptual&MotorSkills,116,817-829.doi:10.1037/t32978-000

DeVellis,R.F.(1991).Scaledevelopment:Theoryandapplications.ThousandOaks,CA:Sage.DoyleT.,&Zakrajsek,T.(2013).Thenewscienceoflearning:Howtolearninharmonywithyour

brain.Sterling,VA:Stylus.Dunlosky,J.,Rawson,K.A.,Marsh,E.J.,Nathan,M.J.,&Willingham,D.T.(2013).Improving

students’learningwitheffectivelearningtechniques:Promisingdirectionsfromcognitiveandeducationalpsychology.PsychologicalScienceinthePublicInterest,14,4-58.doi:10.1177/1529100612453266

*Entwistle,N.J.(2009).Teachingforunderstandingatuniversity:Deepapproachesanddistinctivewaysofthinking.Basingstoke,UnitedKingdom:PalgraveMacmillan.

*Fox,R.A.,McManus,I.C.,&Winder,B.C.(2001).Theshortenedstudyprocessquestionnaire:aninvestigationofitsstructureandlongitudinalstabilityusingconfirmatoryfactoranalysis.BritishJournalofEducationalPsychology,71,511-530.doi:10.1348/000709901158659

*Gredler,M.E.&Schwartz,L.S.(1997).Factorialstructureoftheself-efficacyforself-regulatedlearningscale.PsychologicalReports,81,5-77.doi:10.2466/PR0.81.5.51-57

Gurung,R.A.R.,&Landrum,R.E.(2012).AssessmentandtheScholarshipofTeachingandLearning.InD.Dunn,S.C.Baker,C.M.Mehrotra,R.E.Landrum,&M.McCarthy(Eds.)AssessingTeachingandLearninginPsychology:CurrentandFuturePerspectives.

*Gurung,R.A.R.,&Martin,R.(2011).Predictingtextbookreading:Thetextbookassessmentandusagescale.TeachingofPsychology,38,22-28.doi:10.1177/0098628310390913

Gurung,R.A.R.,&Schwartz,B.(2009).Optimizingteachingandlearning:PedagogicalResearchinPractice.WileyBlackwellPublishing.London.

*Gurung,R.A.R.,Weidert,J.,&Jeske,A.S.(2010).Acloserlookathowstudentsstudy(andifitmatters).JournaloftheScholarshipofTeachingandLearning,10,28-33.

Hattie,J.(2015).TheapplicabilityofVisibleLearningtohighereducation.ScholarshipofTeachingandLearninginPsychology,1,79-91.doi:10.1037/stl0000021

18

Komarraju,M.,&Nadler,D.(2013).Self-efficacyandacademicachievement:Whydoimplicitbeliefs,goals,andeffortregulationmatter?LearningandIndividualDifferences,25,67-72.doi:10.1016/j.lindif.2013.01.005

Komarraju,M.,Ramsey,A.,&Rinella,V.(2013).Cognitiveandnon-cognitivepredictorsofcollegereadinessandperformance:Roleofacademicdiscipline.LearningandIndividualDifferences,24,103-109.doi:10.1016/j.lindif.2012.12.

Lent,R.W.,Brown,S.D.,andLarkin,K.C.(1986).Self-efficacyinthepredictionofacademicperformanceandperceivedcareeroptions.JournalofCounselingPsychology33,265–269.doi:10.1037/0022-0167.33.3.265

Morling,B.(2015).Researchmethodologyinpsychology:Evaluatingtheworldofinformation(2e).NewYork:Norton.

NationalResearchCouncil.(2000).Howpeoplelearn:Brain,mind,experienceandschool.CommitteeontheFoundationsofAssessment.Pelligrino,J.,Chudowsky,N.,&Glaser,R.(Eds.).Washington,DC:NationalAcademyPress.

Noar,S.M.(2003).Theroleofstructuralequationmodelinginscaledevelopment.StructuralEquationModeling,10,622-647.doi:10.1207/S15328007SEM1004

*Pintrich,P.R.,Smith,D.A.,Garcia,T.,&McKeachie,G.J.(1993).Reliabilityandpredictivevalidityofthemotivatedstrategiesforlearningquestionnaire.EducationalandPsychologicalMeasurement,53,801-813.doi:10.1177/0013164493053003024

Robbins,S.,Lauver,K.,Le,H.,Davis,D.,Langley,R.,&Carlstrom,A.(2004).Dopsychologicalandstudyskillfactorspredictcollegeoutcome?Ameta-analysis.PsychologicalBulletin,130,261–288.

*Schraw,G.,&Dennison,R.S.(1994).Assessingmetacognitiveawareness.ContemporaryEducationalPsychology,19,460–475.doi:10.1006/ceps.1994.1033

Schwartz,E.,&Gurung,R.A.R.,(2012).Evidence-basedteachinginhighereducation.Washington,DC:AmericanPsychologicalAssociation.

*Solberg,V.S.,O’Brien,K.,Villareal,P.,Kennel,R.,andDavis,B.(1993).Self-efficacyandHispaniccollegestudents:Validationofthecollegeself-efficacyinstrument.HispanicJournalofBehavioralSciences15,80–95.doi:10.1177/07399863930151004

*Sosu,E.M.(2013).Thedevelopmentandpsychometricvalidationofacriticalthinkingdispositionscale.ThinkingSkillsandCreativity,9,107-119.doi:10.1016/j.tsc.2012.09.002

*Tuckman,B.W.(1991).Thedevelopmentandconcurrentvalidityoftheprocrastinationscale.EducationalandPsychologicalMeasurements,51,473-480.doi:10.1177/0013164491512022

*Tuncer,M.&Kaysi,F.(2013).Thedevelopmentofthemetacognitivethinkingskillsscale.InternationalJournalofLearningandDevelopment,3,70-76.doi:10.5296/ijld.v3i2.3449

*Valenzuela,J.,Nieto,A.M.,&Saiz,C.(2011).Criticalthinkingmotivationalscale:acontributiontothestudyofrelationshipbetweencriticalthinkingandmotivation.ElectronicJournalofResearchinEducationalPsychology,9,823-848.

*Wells,A.&Cartwright-Hatton,S.(2004).Ashortformofthemetacognitionsquestionnaire:propertiesoftheMCQ-30.BehaviourResearchandTherapy,42,385-396.doi:10.1016/S0005-7967(03)00147-5

19

*Wielkiewicz,R.M.&Meuwissen,A.S.(2014).Alifelonglearningscaleforresearchandevaluationofteachingandcurriculareffectiveness.TeachingofPsychology,41,220-227.doi:10.1177/0098628314537971

*Zajacova,A.,Lynch,S.M.,&Epenshade,T.J.(2005).Self-efficacy,stress,andacademicsuccessincollege.ResearchinHigherEducation,46,677-706.doi:10.1007/s11162-004-4139-z

20

Chapter3:SelectingtheRightScale:AnEditor’sPerspectiveAndrewN.Christopher

AlbionCollege

BeforeIbegintooffermyperspectivesonselectingtheappropriatescaleforaresearchproject,itisimportantthatIsharesomeofmyacademicbackgroundbecauseitclearlycolorsmyviewsinthischapter.Ingraduateschool,Iwasinaprograminwhichthegraduatestudentsinsocialpsychologyinteractedregularlywithgraduatestudentsincounselingpsychology,cognitivepsychology,andneuroscience.Becauseitwascommonplaceforsuchinteractionstooccur,Ifiguredthatwasthewayresearchersindifferentareasworked;thatis,theyfrequentlyinteractedwitheachother.Indeed,16yearsafterreceivingmyPh.D.,mybestfriendsfromgraduateschoolwithwhomIstillworkarefromareasoftrainingdifferentthanmine.Ihavelearnedduringthepast16yearsthatsuch“cross-area”communicationisinfactnotasnormalasIperceivedittobe.Sincecompletinggraduateschool,Ihavetaughtatasmallliberalartscollegeinwhichfacultyinalldisciplinesregularlyinteractonscholarlyworkrelatedtobothteachingandresearch.SomeofthebestcoursematerialIhavedevelopedformyindustrial/organizational(I/O)psychologyclassoriginatedfromnumerousdiscussionswithacolleaguefrommycollege’shistorydepartment.Someofmyownresearchfocusesonindividualdifferences,soitisrootedstronglyinacombinationofI/Oandpersonality,bothofwhichrelyheavilyontheuseofscales,apracticethatisincreasinglybeingscrutinizedbypeopleinbothareas.Withthisbackgroundinmind,pleaseallowmetoprovidemyinsightsintoselectingtherightscaleforaresearchproject.A(Needlessly)GreatDivideItisnotuncommonfortheretobegreatdividesbetweenresearchersinseeminglyrelatedareasofresearch.Forexample,itmightmakeintuitivesenseforresearchersinpersonnelselectionandperformanceevaluationtocollaborate,particularlyonappliedtypesofprojects.Havingworkedinbothareas,Ihavebeensurprisedhowlittlecollaborationorevensimplycommunicationthereisbetweenresearchersoneachsideofthisprocessinbothacademicandappliedsettings.1

Withintherealmofeducational-typeresearch,DanielandChew(2013)drewadistinctionbetweenthelearningsciencesandSoTL,theformerofwhichhas“Thegoal….tobetterunderstandthecognitiveandsocialprocessesthatresultinthemosteffectivelearning,andtousethisknowledgetoredesignclassroomsandotherlearningenvironmentssothatpeoplelearnmoredeeplyandmoreeffectively”(Sawyer,2006,p.xi).Itsemphasisisontheapplicationofresearchinbasicareasofpsychology,suchaspersonalityandcognitive,toeducationalsettings.Forexample,Christopher,Furnham,Batey,Martin,Koenig,andDoty(2010)examinedhowworkvaluesmaybecompensatorymechanismsforlowerlevelsofgeneralintelligence.Certainly,suchworkcouldbeapplicabletoalmostanyeducationalsetting.Incontrasttothelearningsciences,muchSoTLwork“…typicallyreportsthesuccessofparticularmethodsinaparticularcontextandwithinaparticularlevelofaspecificdiscipline”(Daniel&Chew,2013,p.365).Indeed,SoTLtendstobequitespecificinitsfocusandgeneralizabilityofitsresults.For

21

example,McCabe(2015)examinedtheextenttowhichthereisagenerationeffectforlearner-createdkeywordmnemonicsandreal-lifeexampleswhenlearningneurosciencematerialintheintroductorypsychologycourse.ThisresearchiscertainlyappropriateforteachersofIntroductoryPsychology.DanielandChewnicelydescribedwhatthey(andI)seeasadividebetweenSoTLandlearningsciences,withresearcherstendingtofallintoonecamportheother,butrarelyoverlapping,despitetheenormousdesirabilityandpotentialforsuchoverlap.Inmanyways,Iseethisdivideassimilartothenotionthatresearcherstendtogetclassified,rightlyorwrongly,aseither“basic”researchersor“applied”researchers(withtheformertypicallybeingperceivedas“better”researchers).Manyperceivethatthereisoftennomiddleground,whichIbelievehinderstheprogressofbothtypesofresearch.Muchlikethedividebetweenpersonnelselectionandpersonnelevaluationresearchers,orthedividebetweenbasicandappliedresearchers,thedividebetweenSoTLandlearningsciencesresearchersis,inmyopinion,unfortunate.Ibelievethatthise-bookisalargesteptowardbridgingthisdividebetweenthelearningsciencesandSoTLresearchers.Indeed,muchoftheinformationitcontains,particularlyinGurung’s(2015)andWilson-Doenges’(2015)chapters,isactuallyrootedinthelearningsciences.Forexample,inchapter2ofthise-book,Gurunglisted10differentscalesthatcouldbeusefulinSoTLresearch.Inreality,allbuttwoofthesescales(theLifelongLearningScaleandtheTextbookAssessmentandUtilityScale)arerootedinbasicindividualdifferencesresearch.Specifically,researchersinthelearningscienceshavelongtendedtousescalesthathaveundergoneextensivepsychometricwork.Forinstance,theNeedforCognitionScale(Cacioppo,Feinstein,Jarvis,&Petty,1996),whichreceivedagreatdealofpsychometricevaluation,hasbeenusedextensivelyinlearningsciencesresearch.Atthetimeofpublicationofthischapter,aquickPsycINFOsearchon“needforcognition”and“learningscience”revealed46hitsinthepastthreeyears,witharticlesappearinginabroadspectrumofjournals,suchasLearningandIndividualDifferences,ComputersinHumanBehavior,CreativityResearchJournal,andPsychologyofWomenQuarterly.Asimilarsearchwith“scholarshipofteachingandlearning”substitutedfor“learningscience”revealedbutonehit.Ofcourse,thisexaminationreliesonasampleofone.However,whenIreviewforjournalssuchastheoneslistedpreviously,itseemsasthoughusingpsychometricallyvettedmeasurementtoolsisanexpectedpractice.InChapter5,Wilson-Doenges(2015)describesthecurrentstateofpsychometricworkonSoTLscales.Asshenotes,suchworkisbecomingmoreprevalentandmorerigorousinSoTLinrecentyears.ItisessentialthatsuchworkcontinueandbedisplayedinprominentSoTLoutlets.SoTListoappliedresearchwhatthelearningsciencesaretobasicresearch.Thisshouldnotbeaproblem;however,extendingthe(whatIbelievetobegrosslyinaccurate)perceptionthatbasicresearchismorescientificandrigorousthatappliedresearch,SoTLworkmaybeperceivedasinferiortoworkinthelearningsciences.WhymightpsychometricissuesreceivelessattentioninSoTLworkthaninlearningsciencework?Icanofferonlytworeasons,andbotharepurelyspeculation.First,perhapsbecauseof

22

therelativelyspecificfocusofitsstudies,itisdifficulttoestablishthepsychometricpropertiesofameasurementtool.Forinstance,inMcCabe’s(2015)researchpresentedearlier,thecontextofherworkwaslimitedtoonecourse(IntroductoryPsychology).Therefore,forherresearch,psychometrically-establishedproceduresandmeasuresweredifficulttocomeby.ForChristopheretal.’s(2010)researchonworkvalues,itwasmucheasiertolocatepsychometrically-testedmeasures.Inadditiontotheissueofresearchspecificity,itcouldbethecasethattraininginthelearningsciencesfocusesmoreonpsychometricissuesthandoestraininginSoTL.SimilartotraininginpersonalityandI/O,bothofwhichhavebeenhistoricallyreliantonsurveymethodologies,learningscienceresearchersmayhavetakenrequiredpsychometriccourseworkintheirgraduatetraining.GraduatetraininginSoTLmayfocusmoreondevelopingclassroomteachingandmentoringskillsthanonmoretraditionalformsofscholarship.Inhisreviewofgraduatetrainingpractices,Buskist(2013)offeredanumberofchallengesandrecommendationsforthefutureofsuchwork.Interestingly,traininggraduatestudentstoperformSoTLwasnotmentioned.Indeed,SoTLseemslikealogicalextensionofgraduatetraininginteaching,whereaslearningsciencesimplicitlyassumescholarshipisapart(ifnottheentirety)ofgraduatetraining.Thus,whereasresearchersinthelearningscienceshavearelativelylongtraditionofemphasizing,atleastimplicitly,theimportanceofscalepsychometrics,researchersinSoTLareonlymorerecentlybeginningtomoreintentionallyundertakethetaskofputtingtheirmeasurestothepsychometrictest.Again,therearecertainlySoTLarticlesthatfocusonissuesofpsychometrics(seeChapter5forsomesuchexamples),butingeneral,SoTLcanbenefitgreatlyfromfollowingtheleadoflearningscienceresearchersandtheiremphasisonscalepsychometricproperties.Inparticular,totheextentthatthelearningsciencesisperceivedtobe“morescientific”thanSoTL,heremaybeanareatotrimthatperceptivedivide.MySuggestionsAsmightbeobviousfrommytone,Iamnothappyaboutthedividesbetweenresearcherswhostudysimilartopics.Here,IwilltryandsuggestswaysthatSoTLcanbridgeitsdividewiththelearningsciencesandthusprofitfromworkinnotonlythelearningsciences,butinpersonalityandI/Opsychologyaswell.Inhischapter,Gurung(2015)presentedanumberofexcellentargumentsforusingalready-publishedscales.Icannotagreestronglyenoughwiththesesuggestions.Mostpublishedscaleswillpossessconstructvalidity,orelsetheyalmostcertainlywouldnothavebeenpublished.AsGurungsaid,suchpsychometricworkisnotdonecasuallyorwhimsically.Toestablishameasurementtool’spsychometricpropertiesisapainstakingprocess.Anyscalethathasmadeittothepointofpublicationinarespectedpeer-reviewjournaldidnotgettherebyaccident.Itlikelyhassomethingtoofferandisarelatively“safe”bet,psychometricallyspeaking.InadditiontotheguidelinesforselectingapublishedscalethatGurungsuggested,allowmetoaddthreeother,somewhatinterrelated,suggestions.First,researchersshouldpaysomeattentiontheoutletinwhichascalewasoriginallypublished.NoteveryscaleneedstobepublishedinPsychologicalBulletintobea

23

psychometricallysoundtool.However,thequalityoftheoutletinwhichascaleappearsdoespotentiallysaysomethingaboutitspsychometricquality.2Second,inadditiontotheoriginalpublicationthatcontainsthescale,looktoseeifsubsequentworkhassomehowimprovedtheoriginalscale.Forexample,theTeacherBehaviorsChecklist(TBC;Buskist,Sikorski,Buckley,&Saville,2002)hasbeensubjectedtosubsequentpsychometrically-focusedresearch(e.g.,Keeley,Smith,&Buskist,2006).Theresult?Ascalethatisbetter,psychometrically-speaking,thanitotherwisewouldbe3.Finally,docitedreferencesearchesonthepublicationthatcontainedtheoriginalscaleandanysubsequentresearchthatamendedandimprovedtheoriginalscale.Ifsuchcitedreferencesearchesrevealfewornocitationstothesesources,itisanindication,albeitanimperfectone,thatperhapsthescaleisnotwell-acceptedinthescientificcommunity.WithmypersonalityandI/Oworldview,Iamnoticingmanyscalesthatseemliketheyaremeasuringthesameconstruct4.Forexample,arecentpublicationexaminedworkethicandGRITaspredictorsofjobsatisfaction,turnoverintention,andjobstress(Meriac,Woehr,&Banister,2015).Workethicwasdefinedas“asetofbeliefsandattitudesreflectingthefundamentalvalueofwork”(p.316).GRITisthe“perseveranceandpassionforlong-termgoals”andentails“workingstrenuouslytowardchallenges,maintainingeffortandinterest….despitefailure,adversity,andplateausinprogress(Duckworth,Peterson,Matthews,&Kelly,2007,pp.1087-1088).Atfirstread,workethicandGRITmaysoundlikecloselyrelatedconstructs,andinfact,theywerecorrelated(r=.44;Meriacetal.,2015).Dowereallyneedbothofthesemeasures?Indeed,Meriacandhiscolleaguessuggestedthatinfacttheydodifferentiallypredictjobsatisfaction,turnoverintentions,andstress.Furthermore,Meriacandhiscolleaguesfoundthatthereisincrementalvalidityinusingbothofthesemeasures.Indeed,atleastwithinthecontextofI/Opsychology,itdoesappearthatthereisvalueinhavingbothofthesemeasures.Researchersneedtoconsiderwhichoneisappropriatetoansweragivenresearchquestion,oriffactitisworthusingbothofthem.Ultimatelythereisnocorrectorincorrectchoice;rather,itisimperativethatsomejustificationforthechoiceofascaleorscalesisneeded,particularlywhenthereareseeminglyoverlappingpossiblescalestochoosefrom.Indeed,withsomanyscalesavailable,lookforthosethathavedemonstratedincrementalvalidityinpriorresearch.Inmyopinion(andmyopiniononly),incrementalvalidityseemstoreceiveshortshriftrelativetootherformsofvalidityinestablishingthepsychometricpropertiesofascale.Thecynicalpartofmedoeswonderhowmanynewscalesactuallypredictoutcomesaboveandbeyondexistingscales.Themore-eruditepartofmebelievesthatnewscalesareindeedaddingtowhat’salreadyavailable,andifsuchevidencehasnotbeenoffered,itisincumbentonthescientificcommunitytoofferevidencesuchisthecase.InChapter14,Richmond(2015)describestheissueofrelyingonself-reportdataandnotactualbehavior,somethinginherentwithanyresearchareathatreliesonscales.InpersonalityandI/O,thereisgreatconcernaboutcommonmethodbias.Commonmethodbiasoccurswhenthesamegeneralmethodology(e.g.,relianceexclusivelyonself-reportdata)isusedtoansweraresearchquestion.ThetrendinpersonalityandI/Oistoavoidcommonmethodbiastothe

24

extentpossible.Richmondmakesanumberofexcellentsuggestionsinhischapteronhowtogoaboutavoidingcommonmethodbias.Nomatterwhatscalearesearcherselects,itishelpfultoreviewersifauthorspresentanexplicitunderstandingoflimitationsofagivenmeasure.Forexample,Wilson-Doenges(2015)mentionstheTen-ItemPersonalityInventory(Gosling,Rentfrow,&Swann,2003)tomeasuretheBigFive.Thisinstrumenthasamassedawhopping1253citedreferencesonPsycINFOasofthiswriting.Therefore,thescientificcommunityhasenthusiasticallyembracedthisscale.Butwithonly2itemstomeasureeachfactor,itisdifficulttomeetthe“goldstandard”ofa.70orstrongerCronbach’salpha.However,ifthereareotherscalesandmeasurementsthatareneededinastudy,thismaywellbeanacceptabletradeoffforavoidingparticipantfatigue.Perhapsreviewersandeditorswon’tagreewiththissentiment,butfailingtoacknowledgethistradeoffissuretoworkagainsttheresearcherduringthepeer-reviewprocess.Wilson-Doenges(2015)providessomeexemplarsofexcellentpsychometricworkinSoTL.Here,Itrytocomplementhersuggestionswithfiveexamplesandbriefsummariesofpsychometricworkfrompersonality,I/O,andthelearningsciences.TheseareallsourcesthatIhaveusedinmyclassestodemonstratecertainpsychometricprinciples.Oneofthecommonalitiesofeachofthesefivemodelsisthateachoneprovidesanexcellentconceptualbackgroundtotheconstruct(s)beingmeasuredinthearticle.ThefirsttwoexemplarsareonesI’vementionedpreviouslyinthischapter.Duckworthetal.(2007)devisedascaletomeasureGRITandconductedsixstudiestoestablishitspsychometricproperties.Acrossthesesixstudies,theyestablishedthefactorstructureoftheirmeasure;itspredictivevaliditybycorrelatingitwithlifetimeschoolingamongpeopleofidenticalage;investigateditsincrementalvalidityovertheBigFivefactors;assesseditsconvergentanddivergentvaliditybycorrelatingitwithGPAandgeneralintelligence;andevaluatedformsofreliabilityincludingtest-retest,internal,anditem-totalcorrelations.Miller,Woehr,andHudspeth(2002)provideanotherexcellentexampleofscaledevelopment.Acrosssixstudiesusingundergraduatestudents,U.S.AirForcepersonnel,andemployeesfromprivately-ownedcompanies,theseresearchersassessedthepsychometricpropertiesoftheMultidimensionalWorkEthicProfile.Specifically,theyincludediscussionsofthemeasure’sfactorstructure,internalreliabilitiesofthefactors,convergentanddivergentvalidity,predictivevalidity,incrementalvalidity,andtest-retestreliability.Withintherealmofpersonalityandsocialpsychology,thereisGlickandFiske’s(1996)workonambivalent(hostileandbenevolent)sexism,whichlikethefirsttwoexemplars,containssixstudiesthatspotlightaplethoraofpsychometricconsiderationinscaledevelopment.Thefirstthreeexemplarscontainedsixstudies.Andindeed,thoroughscaledevelopmentdoesrequireagreatdealofwork,asGurung(2015)emphasized.However,Iknowoftwoparticularlygoodsourcesofpsychometricworkthatcontainfewerstudies.First,thereisMidgleyandcolleagues’(1998)workonscalestomeasuredifferentgoalorientations(i.e.,task-goal,ability-approach,andability-avoidantorientations).Thissourceisparticularlygoodbecauseitprovides

25

ahistoryoftheauthors’workdoneongoalorientationmeasures.Thishistoryisorganizedaroundmajorpsychometricpropertiesoftheirmeasures.Itisaparticularlygoodexemplartousewithstudentsinanundergraduatetestsandmeasurementsclass.Finally,Oleson,Poehlmann,Yost,Lynch,andArkin(2000)presentedtwostudiesthatassessedtwocomponentsofsubjectiveoverachievement:self-doubtandconcernwithperformance.Intheirresearch,Olesonandcolleaguesexaminedthefactorstructure,internalreliabilities,test-retestreliabilities,convergentvalidity,anddivergentvalidityofscalestomeasureself-doubtandconcernwithperformance.Insum,thechoiceofscalesinSoTLpresentsuswithanopportunitytolearnfromourcolleaguesinpersonality,I/O,andlearningsciences,allofwhichhavealonghistoryofscaledevelopment.Inaddition,theseareasalsohavescalesthatmaybeofuseinSoTL.Byintegratingwhattheseareashavetoofferintermsofpsychometricworkandresultingscales,wecanhelpbridgeintellectualdividesthathinderresearchprogressinalloftheseareas.

26

ReferencesReferencesmarkedwithanasteriskindicateascale.Buskist,W.(2013).Preparingthenewpsychologyprofessoriatetoteach:Past,present,and

future.TeachingofPsychology,40,333-339.http://dx.doi.org/10.1177/0098628313501047

*Buskist,W.,Sikorski,J.,Buckley,T.,&Saville,B.K.(2002).Elementsofmasterteaching.InS.F.Davis&W.Buskist(Eds.),Theteachingofpsychology:EssaysinhonorofWilbertJ.McKeachieandCharlesL.Brewer(pp.27-39).Mahwah,NJ:LawrenceErlbaumAssociates,Inc.

*Cacioppo,J.T.,Petty,R.E.,Feinstein,J.A.,&Jarvis,W.B.G.(1996).Dispositionaldifferencesincognitivemotivation:Thelifeandtimesofindividualsvaryinginneedforcognition.PsychologicalBulletin,119,197-253.http://dx.doi.org/10.1037/0033-2909.119.2.197

Christopher,A.N.,Furnham,A.,Batey,M.,Martin,G.N.,Koenig,C.S.,&Doty,K.(2010).Protestantethicendorsement,personality,andgeneralintelligence.LearningandIndividualDifferences,20,46-50.http://dx.doi.org/10.1016/j.lindif.2009.10.003

Daniel,D.B.,&Chew,S.(2013).Thetribalismofteachingandlearning.TeachingofPsychology,40,363-367.http://dx.doi.org/10.1177/0098628313501034

*Duckworth,A.L.,Peterson,C.,Matthews,M.D.,&Kelly,D.R.(2007).Grit:Perseveranceandpassionforlong-termgoals.JournalofPersonalityandSocialPsychology,92,1087-1101.http://dx.doi.org/10.1037/0022-3514.92.6.1087

*Gosling,S.D.,Rentfrow,P.J.,&Swann,W.B.,Jr.(2003).AverybriefmeasureoftheBig-Fivepersonalitydomains.JournalofResearchinPersonality,37,504-528.http://dx.doi.org/10.1002/tl.20071

*Glick,P.,&Fiske,S.T.(1996).TheAmbivalentSexismInventory:Differentiatinghostileandbenevolentsexism.JournalofPersonalityandSocialPsychology,70,491-512.http://dx.doi.org/10.1037/0022-3514.70.3.491

Gurung,R.A.R.(2015).BestpracticesinscaleuseinSoTL.InR.S.Jhangiani,J.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

*Keeley,J.,Smith,D.,&Buskist,W.(2006).TheTeachingBehaviorsChecklist:Factoranalysisofitsutilityforevaluatingteaching.TeachingofPsychology,33,84-91.http://dx.doi.org/10.1207/s15328023top3302_1

McCabe,J.A.(2015).LearningthebraininIntroductoryPsychology:Examiningthegenerationeffectformnemonicsandexamples.TeachingofPsychology,42,203-210.http://dx.doi.org/10.1177/0098628315587617

Meriac,J.P.,Slifka,J.S.,&LaBat,L.R.(2015).Workethicandgrit:Anexaminationofempiricalredundancy.PersonalityandIndividualDifferences,86,401-405.http://dx.doi.org/10.1016/j.paid.2015.07.009

*Midgley,C.,Kaplan,A.,Middleton,M.,Maehe,M.L.,Urdan,T.,Anderman,L.H.,Anderman,E.,&Roesner,R.(1998).Thedevelopmentandvalidationofscalesassessingstudents’achievementgoalorientations.ContemporaryEducationalPsychology,23,113-131.

*Miller,M.J.,Woehr,D.J.,&Hudspeth,N.(2002).Themeaningandmeasurementofworkethic:Constructionandinitialvalidationofamultidimensionalinventory.JournalofVocationalBehavior,60,451-489.http://dx.doi.org/10.1006/jvbe.2001.1838

27

*Oleson,K.C.,Poehlmann,K.M.,Yost,J.H.,Lynch,M.E.,&Arkin,R.M.(2000).Subjectiveoverachievement:Individualdifferencesinself-doubtandconcernwithperformance.JournalofPersonality,68,491-524.http://dx.doi.org/10.1111/1467-6494.00104

Richmond,A.S.(2015).Themissinglink(s)ofSoTLscales:Oneresearcher’sperspective.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Sawyer,R.K.(Ed.).(2006).TheCambridgehandbookofthelearningsciences,preface.NewYork,NY:CambridgeUniversityPress.

Wilson-Doenges,G.(2015).ThestateofscalevalidationinSoTLresearchinpsychology.InR.S.Jhangiani,J.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

28

Footnotes1Oncewhileworkingonapersonnelselectionproject,acolleagueaskedmeifthere

wasreallyanyvalueinpersonnelevaluationprocedures.Thus,whatIamcallinga“divide”mightbebetterdescribedas“contempt”insomecircumstances.

2Althoughafarfromperfectindicatorofjournalquality,onesuchwide-acceptedbenchmarkisajournal’simpactfactor.Youcantypicallyfindthisinformationforajournalonitspublisher’swebsiteandfromtheSocialSciencesCitationIndex.

3IthinkthattheTBC,althoughrootedmoreinSoTLresearch,couldeasilybeusedinlearningsciences.ThoughmostofthecitedreferencesIfoundonKeeleyetal.’s(2006)psychometricpaperwereindeedinSoTL-typeoutlets,thismeasurehascaughtoninoutletsthatmightbeassociatedmorewiththelearningsciences,suchasAssessment&EducationinHigherEducationandReviewofEducationalResearch.

4IkeepissuesofPersonalityandIndividualDifferencesandLearningandIndividualDifferencesonmynightstandandreadthembeforegoingtosleepeachnight.So,Imaybeexposedtoawiderrangeofscalesthanmostpeopleandhencemyimpressionof“scaleoverload”inpsychologicalresearch.

29

Chapter4:APrimeronScaleDevelopmentHeatherD.HusseyandTaraJ.Lehan

NorthcentralUniversity

IntroductionOneoftheprimaryfeaturesofcontemporaryacademic,professional,public,andpersonallifeisarelianceoninformationandargumentsinvolvingnumbers.Thischapterincludesanintroductiontothestepsinvolvedindevelopingandvalidatinginstrumentstocollectnumericaldataforuseinthestudyofteachingandlearning(SoTL)research.Independentscholarsmustbeabletoevaluatequantitativeevidencethoughtfullyandcriticallyaswellasemployquantitativeskillstomakecontributionsintheworkplaceandtosociety.Tobecomecompetentcreatorsandconsumersofquantitativeresearch,readersmustunderstandhownumericalinformationisgenerated,summarized,evaluated,andrepresented.Withthesegoalsinmind,classicalandmoderntesttheories,reliability,validity,factoranalysis,andothertopicsarecoveredinsomedetailinthischapter.However,itiscrucialtonotethatthischapterisonlyastartingpointforthosewhoareinterestedinbetterunderstandingtheuseandcreationofinstrumentsinSoTLresearch.Further,manyofthediscussionslackthedepthnecessarytoprovidereaderswiththeknowledgeneededtobegintodeveloptheirowninstruments.Onlythemostrelevanttopicsarecoveredatasuperficiallevelduetospacelimitations.Althoughtheprocessofscalecreationfrequentlyisiterative,thischapterincludesadescriptionofthegeneralstepsinvolvedindevelopinganinstrument,includingsearchingthescholarlyliteraturetolocateandevaluatepreexistinginstrumentsandthendeterminingwhethertheyareadequateorneedmodification(seeFigure1).Ifresearchersmustmodifyapreexistinginstrumentorcreateanentirelynewone,theyshouldreturntothescholarlyliteraturetoensurethattheyacquirealloftherelevantknowledgeontheconstruct(s)ofinterest.Next,inconjunctionwithmeasurementtheoriesand/ormodels,researchersusethisinformationtodevelopapoolofpotentialitems.Followingthecreationofthispool,researchersoftenhaveapanelofexpertsreviewtheitemsforclarityandconductaninitialscreeningofthedegreetowhichtheyalignwithconstruct(s)ofinterest.Someresearchersalsoconductapilotorfieldtestwiththeirinstrumentwithasmallsample.Theythenimplementallsuggestedchangesandadministertheinstrumenttoalargersample.Followingdatacollection,researchersanalyzetheresponses.Thisprocessofteninvolvesfactoranalysisandassessmentsofreliabilityandvalidity.Basedonthefindings,theymakenecessarymodificationsandrepeattheprocessand/orconductstudiestoreplicateand/orgeneralizetheirfindings.

30

Figure1.Scalecreationprocessdepictedasaniterativeprocess.Note:thisprocessshouldbeinterpretedloosely,asonemightreturntoanystepatanytime,dependingontheoutcome.Forexample,commentsonanexpertreviewmightresultinreturningbacktothescholarlyliteratureandsoon.)UseofExistingScalesResearcherscommonlyemployquestionnairestocollectdata,especiallyinappliedresearch(Boynton&Greenhalgh,2004).Wheneverpossible,researchersshouldlocateandemploypreviouslydevelopedscaleswithevidenceofvalidityandreliability.Onebenefitofusingexistingscalesisthattherearegenerallypublishedstudieswithdatathatresearcherscancomparetothedataobtainedinsubsequentstudies.Whenresearchersconductmultiplestudiesusingthesamescale,theycanbuildabodyofliteratureandtheory.Further,theyneedtodevotefewerresources,especiallytimeandenergy,whenusingexistingscales.Manyofthechaptersinthise-bookaredevotedtosharingsuchscalesthatresearcherscanuseintheirSoTLresearch.Forexample,theTeacherBehaviorsChecklist(TBC;Buskist,Sikorski,Buckley,&Saville,2002)isapopularmeasureusedinSoTLresearch(seeKirk,Busler,Keeley,&Buskist,2015).ItisdiscussedfurtherinChapter13ofthise-book.However,researchersmightfindthatpreexistingscalesdonotmeasurecertainelementsofaconstruct.Forexample,Wilson,Ryan,andPugh(2010)notedthat,althoughtherewereexistingmeasuresofrapport,nonespecificallyaddressedprofessor-studentrapport.Asaresult,theysoughttodevelopsuchameasure.AlsodiscussedfurtherinChapter3ofthise-book,researcherssometimesusescalesbeforethereissufficientresearchtosupporttheirvalidityandreliability(Christopher,2015).Assuch,theymightdrawseeminglysignificantconclusionsfromtheapplicationofnewscales,onlytohavethefindingsofsubsequentstudiescontradicttheirfindings(Cook,Hepworth,Wall,&Warr,1981).Insuchcases,itmightbeappropriatefortheresearcherstomodifyexisting

Review Scholarly Literature

Develop Pool of Items

Expert ReviewPilot/Field Test

Administer Survey

31

scalesordevelopnewones.Ineithercase,itisimportantforthemtoreturntothescholarlyliteraturetocollectasmuchrelevantinformationaspossibleontheconstructofinteresttoinformscaledevelopment.MeasurementTheoriesTherearetwomajorunderlyingtheoriesuponwhichquantitativemeasurementisfounded:classicaltesttheory(CTT)anditemresponsetheory(IRT).Representingtwodistinctframeworks,theyprovidethefoundationfromwhichresearchersoftenbuildinstruments.Usingthesetheories,researcherscanobtaininformationaboutthequalityoftheitemsthattheydevelop.Giventhatthereareentirechaptersandbooksdevotedtothesetopics,thepurposeofthissectionistoprovideageneraloverview(seeHambleton&Jones,1993,anddeChamplain,2010,foranoverviewandcomparisonofthesetheories).

CTTCTTdescribesasetofpsychometricproceduresusedtotestscalereliability,itemdifficulty,anditemdiscrimination(i.e.,theextenttowhichanitemhelpstheresearchtodifferentiatebetweenrespondentswithhigherandlowerabilities).Manywidelyavailablestatisticalpackagesproducethesestatistics.Thereareseveralclassicaltesttheories,allofwhicharefoundedupontheassumptionthatarawscoreiscomprisedofatruescoreandarandomerror(Kline,2005).Inthecontextofteachingandlearning,atruescoremightreflectastudent’strueability.Inthiscase,theunsystematicinfluenceofanyfactorsonthemeasurementofthatabilitywouldbereferredtoasrandomerror.Inaddition,theoverridingconcernofCTTistomanagetherandomerror.Thelessrandomerrorthereis,themoretherawscorereflectsthetruescore.Duetoitsrelativelyweakassumptions,CTTcanbeusedinawidevarietyoftestingsituations(Hambleton&Jones,1993).However,themaincriticismisthatstatisticsthataregeneratedintermsoftheobservedscore,itemdifficulty,anditemdiscriminationaredependentuponthesample(Fan,1998).

IRTAlsoknownasmoderntesttheory,IRTallowsresearcherstoovercomemanyofthelimitationsofCTT.Itcanbeusedtomodelrespondentabilityusingitem-levelperformance,ratherthanaggregatetest-levelperformance.Manytimes,thevariablethatresearcherswanttostudy,suchasintelligence,cannotbemeasureddirectly;therefore,theycreateanumberofitemsthattheybelievecaptureit(Noar,2009).Suchvariablesarereferredtoaslatentvariables.IRTinvolvesestablishingamodelthatspecifiestheprobabilityofobservingeachresponsetoanitemasafunctionofthelatenttraitbeingmeasured,whichisoftenknowledge,askill,oranability(DeVellis,2012).Theitemresponsefunctionillustratestherelationshipbetweenalatentvariableandtheprobabilityofendorsinganitem.Itcanthenbeconvertedintoanitemcharacteristiccurve,whichshowsrespondentabilityasafunctionoftheprobabilityofendorsingtheitem.

32

ItemDevelopment

WordingHigh-qualitytestitemsarecriticaltothedevelopmentofameaningfulinstrument.Preciselanguagemustbeusedforameasuretoadequatelycapturethespecificconstructofinterest,yetcontainnoextraneouscontent.Researchersshouldavoidtheuseoflanguagethatmightbeconfusing,suchasitemsthatincludedoublenegatives,aswellasdoubled-barreledquestionsthattapintomorethanoneconstructbutallowforonlyoneresponse.Inaddition,itemsshouldbewrittenatalevelthatisappropriateforthetargetresponses.Variousresourcesexistthatprovideresearcherswithinformationaboutthegradelevelatwhichanitemiswritten,includingthereadabilitycalculatorathttps://readability-score.com/.Considerthefollowingquestions:

“Towhatextentdoestheteacherutilizeexamplestoexplicateconvolutedconcepts?”“Howmuchdoestheteacheruseexamplestodescribedifficultconcepts?”

Whereasbothquestionsaretappingintothesampleconstruct,thefirstonerequiresthatrespondentshaveahigherreadinglevelthanthesecondone.Iftheintendedrespondentsaresophomoresinhighschool,itsreducedreadabilitycouldpotentiallyimpacttheresponses.Foradditionalconsiderationstoconsiderwhenwordingquestions(e.g.,typesofscalestouse)seeSchaeffer&Presser(2003).

StrategiesAninductiveoradeductiveapproachcanbeusedtogenerateitems.Withaninductiveapproach,thereisnotheoreticalorconceptualframework,whereasthereiswithadeductiveapproach.Itseemsthatitisnecessaryforresearcherstoestablishaclearconnectionbetweenitemsandtheirtheoreticaldomain(Hinkin,1995).Indevelopingitems,researchersoftenemployavarietyofstrategies,includingpullingfromexistingmeasures,interviewingselectpopulations,andusingfocusgroups.SeeKrause(2002)foradescriptionofanine-stepstrategyfordevelopingitems.Forexample,Wilsonandcolleagues(2010)usedstudentsintheirupper-levelpsychologycoursetogeneratepotentialitemsfortheirprofessor-studentrapportmeasure.ExpertReviewandPilotTestingOncetheinstrumentiscreated,researchersoftengothroughreviewprocessesbeforefullyadministeringtheinstrument.Suchprocessesincludehavingapanelofexpertsreviewtheitemsforfeedbackregardingwhethertheitemsareclearandreflecttheconstructofinterest(DeVellis,2012;Worthington&Whittaker,2006).Oncethescaleiscreated,itshouldalsobepilot-testedonarelativelysmallsampleofthepopulationofinteresttodeterminewheretheremightbewordingconfusionorredundancy,whethertherearecompletiontimeissues,andwhetherresponseitemscoveralldesiredoptions(vanTeijlingen&Hundley,2001).Forexample,VanTassel-Baska,Quek,andFeng(2007)pilotedtheirteacherobservationscaleand

33

foundtheir40-itemmeasurecouldbereducedto25itemstodecreaseredundancy,whichcanimpactreliabilityasdiscussedbelow.Researchersmightalsousecognitiveinterviewtechniqueswithparticipantsaftertheycompleteanewlydevelopedinstrumenttogarnerfeedbackabouttheinstrumentaswellashaveparticipantsthinkaloudwhileansweringeachsurveyquestion(Ryan,Gannon-Slater,&Culbertson,2012).Thesefindingsshouldinformmodificationstotheinstrument.Suchprocessesareespeciallyimportantwhenspecialpopulationsareinvolved(e.g.,children,non-nativespeakers,etc.)(Presseretal.,2004).InstrumentAdministration

DeliveryOptionsOncetheinstrumentiscomplete,itisreadytobefullyadministered.However,itisalsoimportanttoconsiderthemanypracticalandlogisticalissuesrelatedtoadministeringtests(Nosek,Banaji,&Greenwald,2002;Wright,2005).Forexample,aresearchermightwonderwhetheritismoreappropriatetohavestudentparticipantscompleteinstrumentsin-personoronline(Milleretal.,2002).Manyscholarshaveoptedtocollectdataonlineduetobenefitssuchaslowercostsandincreasedaccesstoparticipants(Milleretal.,2002;Wright,2005).Inaddition,researchershavefoundcomparablereliabilityandvalidityofinstrumentsadministeredonlineversusin-personwithpaperandpencil(Meyerson&Tryon,2003;Milleretal.,2002).Furthermore,researchershavefoundonlinesamplestoapproximatethecharacteristicsoftraditionalsamples(Gosling,Vazire,Srivastava,&John,2004).

SurveyDesignResearchersalsoneedtobecognizantinhowthedesignofthesurveycanpotentiallyimpactparticipantresponses(e.g.,completiontimeandvalidresponses)(Christian&Dillman,2004;Couper,Traugott,&Lamias,2001).Forexample,Mahon-HaftandDillman(2010)showedthatthedesignofonlinesurveys,includingaestheticsandtypeofopenresponseformat,cannegativelyimpactthequalityofdatacollected(e.g.,omissionsandshorterresponses).Thisissueiscruciallyimportanttoreliabilityandvalidity,bothofwhicharediscussedfurtherbelow.

SampleSizeAdditionalconcernsrelatedtoinstrumentadministrationisadequatesamplesizeandthecharacteristicsofthesample.Itisworthnotingthatalthoughtherearevaryingbeliefsinwhatconstitutesanadequatesamplesizewhendevelopinganinstrument(e.g.,atleast150,300,varyingvariableratiossuchas20:1),itissafetosaythelargerthesamplesize,thebetter(Schmitt,2011;Williams,Onsman,&Brown,2010;Yong&Pearce,2013).Samplesizeandreliabilitybecomecriticallyimportantwhenconsideringtheirimpactonpowertodetectaneffectandtheabilitytoreproduceobservedeffects(Fraley&Vazire,2014;Henson,2001;Simons,2014).Inregardtoscaledevelopment,samplesizeisalsoimportantinthatsmallersamplesizescanlimitthetypeofanalysesthatcanbeperformed(Noar,2009).Itisalsoimportantforresearcherstoconsiderthedemographiccharacteristicsoftheparticipantswho

34

arecompletingtheinstrument,astestbiasandcertainethicalissuesmightbeofconcern(vandeVijver&Tanzer,1997).Inaddition,researchersshouldattempttorecruitparticipantswhoareheterogeneousontheconstruct(Clark&Watson,1995).AnalyzeResponsesAfterresearchersadministertheirinstrumenttoalargegroupofparticipants,theybeginanalyzingtheirfindings.Theanalysestypicallyinvolvedininstrumentdevelopmentincludefactoranalysis,reliabilityestimates,andvaliditytests.However,itisimportanttonotethattheseareoftennottheonlyanalysesinvolved.Giventhatthereisnotnearlyenoughroominthisentiree-booktocovereachtopicfully,thefollowingdiscussiontouchesuponsomeofthemainconceptsrelatedtoanalyzingitemsforinstrumentdevelopmentinordertofamiliarizereadersandguidethemtowardadditionalreadings.

FactorAnalysisAsdiscussedfurtherbelow,researcherscannotdirectlymeasurelatentvariables.Instead,theyrelyonobservableitems/variablesthatarebelievedtoreflectthelatentvariableandthenworktodeterminehowwelltheyhaveachievedthisgoal.Oftentimes,measuresofreliabilityarenotadequateininformingthedimensionalityofaninstrument(John&Benet-Martinez,2000).Inotherwords,aninstrumentwithahighalphaleveldoesnotnecessarilymeanthatalltheitemsreflectasingleconstructorfactor(DeVellis,2012).Forexample,theTeachingBehaviorsChecklist(TBC)(Keeley,Smith,&Buskist,2006),discussedinmoredepthlaterinthise-book,canbeusedasasinglefactorinstrumentforidentifyingeffectiveteachingbehaviorsoraninstrumentwithtwofactorsrelatedtoevaluatingteachingbehaviors.Oneofthemosteffectivewaystodeterminehowitemsrelateandreflectalatentvariable(s)isusingfactoranalysis(Byrne,2001).Factoranalysisisasetofproceduresthatallowthedeterminationofhowmanyfactorsthereareandhowwelltheitemsreflectthosefactors.Therearetwomaintypesoffactoranalysis:exploratoryfactoranalysis(EFA)andconfirmatoryfactoranalysis(CFA),withtheformerbeingwheremoststartwhenfirstdevelopingorrevisinganinstrument(Yong&Pearce,2013).Noar(2009)describesthestepsofscaledevelopmentincludingfactoranalysisusingastructuralequationmodeling(SEM)technique,whichisbecomingmorepopular(Worthington&Whittaker,2006).Giventhespaceconstraintsofthischapterandthevastamountofinformationandconsiderationsrelatedtofactoranalysis(Schmitt,2011),readersareencouragedtoexamineprimers(e.g.,Williamsetal.,2010;Yong&Pearce,2013),texts(e.g.,Thompson,2004),andresearchedbestpractices(Worthington&Whittaker,2006)formorein-depthinformationregardingfactoranalysisinscaledevelopment.Asdiscussedearlier,researchersgenerallystartwithapoolofitemsthattheyadministertoalargegroupofparticipants.Throughtheproceduresinvolvedinfactoranalysis,researcherscandetermineiftheirpoolofitemsreflectasinglefactorormultiplefactors(Yong&Pearce,2013).GoingbacktotheTBC(Keeleyetal.,2006)example,theauthorsusedthisinstrumentasanevaluativetoolandthroughfactoranalysisfoundall28itemscouldbeusedasasinglefactorto

35

reflectoverallteacherbehaviorsorseparatedintotwosubscales(ortwofactors)representingcaring/supportiveteachingbehaviorsandprofessionalcompetency/communicationskills.Researchersalsousefactoranalysisforreductionpurposes,includinglimitingthenumberofitemsinaninstrumentaswellasthenumberoffactorsincludedinaninstrument(Yong&Pearce,2013).Forexample,anotherscalediscussedfurtherinChapter11ofthise-bookistheSatisfactionwithLifeScale(SWLS)(Diener,Emmons,Larsen,&Griffin,1985;seeLayous,Nelson,&Legg,2015).Theseauthorsreviewedtheliterature,foundpreexistingscalesoflifesatisfactiontobeinadequate,andtherefore,begantheprocessofdevelopinganewinstrument.Inthefirstphase,theydeveloped48itemsrelatedtosatisfactionwithone’slife,whichwasdecreasedto5itemsbyusingfactoranalysis.Proceduressuchasthesehelpresearcherstofindwaystomoreeffectivelymeasureaconstruct,whilealsoreducingparticipantfatiguewithansweringnumerousitemsandincreasingthelikelihoodofreplicatingfindings(Henson&Roberts,2006).Itisalsoimportanttonotethatstatisticalprogramswillonlyshowwhereandhowitemsload;itisuptotheresearchertointerpretthefindingsusingtheliteratureand/oratheoreticalframework(Henson&Roberts,2006;Yong&Pearce,2013).Forexample,deletingitemsassuggestedbySPSSdoesnotalwaysincreasereliability(Cho&Kim,2015).Dependingonthefindingsofthefactoranalysis,researchersmightgobacktotheliteratureanddevelopadditionalitemstopilotordeterminetheinstrument’sstructureandassessitsreliabilityandvalidity(Warner,2013).ReliabilityRegardlessofwhetheraninstrumentbeingusedforSoTLispreexisting,anadaptedmeasure,ornewlydeveloped,itisimportanttoassessitsreliability.Generallyspeaking,reliabilityreferstohowconsistentlyaninstrumentmeasuresacertainconstructorvariableofinterest(John&Benet-Martinez,2000).Scoresontheinstrumentshouldnotchangeunlessthereisachangeinthevariable.Itisnotpossibletocalculatereliabilityexactly,asthereisinevitablemeasurementerror.Therefore,researchershavetoestimatereliability.Indoingso,allattemptstoremoveorcontrolsourcesoferrorshouldbemadesoastoobtainatruerscoreofthelatentvariable(DeVellis,2012).Thequestionthenbecomes,howdoweknowifaninstrumentisreliable?Themethodsforassessingreliabilitycanvarydependingontypeofdataandinformationsought(John&BenetMartinez,2000);however,theyareallbasedontheideathatreliabilityistheproportionofvarianceoftheobservedscorethatcanbeattributedtothetruescoreofthelatentvariablebeingexamined(DeVillis,2012).Manyresearcherswillreportasinglecoefficientalpharegardingreliability(Hogan,Benjamin,&Brezinski,2000),butthatisalimitedpictureofreliabilityandnotalwaysthebestindicator(Cho&Kim,2015;Cronbach&Shavelson,2004;John&Benet-Martinez,2000).Forexample,Cortina(1993)demonstratedhowalphacanincreasewiththenumberofitemsinaninstrumentdespitelowaverageitemintercorrelations.Therefore,itisimportanttoexaminewhetheradditionalreportsofitemrelationshipsand/orreliabilityaregiven(Schmitt,1996).Thereisalsosomewhatofachallengeindetermining

36

whetherreportsofreliabilityareadequate.Forexample,mostresearchersuseacutoffforalphaat.70orhigherasadequate(Hoganetal.,2000;Schmitt,1996);however,othersnote.80orhighertobetherecommendedlevelsforalpha(Henson,2001;Lance,Butts,Michels,2006).However,higherisnotalwaysbetterasanoverlyhighalphamightbetheresultofredundancyintheitems(Ryan&Wilson,2014;Streiner,2003;Tavakol&Dennick,2011).RyanandWilson’s(2014)briefversionoftheProfessor-StudentRapportScalewithaCronbach’salphaof.83isagoodexampleofacceptableinternalconsistencythatisnotsohighastosuggestissueswithredundancy.Someofthemostcommonmethodsofassessingreliabilityincludeinter-observeragreement,test-retestreliability,andmeasuresofinternalconsistency.Whatfollowsisabriefdiscussionofthesecommonwaysofassessingreliabilityaswellasadditionalresourcesfacultycanexaminetolearnmoreaboutthesetopics.Itisimportanttokeepinmindthatthisdiscussionisabriefintroductionintoconceptsrelatedtoreliability;entirebooksaredevotedtothistopic(e.g.,seeFink&Litwin,1995;Litwin,2003).

Inter-ObserverAssessinginter-observeragreementcomesintoplaywhenresearcherswanttolookatthereliabilityofresponsesgivenbytwoormoreratersorjudges(Warner,2013).Dependingonthegoaloftheresearcher,thelevelofagreementbetweenraterscouldbeallowedtovaryornot.Forexample,inapsychologystatisticsclass,twoteachersshouldbeinperfectagreementregardingwhetherastudentcorrectlycomputedaspecificstatisticalanalysis.Incontrast,agroupofteachingassistantsmightbeaskedtocomeintoaclassroomandjudgestudents’posterprojectsforthebestdesignedpostertoexamineifacertainteachinginterventionwaseffective.Inthiscase,perfectscoreagreementmightnotbeneededorbeneficial.Instead,wemightlookathowthejudgesrankedthepostersfrombesttoworst.Researcherscanalsoexaminepercentageofagreementbetweenratersbycountingthenumberoftimestheraterswereinagreementanddividingbythetotalnumberofjudgmentsmade.However,thismethodofassessmentdoesnottakeintoaccountchancelevelsofagreementbetweenratersandtendstoworkbestwhentheratedvariableisobjectiveratherthansubjective.Totakeintoaccountchanceagreements,onewouldneedtocalculateCohen’skappa(kscoefficient(Viera&Garrett,2005),withcoefficientsbelow.40beingpoor,between.40and.59fair,.60and.74good,and.75and1.00excellent(Cicchetti,1994).Sourcesofmeasurementerrortobemindfulofincludejudges’backgroundandtrainingregardingthevariableofinterestandthetoolbeingusedaswellasprevalenceofthefinding(DeVillis,2012;Guggenmoos-Holzmann,1996;Viera&Garrett,2005).

Test-RetestReliableinstrumentsshouldprovideconstantscoresifusedtoassessavariableatonetimepointandthenagainatanothertimepoint(solongastheoreticallythatvariableshouldremainconstantacrosstime);however,theerrorofmeasurementeachtimewillvary.Acorrelationcouldthenbecomputedtoassessconsistencyinscoresforthetwoassessmentpoints.

37

Althoughthereissomevariabilityintheliterature,thereappearstobeconsensusthataretestreliabilitycoefficientof.80orhigherisrecommended(Polit,2014).Forexample,teachersmightbeinterestedintheirpsychologystudents’levelofsatisfactionwiththeiracademicadvisorsandmeasuresaidlevelsattwopointsintime.Althoughthemeasurementoferrorwilldifferateachassessmentpoint,thescoringofsatisfactionshouldstayconstantiftheinstrumenttrulyreflectslevelsofsatisfaction.However,itisimportanttonotethatchangesinscoringfrompointonetopointtwocanvaryforanumberofreasonsandnotnecessarilyreflect(un)reliability.Forexample,actualchangesinthevariableofinterest(e.g.,anoverallincreaseordecreaseinstudents’levelofsatisfactionwiththeiradvisors)wouldaffecttest-retestreliabilityscoreswithoutreflectingunreliability.Inaddition,test-retestreliabilitycanbeaffectedbyparticipantfactorssuchaspurposelytryingtoanswerconsistently(ordifferently)eachtimetheycompletetheinstrument(DeVillis,2012;Polit,2014).

InternalConsistencyAsmentionedpreviously,weareunabletodirectlymeasurelatentvariables;instead,werelyonproxies.Ifwearetoassumetheitemsofaninstrumentrepresentthelatentvariableofinterest,thenthestrengthoftherelationshipbetweenthoseitemsshouldreflectthestrengthofthemeasurementofthelatentvariable(DeVillis,2012).ThemostcommonwaytodeterminewhetheraninstrumentisinternallyconsistentistocomputeCronbach’salphaforitemsthathavemultipleresponseoptionsortheKuder-Richardsonformula20(KR-20)foritemsthathavedichotomousresponses.Inadditiontocomputingacoefficientalpha,researchersmightincludeameasureofsplit-halfreliability.Exactlyhowtheitemsaresplit(e.g.,inthemiddle,oddsvs.evens,etc.)dependsonthetypesofitemsinthemeasurementaswellasthegoalsoftheresearcher.Forexample,ameasureregardinggay,lesbian,bisexual,andtransgenderstudents’perceptionsofdiversityinclusionincurriculumthatissplitinthemiddlemighthaveanunbalancednumberofsurveyitemsfocusedonlesbianstudents.Inthiscase,itmightmakemoresensetoensuresubgroupperceptionswereequallydistributedintotwohalves.Overall,whencomputinganalphathereareanumberoffactorsresearchersneedtoconsider,includingthenumberofitemsintheinstrumentandwhethertherearereverse-scoreditemsasthesecouldaffectreliabilityscores(Warner,2013).ValidityWhetherpreexistingornewlydeveloped,itisalsoimportanttoexaminethevalidityofinstruments.Generally,validityindicateshowwellaninstrumentmeasureswhatitclaimstomeasure;however,thismetricofaninstrumentissomewhatmoredifficulttodetermine(Warner,2013).Althoughsomehavepurportedmultipletypesofvalidity,thegeneralconsensusappearstobearoundthreetypes:contentvalidity,constructvalidity,andcriterion-relatedvalidity(DeVellis,2012).FurtherdiscussionofvalidityofinstrumentsintheSoTLresearchiscoveredinChapter5ofthise-book.

38

ContentValidityAlthoughtherearesomedifferencesinhowcontentvalidityisdefinedintheliterature,“thereisgeneralagreementinthesedefinitionsthatcontentvalidityconcernsthedegreetowhichasampleofitems,takentogether,constituteanadequateoperationaldefinitionofaconstruct”(Polit&Beck,2006,p.490).However,contentvalidityisoftenlefttothejudgmentoftheresearcherand/orapanelofexpertstoensureonlyrelevantitemsrelatedtotheconstructareincluded(DeVellis,2012).Oneissuewithstoppingatthisstepisthatitisasubjectivejudgmentofwhetheraninstrumenthasadequatecontentvalidity.Someresearcherssuggestthecomputationofacontentvalidityindex(CVI)tobetterjudgethepsychometricsofaninstrument(Polit&Beck,2006;Wynd,Schmidt,&Schaefer,2003).Genericallyspeaking,thisinvolveshavingapanelofratersrevieweachiteminthemeasure,ratehowrelevanteachitemistotheconstructbeingmeasured,andthentheresearcher(s)assessagreementbetweenraters.

ConstructValidityConstructvalidityisbasedonatheoreticalframeworkforhowvariablesshouldrelate.Specifically,“itistheextenttowhichameasure‘behaves’thewaythattheconstructitpurportstomeasureshouldbehavewithregardtoestablishedmeasuresofotherconstructs”(DeVellis,2012,p.64).Somedeemconstructvalidityasoneofthemostimportantconceptsinpsychology(John&Benet-Martinez,2000;Westen&Rosenthal,2003).However,thereisnoagreeduponorsimplewaytoassessconstructvalidity(Bagozzi,Yi,&Phillips,1991;Westen&Rosenthal,2003).Instead,manyresearchersusecorrelationstoshowrelationshipsbetweenvariablesassuggestedbytheliterature,oftenreferredtoasconvergentanddiscriminantvalidity(Messick,1995).Forexample,RyanandWilson(2014)examinedthevalidityofabriefversionoftheProfessor-StudentRapportScalebycomparingthebriefmeasuretoscalesthatweresimilar(e.g.,ImmediacyScale,towhichitshouldpositivelyrelate)anddissimilar(i.e.,VerbalAggressivenessScale,towhichitshouldnegativelyrelate).Therearealsocomputationsthatcanbeperformedtodemonstratelevelsofconstructvalidityincludingconfirmatoryfactoranalysis,effectsizecorrelations,andstructuralequationmodeling(Bagozzietal.,1991;Westen&Rosenthal,2003).

Criterion-RelatedValidityCriterion-relatedvalidityismainlyconcernedwithpracticalityversustheoreticalunderpinningslikeconstructvalidity;“itisnotconcernedwithunderstandingaprocessbutmerelywithpredictingit”(DeVellis,2012,p.61).However,manyconfuseconstructvalidityandcriterion-relatedvaliditybecausethesameexamplelistedabovecanbeusedforeithertypeofvalidity;thedifferenceisintheintentoftheresearcher(e.g.,toexplainorexplorevariablerelationshipsversussimplyidentifyingandpredicting)(DeVellis,2012).Inthecaseofthelatter,therearemultipletypesofcorrelationsthatcanbecomputedtoidentifyspecificrelationshipsamongvariablesandinturn,differenttypesofvalidity.Forexample,researcherscanexaminewhetherthescoresontheirinstrumentpredictfuturebehaviors(i.e.,predictivevalidity),correlatewithscoresonanotherinstrumentshowntomeasurethesameconstruct(i.e.,convergentvalidity),

39

and/ordonotcorrelatewithinstrumentsthatareunrelatedtotheconstructofinterest(i.e.,discriminantvalidity;Warner,2013).Inmanycases,researcherswillwantbothconstructandcriterion-relatedvalidityduetobasingtheirsurveydevelopmentintheoryandstrivingtowardtheabilitytopredictanoutcome.ConclusionsInsum,scaledevelopmentisaniterative,ratherthanalinearprocess.Itislikelythatresearchersmighthavetoreturntoearlierstepsinthescale-developmentprocessattimes.Inaddition,theprocessshouldalsobedeliberateandbeguidedbyatheoreticalfoundationwhereappropriate.Onlywhensuchaprocessisemployedcanapsychometricallysoundmeasurebedevelopedthatyieldsmeaningfuldata.Asthesayinggoes,“alittleknowledgeisadangerousthing,”yetthisisexactlywhatwasprovidedinthischapter.Thepurposewastofamiliarizereaderswithsomeofthemainconceptsandprocessesthatresearchersmustconsiderwhendevelopingormodifyingascale.Thehopeisthatreadersmightusetheeasilydigestiblesourcescitedinthischapterasaspringboardforfurtherskillbuildingrelatedtoscaledevelopment.

40

ReferencesBagozzi,R.P.,Yi,Y.,&Phillips,L.W.(1991).Assessingconstructvalidityinorganizational

research.AdministrativeScienceQuarterly,36(3),421-458.Boynton,P.M.,&Greenhalgh,T.(2004).Selecting,designing,anddevelopingyour

questionnaire.BMJ,328,1312-1315.doi:10.1136/bmj.328.7451.1312Buskist,W.,Sikorski,J.,Buckley,T.,&Saville,B.K.(2002).Elementsofmasterteaching.InS.F.

Davis&W.Buskist(Eds.),Theteachingofpsychology:EssaysinhonorofWilbertJ.McKeachieandCharlesL.Brewer(pp.27–39).Mahwah,NJ:LawrenceErlbaumAssociates,Inc.

Byrne,B.M.(2001).StructuralequationmodelingwithAMOS:Basicconcepts,applications,andprogramming.Mahwah,NJ:LawrenceErlbaumAssociates,Publishers.

DeChamplain,A.F.(2010).Aprimeronclassicaltesttheoryanditemresponsetheoryforassessmentsinmedicaleducation.Medicaleducation,44(1),109-117.doi:10.1111/j.1365-2923.2009.03425.x

Cho,E.,&Kim,S.(2015).Cronbach’scoefficientalpha:Wellknownbutpoorlyunderstood.OrganizationalResearchMethods,18(2),207-230.doi:10.1177/1094428114555994

Christian,L.M.,&Dillman,D.A.(2004).Theinfluenceofgraphicalandsymboliclanguagemanipulationsonresponsestoself-administeredquestions.PublicOpinionQuarterly,68(1),57-80.doi:10.1093/poq/nfh004

Christopher,A.N.(2015).Selectingtherightscale:Aneditor’sperspective.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Cicchetti,D.V.(1994).Guidelines,criteria,andrulesofthumbforevaluatingnormedandstandardizedassessmentinstrumentsinpsychology.PsychologicalAssessment,6(4),284-290.doi:10.1037/1040-3590.6.4.284

Clark,L.A.,&Watson,D.(1995).Constructingvalidity:Basicissuesinobjectivescaledevelopment.PsychologicalAssessment,7(3),309-319.

Cook,J.D.,Hepworth,S.J.,Wall,T.D.,&Warr,P.B.(1981).Theexperienceofwork.SanDiego,CA:AcademicPress

Cortina,J.M.(1993).Whatiscoefficientalpha?Anexaminationoftheoryandapplications.JournalofAppliedPsychology,78(1),98-104.doi:10.1037/0021-9010.78.1.98

Couper,M.P.,Traugott,M.W.,&Lamias,M.J.(2001).Websurveydesignandadministration.PublicOpinionQuarterly,65,230-253.doi:10.1086/322199

Cronbach,L.J.,&Shavelson,R.J.(2004).Mycurrentthoughtsoncoefficientalphaandsuccessorprocedures.EducationalandPsychologicalMeasurement,64(3),391-418.doi:10.1177/0013164404266386

DeVellis,R.F.(2012).Scaledevelopment:Theoryandapplications(3rded.).ThousandOaks,CA:Sage

Diener,E.,Emmons,R.A.,Larsen,R.J.,&Griffin,S.(1985).TheSatisfactionwithLifeScale.JournalofPersonalityAssessment,49(1),71-75.

Fan,X.(1998).Itemresponsetheoryandclassicaltesttheory:Anempiricalcomparisonoftheiritem/personstatistics.EducationalandPsychologicalMeasurement,58(3),357-381.

Fink,A.,&Litwin,M.S.(1995).Howtomeasuresurveyreliabilityandvalidity(Vol.7).ThousandOaks,CA:Sage.

41

Fraley,R.C.,&Vazire,S.(2014).TheN-PactFactor:Evaluatingthequalityofempiricaljournalswithrespecttosamplesizeandstatisticalpower.PLoSONE,9(10),e109019.doi:10.1371/journal.pone.0109019

Gosling,S.D.,Vazire,S.,Srivastava,S.,&John,O.P.(2004).Shouldwetrustweb-basedstudies?Acomparativeanalysisofsixpreconceptionsaboutinternetquestionnaires.AmericanPsychologist,59(2),93-104.doi:10.1037/0003-066X.59.2.93

Guggenmoos-Holzmann,I.(1996).ThemeaningofKappa:Probabilisticconceptsofreliabilityandvalidityrevisited.JournalofClinicalEpidemiology,49(7),775-782.doi:10.1016/0895-4356(96)00011-X

Hambleton,R.K.,&Jones,R.W.(1993).Comparisonofclassicaltesttheoryanditemresponsetheoryandtheirapplicationstotestdevelopment.EducationalMeasurement:IssuesandPractice,12(3),38-47.doi:10.1111/j.1745-3992.1993.tb00543.x

Henson,R.K.(2001).Understandinginternalconsistencyreliabilityestimates:Aconceptualprimeroncoefficientalpha.Measurement&EvaluationinCounseling&Development,34(3),177-189.

Henson,R.K.,&Roberts,J.K.(2006).Useofexploratoryfactoranalysisinpublishedresearch:Commonerrorsandsomecommentonimprovedpractice.EducationalandPsychologicalMeasurement,66(3),393-416.doi:10.1177/0013164405282485

Hinkin,T.R.(1995).Areviewofscaledevelopmentpracticesinthestudyoforganizations.JournalofManagement,21(5),967-988.doi:10.1016/0149-2063(95)90050-0

Hogan,T.P.,Benjamin,A.,&Brezinski,K.L.(2000).Reliabilitymethods:Anoteonthefrequencyofuseofvarioustypes.EducationalandPsychologicalMeasurement,60(4),523-531.doi:10.1177/00131640021970691

John,O.P.,&Benet-Martinez,V.(2000).Measurement:Reliability,constructvalidation,andscaleconstruction.InH.T.Reis&C.M.Judd(Eds.),HandbookofResearchMethodsinSocialandPersonalityPsychology(pp.339-369).NewYork:CambridgeUniversityPress.

Keeley,J.,Smith,D.,&Buskist,W.(2006).TheTeacherBehaviorsChecklist:Factoranalysisofitsutilityforevaluatingteaching.TeachingofPsychology,33(2),84-91.doi:10.1207/s15328023top3302_1

Kirk,C.,Busler,J.,Keeley,J.,&Buskist,W.(2015).Effectivetoolsforassessingcharacteristicsofexcellentteaching:TheTeacherBehaviorsChecklistasexemplar.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Kline,T.J.B.(2005).Classicaltesttheory.InT.J.B.Kline(Ed.),Psychologicaltesting:Apracticalapproachtodesignandevaluation(pp.91-105).ThousandOaks,CA.Sage.

Krause,N.(2002).Acomprehensivestrategyfordevelopingclosed-endedsurveyitemsforuseinstudiesofolderadults.JournalofGerontology:SocialScience,57B(5),S263-S274.

Lance,C.E.,Butts,M.M.,&Michels,L.C.(2006).Thesourcesoffourcommonlyreportedcutoffcriteria.Whatdidtheyreallysay?OrganizationalResearchMethods,9(2),202-220.doi:10.1177/1094428105284919

Layous,D.,Nelson,S.K.,&Legg,A.M.(2015).Measuringpositiveandnegativeaspectsofwell-beinginthescholarshipofteachingandlearning.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

42

Litwin,M.S.(2003).Howtoassessandinterpretsurveypsychometrics(Vol.8).ThousandOaks,CA:Sage.

Mahon-Haft,T.A.,&Dillman,D.A.(2010).Doesvisualappealmatter?Effectofwebsurveyaestheticsonsurveyquality.SurveyResearchMethods,4(1),43-59.doi:10.18148/srm/2010.v4i1.2264

Messick,S.(1995).Validityofpsychologicalassessment:Validationofinferencesfrompersons’responsesandperformancesasscientificinquiryintoscoremeaning.AmericanPsychologist,50(9),741-749.

Meyerson,P.,&Tryon,W.W.(2003).Validatinginternetresearch:Atestofthepsychometricequivalenceofinternetandin-personsamples.BehaviorResearchMethods,Instruments, &Computers,35(4),614-620.doi:10.3758/BF03195541

Miller,E.T.,Neal,D.J.,Roberts,L.J.,Baer,J.S.,Cressler,S.O.,Metrik,J.,&Marlatt,G.A.(2002).Test-retestreliabilityofalcoholmeasures:Isthereadifferencebetweeninternet-basedassessmentandtraditionalmethods?PsychologyofAddictiveBehaviors,16(1),56-63.doi:10.1037/0893-164X.16.1.56

Noar,SM.(2009).Theroleofstructuralequationmodelinginscaledevelopment.StructuralEquationModeling:AMultidisciplinaryJournal,10(4),622-647,doi:10.1207/S15328007SEM1004_8

Nosek,B.A.,Banaji,M.R.,&Greenwald,A.G.(2002).E-research:ethics,security,design,andcontrolinpsychologicalresearchontheinternet.JournalofSocialIssues,58(1),161-176.doi:10.1111/1540-4560.00254

Polit,D.F.(2014).Gettingseriousabouttest-retestreliability:acritiqueofretestresearchandsomerecommendations.QualityofLifeResearch,23,1713-1720.doi:10.1007/s11136-014-0632-9

Polit,D.F.,&Beck,C.T.(2006).TheContentValidityIndex:Areyousureyouknowwhat’sbeingreported?Critiqueandrecommendations.ResearchinNursing&Health,29,489-497.

Presser,S.,Couper,M.P.,Lessler,J.T.,Martin,E.,Martin,J.,Rothgeb,J.M.,&Singer,E.(2004).Methodsfortestingandevaluatingsurveyquestions.PublicOpinionQuarterly,68(1),109-130.doi:10.1093/poq/nfh008

Ryan,K.,Gannon-Slater,N.,&Culbertson,M.J.Improvingsurveymethodswithcognitiveinterviewsinsmall-andmedium-scaleevaluations.AmericanJournalofEvolution,33(3),414-430.doi:10.1177/1098214012441499

Ryan,R.&Wilson,J.H.(2014).Professor-studentrapportscale:Psychometricpropertiesofthebriefversion.JournaloftheScholarshipofTeachingandLearning,14(3),64-74.doi:10.14434/josotl.v14i3.5162

Schaeffer,N.C.,&Presser,S.(2003).Thescienceofaskingquestions.AnnualReviewofSociology,29(1),65-88.doi:10.1146/annurev.soc.29.110702.110112

Schmitt,N.(1996).Usesandabusesofcoefficientalpha.PsychologicalAssessment,8(4),350-353.doi:10.1037/1040-3590.8.4.350

Schmitt,T.A.(2011).Currentmethodologicalconsiderationsinexploratoryandconfirmatoryfactoranalysis.JournalofPsychoeducationalAssessment,29(4),304-321.doi:10.1177/0734282911406653

Simons,D.J.(2014).Thevalueofdirectreplication.PerspectivesonPsychologicalScience,9(1),76-80.doi:10.1177/1745691613514755

43

Streiner,D.L.(2003).Startingatthebeginning:Anintroductiontocoefficientalphaandinternalconsistency.JournalofPersonalityAssessment,80(1),99-103.doi:10.1207/S15327752JPA8001_18

Tavakol,M.,&Dennick,R.(2011).MakingsenseofCronbach’salpha.InternationalJournalofMedicalEducation,2,53-55.doi:10.5116/ijme.4dfb.8dfd

Thompson,B.(2004).Exploratoryandconfirmatoryfactoranalysis.Washington,DC:AmericanPsychologicalAssociation.

vandeVijver,F.J.R.,&Tanzer,N.K.(1997).Biasandequivalenceincross-culturalassessment:Anoverview.EuropeanReviewofAppliedPsychology,47(4),263-280.doi:10.9707/2307-0919.1111

VanTassel-Baska,J.,Quek,C.,&Feng,A.X.(2007).Thedevelopmentanduseofastructuredteacherobservationscaletoassessdifferentiatedbestpractice.RoeperReview:AJournalonGiftedEducation,29(2),84-92.doi:10.1080/02783190709554391.

vanTeijlingen,E.,&Hundley,V.(2001).Theimportanceofpilotstudies.SocialResearchUpdate,35,1-4.doi:10.1046/j.1365-2648.2001.01757.x

Viera,A.J.,&Garrett,J.M.(2005).Understandinginterobserveragreement:TheKappastatistic.FamilyMedicine,37(5),360-363.

Warner,R.M.(2013).Appliedstatistics:Frombivariatethroughmultivariatetechniques(2nded.).LosAngeles,CASage.Westen,D.,&Rosenthal,R.(2003).Quantifyingconstructvalidity:Twosimplemeasures.

JournalofPersonalityandSocialPsychology,84(3),608-618.doi:10.1037/0022-3514.84.3.608

Williams,B.,Onsman,A.,&Brown,T.(2010).Exploratoryfactoranalysis:Afive-stepguidefornovices.AustralasianJournalofParamedicine,8(3),1-13.

Wilson,J.H.Ryan,R.G.,&Pugh,J.L.(2010).Professor-studentrapportscalepredictsstudentoutcomes.TeachingofPsychology,37,246-251.doi:10.1080/00986283.2010.510976

Worthington,R.L.,&Whittaker,T.A.(2006).Scaledevelopmentresearch:Acontentanalysisandrecommendationsforbestpractices.TheCounselingPsychologist,34(6),806-838.doi:10.1177/0011000006288127

Wright,K.B.(2005).Researchinginternet-basedpopulations:Advantagesanddisadvantagesofonlinesurveyresearch,onlinequestionnaireauthoringsoftwarepackages,andwebsurveyservices.JournalofComputer-MediatedCommunication10(3).Retrievedfromhttp://onlinelibrary.wiley.com/doi/10.1111/j.1083-6101.2005.tb00259.x/full

Wynd,C.A.,Schmidt,B.,&Schaefer,M.A.(2003).Twoquantitativeapproachesforestimatingcontentvalidity.WesternJournalofNursingResearch,25(5),508-518.doi:10.1177/0193945903252998

Yong,A.G.,&Pearce,S.(2013).Abeginner’sguidetofactoranalysis:Focusingonexploratoryfactoranalysis.TutorialsinQuantitativeMethodsforPsychology,9(2),79-94.

44

Chapter5:TheStateofScaleValidationinSoTLResearchinPsychologyGeorjeannaWilson-Doenges

UniversityofWisconsin-GreenBay

Inpsychology,effortstooperationalizeandmeasureconstructsremainachallengingandcomplextask.Muchofthisworkreliesonself-reportandother-reportratingsthatrequiredevelopingitemsandscalestotapintocomplicatedpsychologicalconstructs.Asafield,psychologyhashighstandardsforassessingthereliabilityandvalidityofsuchscales.Thescholarshipofteachingandlearning(SoTL)shouldbeheldtothesesamehighstandardsofscalevalidationintheassessmentoflearning.AsNoar(2003)states,“Acrossavarietyofdisciplinesandareasofinquiry,reliableandvalidmeasuresareacornerstoneofqualityresearch”(p.622).ThepurposeofthischapteristoprovideabriefoverviewofthestateofscalevalidationinrecentSoTLinpsychologyandprovidesomeexemplarsofwaysinwhichSoTLresearchershaveemployedthebestpracticesforscalevalidation.Itshouldnotgounnoticedthatclassroomresearchisoftenmessy,addingalayerofcomplexitytoSoTLresearch.Duetothelackoftotalcontrolinthistypeofresearchenvironment,trouble-shootingandwork-aroundsarepartofaSoTLresearcher’stoolbox.Forexample,SoTLresearchersareoftenconstrainedbythesamplesizeandmake-upoftheirclasses(wheremuchSoTLisdone),aswellasthetimeconstraintofthelengthofatypicalsemester.Takingthetimetoassessreliabilityandvalidityundertheseconstraintscanbechallengingandcanevenrequireextensionoftheresearchoverseveralsemesters.Thiscanthenleadtoissueswithreliabilityandvalidityoverdifferentclasssamplesandinstructors,whichcouldhinderthequalityoftheresearch.SomeexamplesofwaystomaximizescalevalidationinSoTLresearchinpsychologywillalsobediscussed.MethodInordertoassessthestateofscalevalidationinrecentSoTLinpsychology,acontentanalysisofthelasttenissues(volume40,issue1tovolume42,issue2)ofaprominentSoTLjournalinpsychology,TeachingofPsychology,wasperformed.Onlyempiricalarticleswereexamined.Ofthe141articlesassessed,47articlesusedatleastonescaleintheresearch.Ascalewasdefinedasavariableforwhichmorethanoneitemwasusedtomeasurethesameconstruct.Wherescaleswerepresent,theassessmentandreportingofreliability,validity,andthepsychometricdevelopmentofthescaleswererecorded.Exemplarswerealsoidentifiedduringthiscontentanalysis.Results

AssessingandReportingReliability

AlphaScalevalidationhasseveralimportantcomponents,oneofthembeingtheinternalconsistencyofthescalesused.InternalconsistencyismostoftenmeasuredbycorrelationalstatisticslikeCronbach’salpha(Cronbach,1951).SoTLinpsychologyhasbeenconsistentinestablishingandreportinginternalreliabilityofscales.InthecontentanalysisofrecentissuesofTeachingof

45

Psychology,36ofthe47articles(76.6%)wherescaleswereusedtoassessSoTLoutcomes,internalreliabilitywasreported.Thirty-onestudies(66.0%)reportedCronbach’salphavaluesthatwereallacceptable(>.70;Cronbach,1951;DeVellis,1991),whereasfive(10.6%)reportedatleastonealphavaluethatwasbelowtheacceptable.70value.OneexampleofreportinginternalconsistencyisMaybury’s(2013)studyoftheinfluenceofapositivepsychologycourseonseveralmeasuresofstudentwell-beingusingpreviouslyestablishedscalesandreportingtheCronbach’salphaofeachscaleandsubscale.Thealphasinthisstudyrangedfrom.72to.89,allabovethe.70minimumforacceptablereliability(seeChapter11inthise-bookbyLayous,Nelson,&Leggformoredetailofthisstudy).Anotherexampleofreportinginternalconsistencyisastudyofateamapproachtoundergraduateresearchthatuseditemsthattheresearchersdeveloped,ratherthanapreviouslyestablishedscale(Woodzicka,Ford,Caudill,&Ohanmamooreni,2015).Inthisstudy,theresearchersdevelopeda19-item,5-pointLikert-scaleonlinesurveywithfoursubscales.EachsubscalehadaCronbach’salphaaboveacceptable,evenwitharelativelysmallsamplesize(Woodzickaetal.,2015).Reportingthereliabilityofnewlydevelopeditemsusedforthefirsttimeisparticularlyimportantbecauseofthelackofestablishedhistory.Although,themajorityofinternalconsistencyvaluesofthescalespublishedinSoTLinpsychologyexaminedwereabovetheacceptableminimum(83.3%ofthestudiesreportingacceptableCronbach’salphas),therewereseveralinstanceswhereatleastonescaleassessedwasnot.Oneparticularstudyaddressedthisissuedirectly,byassessingreliabilityandnotingthatitwasnotacceptableforoneofthesubscalesandthengivinganappropriaterationaleforstillusingthesubscaleintheresearch(Boysen,2015).Specifically,therationalewasthattherewasastrongtheoreticalrelationoftheitemsinthesubscaleandthattheitemsshowedequalpatternsofresults;therefore,theresearcherkeptthesubscaleinthemodeleventhoughitwasfoundtobeunreliable(Boysen,2015).AnotherexampleoftheuseofascalewithpoorreliabilityisBoysen,Richmond,andGurung’s(2015)studyofmodelteachingcriteriausingtheTen-ItemPersonalityInventory(TIPI;Gosling,Rentfrow,&Swann,2003).AlthoughtheCronbach’salphareliabilitycoefficientsforfourofthefivesubscaleswerebelow.70,theauthorsjustifiedtheirusebecauseofpoorreliabilityreportedinpreviousstudiesandthefactthateachsubscaleonlyhadtwoitems,makinggoodreliabilitylesslikely(Boysenetal.,2015).Thesearegoodexamplesofhowtohandlesituationswhendatathathavealreadybeencollectedarenotreliableorvalid.Anothercommonoccurrenceinthereportingofinternalconsistencyislistingthereliabilityofanestablishedscalefrompreviouslypublishedstudiesratherthanreportingthereliabilityofthatscaleinthepresentstudy.Ofthe47studiesthatreportedscales,30studiesreportedCronbach’salphasfromthecurrentstudyandsixreportedinternalconsistencyvaluesfrompreviouslypublishedstudies,butnotthecurrentsample.Anexampleofusingpreviouslypublishedinternalconsistencyvaluesisastudylookingatchangingstereotypesofolderadults(Wurtele&Maruyama,2013),asmeasuredbytheFraboniScaleofAgeism(FSA;Fraboni,Saltstone,&Hughes,1990).Theauthorsreportedsupportforinternalconsistency,constructvalidity,andtest-retestreliabilityfrompreviouslypublishedworkbutdidnotreportanysourcesofreliabilityfromthecurrentstudy(Wurtele&Maruyama,2013).Althoughusing

46

previouslyestablishedscaleswithprovenreliabilityandvalidityiscertainlyabestpractice,reportingcontinuedsupportforthepsychometricpropertiesofascalefurthersresearchers’confidenceintheiruseandapplicabilityindiversecontexts.Inaddition,becauseCronbach’salphaisaneasilyaccessiblestatistic,notreportingacurrentalphavaluemayseemliketheauthoristryingtohideapoorreliabilityvalue.AnexampleofcontinuedpsychometrictestingofanalreadyestablishedscalehasbeentheassessmentoftheTeacherBehaviorsChecklist(TBC;Keeley,Smith,&Buskist,2006)invariousstudieswithreportedreliabilityinthesedifferentsituations,includingadaptingtoundergraduateteachingassistants(Filz&Gurung,2013),andcorrelatingtheTBCwithacademicself-conceptandmotivation(Komarraju,2013).Asecondformofreliabilityisinter-raterreliability,establishedwhenmorethanonecoderobservesbehaviorsorratesqualitativedataandassessestheiragreementontheresultingrating.Inter-raterreliabilityalsohasastrongrecordinrecentpublishedSoTLresearchinpsychology.InlasttenissuesofTeachingofPsychologyexamined,17of22studies(77.3%)wherebehaviorsorwritingassignmentswereratedastheSoTLoutcome,inter-raterreliabilitywasassessedresultinginreliablecoding.Althoughnotperfectagreement,moststudieshadan80%agreementrateorbetter.Wheretherewerediscrepancies,moststudiesusedaprocessofconsensusontheratingsthatwerenotinagreementwhencodedindividually.Aparticularlygoodexampleofreportinginter-raterreliabilityandtheprocessofresolutionofdiscrepanciesisLawsonandCrane’s(2014)studyofideomotoractiontoincreasecriticalthinking.Inthestudy,bothanexperimentalandcontrolgroupwereaskedanopen-endedquestiontoprovideanexplanationforagivenphenomenon.Studentresponseswereindependentlycodedbytwoblindcodersforpresenceofideomotoractionintheirresponse.Theyachievedexcellentinter-raterraterreliability(96%)andthenstatedthat,“disagreementswerediscussedtoconsensus”(Lawson&Crane,2014,p.54).Thepracticeofdiscussionandconsensuswheretherearediscrepanciesincodingisabestpracticeintheliteraturewithseveralexemplarsofhowtoreporttheprocess(e.g.,Daniel&Braasch,2013;Simon-Dack,2014).Thelastformofreliability,test-retestreliability,istheassessmentofcorrelationofthesameoutcomeovertime(e.g.,atwotoeightweektimeperiod)tounderstandtheconsistencyofthescaleovertime.Test-retestreliabilitywasassessedfarlessoftenintheSoTLinpsychologyliteratureexaminedthantheothertypesofreliability.OnlytwoofthearticlesinthepasttenissuesofTeachingofPsychologyreportedtest-retestreliability(notincludingstudiessolelyfocusedonestablishingpsychometricpropertiesofthescaleused),andthosevalueswerefrompreviouslypublishedstudies,notfromthecurrentsample.Oneexampleofreportingtest-retestreliabilityasaroutinepartoftheMethodsectionisWurteleandMaruyama’s(2013)studyofolderadultstereotypes.Asdiscussedearlierinthischapter,theresearchersusedtheFSAscaletoteststereotyping,andintheirmaterialssectiontheyreportedtest-retestcorrelationcoefficientsfrompreviouslypublishedwork(Wurtele&Maruyama,2013).Perhapsbecausetest-retestreliabilityisusedtoestablishthatscalesdonotchangeovertime,thistypeofreliabilityisrarelyreportedinSoTLworkbecausechangeisexpectedovertimeinsomanySoTLprojects,makingtest-retestreliabilitypotentiallylessrelevant.

47

AssessingandReportingValidityInadditiontoformsofreliability,anotherimportantcomponentofthepsychometricpropertiesofscalesisvalidity.Asisnotedearlierinthise-book(seeHussey&Lehan,2015),thereareseveralformsofvalidity(e.g.,face,construct,criterion)whichconstitutewaysinwhichresearcherstestifvariablesareactuallymeasuringtheconstructstheyareintendedtomeasure.AlthoughreliabilitywasconsistentlyassessedandreportedinSoTLresearchusingscales,validitywasfarlesslikelytobeassessedandreported.Onlyeightof47(17%)publishedstudiesusingscalestomeasureSoTLoutcomesinthepasttenTeachingofPsychologyissues(notincludingscaledevelopment-focusedarticles)assessedanyformofvalidityandreportedthoseresults.Twoofthoseeightstudiesreportedpreviouslypublishedvalidityinformationratherthanvalidityfromthepresentstudy(e.g.,Wielkiewicz&Meuwissen,2014).OneexampleofastudythatassessedandreportedvalidityinthecurrentstudyisBuckelew,Byrd,Key,ThorntonandMerwin’s(2013)studyofperceptionofluckoreffortleadingtogoodgrades.Theresearchersadaptedascaleofattributionsforgoodgradesfromapreviousstudy,notingthatnopublishedvalidityinformationwasavailablefortheoriginalscale.Researchersthenestablishedvaliditythroughapilottestwheretheycorrelatedvaluesoftheiradaptedscaletoapreviouslyestablishedcriterionandfoundsignificantpositivecorrelationswiththetotalscaleandsubscales.Thisexampleofestablishingcriterionvaliditythroughapilottestisonewaytoensurevalidmeasuresbeforecollectingactualstudydata.However,itshouldbenotedthatvalidityistrulyestablishedovertimeandthroughmanystudies,providingacompellingreasonwhyroutinelyreportingvalidityisvitallyimportantdespitethedearthofstudiesassessingit.Validityinitsseveralforms(e.g.,content,construct,andcriterion)arecrucialtoresearchers’confidencethatthedatacollectedaremeasuringtheconstructstheywereintendedtomeasure.SoTLresearchersareurgedtoincorporatevaliditytestingandreportingintotheworktheydo.

TheDevelopmentandValidationofScalesTherehavebeensomegreatadvancesinthewaysinwhichscalesaredevelopedandvalidated.Thepracticeofusingbothexploratoryanalyses(e.g.,principalfactoranalysis)andthenconfirmatoryfactoranalysishasgreatlyenhancedresearchers’confidenceinthepsychometricpropertiesofthescalesusedtomeasureoutcomes(Noar,2003).Theincreasinglycommonuseofstructuralequationmodelinghasenhancedscaledevelopmentbyprovidingeasilyaccessiblewaystoconductconfirmatoryfactoranalysisandlatentvariablemodeling(Noar,2003).Therewerefivestudiesoutof141articlespublishedinthelasttenissuesofTeachingofPsychologydevotedtothedevelopmentandvalidationofascale.Thesestudiesemployedsomeofthemostthoroughmethodologyinscaledevelopmentandvalidation.OneparticularlygoodexampleofsuchscalevalidationisRenken,McMahan,andNitkova’s(2015)initialvalidationofthePsychology-SpecificEpistemologicalBeliefScale(Psych-SEBS).Inthistwo-partstudy,researchersfirstdraftedandthenrefinedanitempoolusingexploratoryfactoranalysis,followedbyconfirmatoryfactoranalysisandanassessmentofinternalconsistency,test-retestreliability,andconvergentvalidityofthePsych-SEBS.Inthesecondstudy,researchersassessedthecriterionvaliditybycomparingthePsych-SEBSwithanestablishedcriterionandalsotested

48

theincrementalvalidityofthePsych-SEBSaboveandbeyondthevarianceexplainedbythatcriterion.Byallmeasures,thePsych-SEBS13-itemscaleisareliableandvalidmeasureofpsychology-specificepistemologicalbeliefs,andthewayinwhichtheresearchersconductedthisscalevalidationisamodelforothersinthefield.AnotherexampleofscalevalidationisRogers’(2015)furthervalidationoftheLearningAllianceInventory(LAI).Usingcorrelationandpathanalysis,Rogersestablishedinternalandtest-retestreliability,criterionandconvergentconstructvalidity,aswellasestablishedthattheLAIpredictedstudentlearningbeyondthealreadyestablishedpredictorsofimmediacyandrapport(Rogers,2015).Thisstudyisanexemplarbecauseoftheuseofthemostrecentstatisticaltechniques(e.g.,pathanalysis)tothoroughlyassessallofthepsychometricpropertiesoftheLAIscale.Discussion

ReliabilityandValidityChallengesParticulartoSoTLResearchOneofthechallengesofdoingSoTLresearchisthattheclassroomisnotonlytheplacewheredataarecollected,butitisalsotheplacewhereteaching,learning,andassessmentarehappening.BecauseinstructorsareoftenteachingandsimultaneouslydoingSoTLresearchintheirownclasses,beingmindfulnottooverburdenthestudentsorcompromiselearninginthepursuitofdatacollectioncancreatechallenges.Sometimesusingexistingsourcesofdatamaketheoverlapofteachingandresearchalittlelesscomplicated,butsuchmethodsofovercomingSoTLresearchobstaclescancomeatacost.

StudentRatingsofInstructorsOneofthemostcommonsourcesofdataisstudentratingsofinstructors(SRIs),oftenusedtodeterminetheeffectivenessoftheteacherandthecourse(Heckert,Latier,Ringwald,&Silvey,2006).ThereisanextensivebodyofliteratureassessingtheimpactofdifferentlearningexperiencesonSRIs,includingtheinfluenceofimmersionscheduling(Richmond,Murphy,Curl,&Broussard,2015),professor-studentrapport(Ryan,Wilson,&Pugh,2011),humor(Richmond,Berglund,Epelbaum,&Klein,2015),andstudentengagement(Handlesman,Briggs,Sullivan,&Towler,2005).MostofthetimetheSRIthatisusedistheonethattheresearcher’shomeuniversityuses,whichnormallydoesnothaveatrackrecordofestablishedreliabilityandvalidity(althoughsometimestheseSRIsarereliable,e.g.Richmondetal.,2015).Infact,thereisagrowingliteraturenotingthelackofreliabilityandvalidityofSRIsinmeasuringstudentsatisfactionorlearning(Beran&Violato,2005;Catano&Harvey,2011;Clayson,2009;Heckertetal.,2006).Yet,SRIsstillremainaverypopularmeasureinSoTLresearch(andasadeterminingfactorinfacultymeritandpromotion)despitethisfact(Clayson,2009).However,itisimportanttonotethatSRIsarenotstructuredinawaytotrulymeasurelearning,butratherstudents’perceptionsofteachereffectiveness.Itmakessensetosaythataneffectiveteacherenhancesstudentlearning,butlearningisnottheintendedtargetconstructofSRIs.Inaddition,becauseSRIsarenotstandardizedacrossuniversities,comparingSRIsisnotpossibleintheircurrentform(Richmondetal.,2015).

49

GradesAnothercommonsourceofalready-existingdatainthelearningenvironmentisgradepointaverage(GPA).AlthoughGPAisgenerallythoughttobeareliableandvalidmeasureofacademicachievement,andthereforeagoodproxyforlearning,thatmaynotnecessarilybethecase(Imose&Barber,2015).TheliteraturehasestablishedthatGPAhasgoodinternalconsistency,althoughthetest-retestreliabilityandthecomparisonofGPAstandardsacrossschoolshavenotbeenestablished.Forexample,inonestudyfocusedonbusinessmajors,four-yearcumulativeGPAwasreliable(Cronbach’salpha=.94),showingexcellentinternalconsistency,buttheywerenotabletoassesstest-retestreliabilityorcompareacrossinstitutions(Bacon&Bean,2006).Thedifferentstandardsemployedbyuniversitiesalsoimpactcomparability.Forinstance,a3.5GPAatoneuniversitymaymakethestudentinthetop10%oftheclass,whereasthatsameGPAwouldbeinthetop25%atanotheruniversity(Imose&Barber,2015).Inaddition,althoughGPAmaybemeasuringgeneralcognitiveabilityandconscientiousness,GPAhasnotbeenestablishedasavalidmeasureoflearning.Forinstance,overallGPAhadstrongpredictivevalidityofindividualmeasuresofacademicperformance,butmainlywhenthatperformancewasreliablymeasuredandwasbasedontheindividualstudent’sabilityandeffort,notnecessarilytheirlearning(Bacon&Bean,2006).StudentratingsofinstructorsandGPAareeasilyaccessibleoutcomemeasuresinSoTLbecausethesevariablesarealreadybeingassessedaspartoftheuniversityclassexperience.Becauseoftheiraccessibility,usingalready-measuredvariablesreducestheburdenonstudents.However,theaccessibilityofthesesourcesofdatadoesnotmeanthattheyarethebestmeasuresoflearning.ContinuingtoestablishthereliabilityandvalidityofthesecommonlyusedmeasuresinSoTLindiversesituationsiscrucialtoensureconfidencethatthemeasuresarehighqualityandaretappingintotheconstructsweintendtomeasure. ConclusionReliabilityandvalidityareparamounttoresearchersacrossfieldstoinsurethatinformationgatheredandreportedismeasuringwhattheresearcherwantstomeasureandmeasuringitwell.IntheSoTLliteratureinpsychology,internalconsistencyhasbeenassessedandreportedveryconsistently,whileotherformsofreliabilityandvalidityhavenot.ThisisacallforSoTLresearcherstobemorethoroughintheirreportingofpsychometrics.PublishingthereliabilityandvalidityofscalesforeverynewstudywillelevatethequalityofthebodyofresearchinSoTL.Assessingthepsychometricpropertiesofnewlydevelopedscalesisalsoimportanttoprovideresearcherswithnovel,reliable,andvalidwaystomeasureSoTLoutcomes.ContinuingtoreportthereliabilityandvalidityofpreviouslyestablishedscalesalsoengendersmoreconfidencethatthewaysinwhichwemeasureSoTLoutcomesarepsychometricallysoundinvariedenvironmentsandwithdiversesamples.Particularlyimportantisincreasingthefocusonestablishingthevalidityofscales,especiallyduetothefindingthatthereportingofvaliditywassignificantlylowerthanreliabilityinrecentSoTLinpsychology.Lastly,inlookingatsomeofthecommonwayswemeasurelearning,particularlystudentratingsofinstructorsandGPA,wemustdobetterinstandardizingandmeasuringreliabilityandvaliditysothatwecanbuildasolidtrackrecordforthemostcommonlyusedmeasuresinthefield.ThiswouldincludestudiesacrossinstitutionswithstandardizedSRIandGPAstandardstoestablishreliabilityand

50

validitywithawiderandmorediversepopulation.Intheotherchaptersinthise-book,suggestionsforreliableandvalidwaystomeasurethemostcommonSoTLoutcomeshavebeencollected.UtilizingthesescalesandfurtheringthecasefortheirreliabilityandvaliditywillbevitallyimportanttothefuturesuccessofSoTLresearchandourunderstandingofhowstudentslearn.

51

ReferencesReferencesmarkedwithanasteriskindicateascale.*Bacon,D.R.,&Bean,B.(2006).GPAinresearchstudies:Aninvaluablebutneglected

opportunity.JournalofMarketingEducation,28,35–42.doi.org/10.1177/0273475305284639

Beran,T.,&Violato,C.(2005).Ratingsofuniversityteacherinstruction:Howmuchdostudentandcoursecharacteristicsreallymatter?Assessment&EvaluationinHigherEducation,30,593-601.doi:10.1080/02602930500260688

*Boysen,G.A.(2015).Usesandmisusesofstudentevaluationsofteaching:Theinterpretationofdifferencesinteachingevaluationmeansirrespectiveofstatisticalinformation.TeachingofPsychology,42,109-118.doi:10.1177/0098628315569922

*Boysen,G.A.,Richmond,A.S.,&Gurung,R.A.R.(2015).Modelteachingcriteriaforpsychology:Initialdocumentationofteachers’self-reportedcompetency.ScholarshipofTeachingandLearninginPsychology,1,48-59.doi:10.1037/stl0000023

*Buckelew,S.P.,Byrd,N.,Key,C.W.,Thornton,J.,&Merwin,M.M.(2013).Illusionsofagoodgrade:Effortorluck?TeachingofPsychology,40,134-138.doi:10.1177/0098628312475034

*Catano,R.M.,&Harvey,S.(2011).Studentimpressionsofteachingeffectiveness:DevelopmentandvalidationoftheEvaluationofTeachingCompetenciesScale(ETCS).Assessment&EvaluationinHigherEducation,36,701-717.doi:10.1080/02602938.2010.484879

Clayson,D.E.(2009).Studentevaluationsofteaching:Aretheyrelatedtowhatstudentslearn?Ameta-analysisandreviewoftheliterature.JournalofMarketingEducation,31,16-30.doi:10.1177/0273475308324086

Cronbach,L.J.(1951).Coefficientalphaandtheinternalstructureoftests.Psychometrika,16,297-334.

*Daniel,F.,&Braasch,J.L.G.(2013).Applicationexercisesimprovetransferofstatisticalknowledgeinreal-worldsituations.TeachingofPsychology,40,200-207.doi:10.1177/0098628313487462

DeVellis,R.F.(1991).Scaledevelopment:Theoryandapplications(AppliedSocialResearchMethodsSeries,Vol.26).London:Sage.

*Filz,T.,&Gurung,R.A.R.(2013).Studentperceptionsofundergraduateteachingassistants.TeachingofPsychology,40,48-51.doi:10.1177/0098628312465864

*Fraboni,M.,SaltstoneR.,&Hughes,S.(1990).TheFraboniscaleofageism(FAS):Anattemptatamoreprecisemeasureofageism.CanadianJournalonAging,9,56-66.

*Gosling,S.D.,Rentfrow,P.J.,&Swann,W.B.,Jr.(2003).AverybriefmeasureoftheBig-Fivepersonalitydomains.JournalofResearchinPersonality,37,504-528.doi.org/10.1002/tl.20071

Heckert,T.M.,Latier,A.,Ringwald,A.,&Silvey,B.(2006).Relationofcourse,instructor,andstudentcharacteristicstodimensionsofstudentratingsandteachingeffectiveness.JournalofInstructionalPsychology,28,161-168.

52

Hussey,H.D.,&Lehan,T.J.(2015).Aprimeronscaledevelopment.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Imose,R.,&Barber,L.K.(2015).Usingundergraduategradepointaverageasaselectiontool:Asynthesisoftheliterature.ThePsychologist-ManagerJournal,18,1-11.

*Keeley,J.,Smith,D.,&Buskist,W.(2006).Theteacherbehaviorschecklist:Factoranalysisofitsutilityforevaluatingteaching.TeachingofPsychology,33,84-91.doi:10.1207/s15328023top3302_1

*Komarraju,M.(2013).Idealteachingbehaviors:Studentmotivationandself-efficacypredictpreferences.TeachingofPsychology,40,104-110.doi:10.1177/0098628312475029

*Lawson,T.J.,&Crane,L.L.(2013).Dowsingrodsdesignedtosharpencriticalthinkingandunderstandingofideomotoraction.TeachingofPsychology,41,52-56.doi:10.1177/0098628313514178

Layous,K.,Nelson,S.K.,&Legg,A.M.(2015).Measuringwell-beinginthescholarshipofteachingandlearning.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

*Maybury,K.K.(2013).Theinfluenceofapositivepsychologycourseonstudentwell- being.TeachingofPsychology,40,62-65.doi:10.1177/0098628312465868Noar,S.M.(2003).Theroleofstructuralequationmodelinginscaledevelopment.

StructuralEquationModeling,10,622-647.*Renken,M.D.,McMahan,E.A.,&Nitkova,M.(2015).Initialvalidationofan

instrumentmeasuringpsychology-specificepistemologicalbeliefs.TeachingofPsychology,42,126-136.doi:10.1177/0098628315569927

*Richmond,A.S.,Berglund,M.B.,Epelbaum,V.B.,&Klein,E.M.(2015).a+(b1)Professor-studentrapport+(b2)humor+(b3)studentengagement=(Y)studentratingsofinstructors.TeachingofPsychology,42,119-125.doi:10/1177/0098628315569924

*Richmond,A.S.,Murphy,B.C.,Curl,LS.,&BroussardK.A.(2015).Theeffectofimmersionschedulingonacademicperformanceandstudents’ratingsofinstructors.TeachingofPsychology,42,26-33.doi:10/1177/0098628314562674

*Rogers,D.T.(2015).FurthervalidationoftheLearningAllianceInventory:Therolesofworkingalliance,rapport,andimmediacyinstudentlearning.TeachingofPsychology,42,19-25.doi:10.1177/0098628314562673

*Ryan,R.G.,Wilson,J.H.,&Pugh,J.L.(2011).Psychometriccharacteristicsoftheprofessor-studentrapportscale.TeachingofPsychology,38,135-141.doi:10.1177/0098628311411894

*Simon-Dack,S.L.(2014).Introducingtheactionpotentialtopsychologystudents.TeachingofPsychology,41,73-77.doi:10.1177/0098628313514183

*Wielkiewicz,R.M.,&Meuwissen,A.S.(2014).Alifelonglearningscaleforresearchandevaluationofteachingandcurriculareffectiveness.TeachingofPsychology,41,220-227.doi:10.1177/0098628314537971

*Woodzicka,J.A.,Ford,T.E.,Caudill,A.,&Ohanmamooreni,A.(2015).Asuccessful

53

modelofcollaborativeundergraduateresearch:Amulti-faculty,multi-project,multi-institutionteamapproach.TeachingofPsychology,42,60-63.doi:10/1177/0098628314549711

*Wurtele,S.K.,&Maruyama,L.(2013).Changingstudents’stereotypesofolderadults.TeachingofPsychology,40,59-61.doi:10.1177/0098628312465867

Section2:ScalesforUseintheScholarshipofTeachingandLearning

55

Chapter6:MeasuringLearningandSelf-EfficacyPamMarek,AdrienneWilliamson,andLaurenTaglialatela

KennesawStateUniversity

Learningandself-efficacyarecloselyintertwined.Inameta-analysisof36studiesineducationalsettings(fromelementaryschoolthroughcollege)withavarietyofperformancemeasures(e.g.,standardizedtests,coursegrades,GPA),perceivedself-efficacypositivelypredictedacademicperformance(Multon,Brown,&Lent,1991).Moreover,aburgeoningliteraturehasrevealedthatlearningandself-efficacybothrelatetomultiplevariablessuchasmotivation(Pintrich&DeGroot,1990),self-regulation(Pintrich,Smith,Garcia,&McKeachie,1991;Zimmerman&Martinez-Pons,1988),andmetacognitiveawareness(Schraw&Dennison,1994)thatpromotesuccess.Comparedtomostconstructsaddressedinthise-book,themeasurementoflearningisuniqueinseveralrespects.First,theconstructoflearningisbroaderthanmostothersaddressed.Second,thedefinitionoflearningisdebated.Forexample,Barronandcolleagues(2015)havepointedoutthatpositioninglearningasachangeinbehaviorattributabletoexperiencemaybetoolimitingtoapplyacrossdisciplines;instead,theysuggestthattheremaybegreaterconsensuswiththedefinitionoflearningas,“astructuredupdatingofsystempropertiesbasedontheprocessingofnewinformation”(p.406).Third,measurementoflearningvariesgreatlywithintwobroadtypesofassessment:formativeandsummative.Formativeassessmentinvolvesclassroomtechniquesthatinformbothstudentsandfacultyabouthowwellstudentsaregraspingconceptsin“realtime”(Wiliam&Black,1996).Anadvantageofthistypeofassessmentisthatitprovidesinstantfeedbackthatmaypotentiallyhighlightwaystoenhancethelearningprocess,sostudentsandfacultyareabletoinitiateimmediatechangesintheirbehaviors.Thistypeofassessmentcanscaffoldlearningforcurrentstudents,whereasonlyfuturestudentsmaybenefitfromend-of-semesterevaluations(Angelo&Cross,1993).Summativeassessmentinvolvesdeterminingwhatstudentshavelearnedinculminationaboutaparticulartopicorunitcomparedtospecificcriteriaataparticularpointduringinthelearningprocess(Dunn&Mulvenon,2009).Suchassessmentsincludebothgradedassignmentswithinacourseandstandardizedtestsattheendofaprogram(Wiliam&Black,1996).Amajordistinctionbetweenformativeandsummativeassessmentsrelatestohowtheresultsareusedratherthantheactualformatortypeofassessment.Infact,thesametypeofassessmentmaybeusedaseitheraformativeorsummativeevaluation,orevenforboth.Forexample,rubricscanbeusedineitherformativeorsummativecontexts.Quizzesmayalsoservemultiplepurposes.Althoughinstructorsoftenuseaquizasagraded,summativeassessmentattheendofaunit,iftheyprovidefeedbacktoclarifystudents’understanding,thenthequizwouldalsoserveaformativepurposeforfutureexaminations.

56

Severaltechniquesforassessmentoflearning,suchasexams,presentations,andpapers,maynotbeclassifiedasorevaluatedbyusingtraditionalscales.Theseassessmentstypicallyvaryamongcoursesandamonginstructorsteachingagivencoursetoreflectdifferencesinlearningobjectiveslinkedtocontentandskills.Consideringthenatureoflearning,bothasaprocessandanoutcome,evaluativetoolsoftenstretchbeyondtraditionalscales.Ideally,thesemeasurementsoflearningareconstructedandscoredtobeconsistentwithcourseandassignmentgoals.Montgomery(2002)discussestheimportantrolethatrubricsplayinevaluatingsuchassignments.Whenusingarubric,instructorsclearlydefinecriteriaforidentifyingdifferentlevelsofperformanceonspecificelementsofanassignment;studentsshouldusethesecriteriatoevaluatetheirownworkastheycomposeit(Montgomery,2002;Reddy&Andrade,2010).(SeeMueller,2014fordetailedinformationonrubricconstruction.)Inthischapter,wedescribebothformativeandsummativeassessments.Toillustrateformativeassessments,wediscussclassroomassessmenttechniques(CATs)becausetheyareoneofthemostcommontoolsinthiscategory.Wealsodiscussrubricsusedforformativepurposes.Asexamplesofsummativeassessments,wediscussrubricsforwritingassignmentsandstandardizedtests.Wefocusonwritingbecauseofitsprevalenceandimportanceinpsychologyandbecauseofthevalueofwritingskillsinmanyemploymentsettings.Forexample,inanationalsurvey,chairsof278psychologydepartmentsindicatedthenumberofcoursescompletedbyatleast80%oftheirstudentsthatemphasizedskillsspecifiedbytheAmericanPsychologicalAssociation(APA).Responsesindicatedthat59%ofprogramsemphasizedwritingAPAstylearticles,and68%ofprogramsemphasizedwritinginotherformsinatleastfouroftheirclasses(Stoloff,Good,Smith,&Brewster,2015).Regardingtheworkplace,inajoboutlooksurvey,73%ofemployersselectedwrittencommunicationskillsasavaluedattributeoncandidates’resumes(NationalAssociationofCollegesandEmployers,2015).Asasecondtypeofsummativeassessment,wechosetofocusonstandardizedtestsinsteadofcourseexamsbecauseofthevariabilitythatexistsforexamswithinandbetweenspecificcourses.Inaddition,coverageofstandardizedtestsallowsustoexpandourdiscussionfromclassroomevaluationtoprogramevaluation.Thechapterconcludeswithinformationaboutmeasuresofperceivedlearningandperceivedself-efficacy.FormativeAssessments

ClassroomAssessmentTechniquesAngeloandCross(1993)recommendedusingformativeassessmentsbeforepresentationofnewmaterialtoassesspriorknowledge,duringpresentationtodeterminestudents’levelofunderstanding,andafterpresentationtostrengthenstudents’learning.Itisimperativefortheinstructortoprovidetimelyfeedbackfromtheassessmentforthestudentstobenefitfully.TherearedozensoffrequentlyrecommendedCATsthatcanbeusedasformativeassessmentssuchasConceptMapping,Quizzes,Student-GeneratedExamQuestions,ReactionPapers,PollingtheClass,ProandConGrids,Think-Pair-Share(studentsareaskedtothinkbrieflyaboutananswertoaquestionprovidedbytheinstructor,thenpairupwithanotherstudenttodiscusstheiranswers,andfinallytosharetheinformationwiththeclass),andJigsaw(teamsofstudentsbecome“experts”withinaspecificarea,andtheneachmemberoftheoriginalteams

57

joinsanewteamtoteachthemabouttheirareaofexpertise).SeeAngeloandCross(1993)foradetaileddescriptionof50CATs.ComparisonsofclassesthatrepeatedlyusedCATswiththosethatdidnotuseCATsyieldinconsistentfindingsregardingtheireffectongrades.Repeateduseof5-question,multiple-choice,nongraded,formativequizzeswasfoundtoincreaseexamscoresinapsychologycourse(Short&Martin,2012).However,CottellandHarwood’s(1998)examinationoftheeffectofusingmultipleCATs(BackgroundKnowledgeProbe,MinutePaper,FeedbackForm,DirectedParaphrasing,ProandConGrid,WhatdidIlearnfromtheexam?,ClassroomAssessmentQualityCircleGroupInstructionalFeedbackTechnique,andGroup-WorkEvaluationForm)inanaccountingclassrevealednosignificantdifferencesbetweenclassesingradesforthecourseoronexams,groupprojects,orquizzes.Similarly,inacriminaljusticeclass,Simpson-Beck(2011)foundnosignificantdifferencesingradesforthecourse,onchaptertests,oronacumulativefinalexamwhenusingtheMuddiestPoint,wherestudentswereaskedtoidentifythemostconfusingpointfromtheday’sclass.Giventhevariabilityinusingformativeassessment(e.g.,choiceofCAT,course,frequencyofadministration,qualityofimplementation),additionalstudiesareneededtodeterminethebestpracticesofCATapplication.OneofthemostcommonlyusedCATsistheMinutePaper(alsoknownastheOne-MinutePaperortheHalf-SheetResponse),atwo-questionmeasurethathasbeendescribedbymultipleauthors(e.g.,Angelo&Cross,1993;Lom,2012;Stead,2005).ThefirstquestionontheMinutePaper(“Whatwasthemostimportantthingyoulearnedduringthisclass?”)requiresstudentstopracticeidentifyingthemostimportantpointsfromthelecture.Thesecondquestion(“Whatimportantquestionremainsunanswered?”)encouragesstudentstoreflectontheoverallmaterialtodeterminewhattheyunderstoodduringtheclassperiodandwhattheydidnotcomprehendascompletely.Thisassessmenttoolisbriefandeasytouse,andthequestionscanbeadaptedtoaddressmorefocused,course-specificmaterialthatcaninformaninstructor’sindividuallearningobjectives.Theinstructorcanquicklyreadandgrouptheresponsesandprovidefeedbackeitheratthebeginningofthenextclassorviaemailtotheentireclassortoindividualstudents.AngeloandCross(1993)providedanexampleoftheusefulnessofthisCAT.Attheendofeachclasssession,astatisticsinstructorusedaslightlymodifiedversionoftheMinutePaperbyaskingstudentstoselectthefivemostimportantpointsoftheday’sclassandaskoneortwoquestions.Initially,theinstructorfoundthattheclasscollectivelylistedapproximately20mainpoints,someofwhichwerelessimportantdetailsorincludedpartiallyorcompletelyincorrectinformation.Thisdiscoverypromptedtheinstructortomodifyhisteaching;atthebeginningofeachclass,helistedmanyofthepointsthestudentshadsubmittedanddiscussedtheirrelativeimportance.Additionally,helistedthepointsheconsideredmostimportant,comparedthelists,andaddressedcommonquestions.Thisapproachreducedthestudents’listofmainpointsfrom20tolessthan10withinamonth.TheseresultsshowhowtheuseofasimpleCATcanprovideinformationaboutstudents’levelofunderstandingandcanleadtoimmediatechangesinteaching.

58

Ingeneral,manyfacultyandstudentswhohaveusedCATshavereportedperceivedbenefitsthatincludebetterstudentlearning(Anderson&Burns,2013;Stead,2005),moreactiveengagement,andanenhancedclassroomenvironment(Soetaert,1998).However,CottellandHarwood(2005)foundnodifferenceinperceivedlearningbetweenstudentsinclasseswhousedCATsandclassesthatdidnotuseCATs.InareviewoftheMinutePaper,Stead(2005)foundthatmostresearchersreportedsignificantlyhighertestscoresforstudentsincourseswherethistoolwasused.Chiou,Wang,andLee(2014)alsofoundevidenceofenhancedlearningandreportedanadditionalbenefitofreducedcourse-relatedanxietyinastatisticscourse.However,bothAngeloandCross(1993)andSteadcautionagainsttheoveruseofthisassessmenttoavoidit“beingseenasagimmick”(p.153,Angelo&Cross)orbecomingtediousforboththestudentsandinstructor.AlsoseeAngeloandCrossandSteadforfurtherdiscussionoftheadvantagesanddisadvantagesofthisassessmenttool.

FormativeRubricsGreenberg(2015)evaluatedarubric(originallydesignedforsummativeuse)foritspotentialutilityasaformativetool.TherubricisforanAPA-styleresearchreportandcontains60learningobjectivesencompassingcontent,expression,andformatting.Eachoutcomeisratedona4-pointscalerangingfromabsenttoachieved.Studentsinintroductorypsychology(n=78)andadvanced(n=60)psychologycoursesweretoldtousetherubricwhencreatingawritingassignmentspecifictothecourse,whereasstudentsinothersectionsofthesamecourseswerenotprovidedwiththerubric(n=68,n=58,respectively).Studentswhousedtherubricscoredhigheronthewritingassignmentthanstudentswhodidnotusetherubric,indicatingthatutilizationoftherubrichasformativebenefits.Greenberg(2015)alsofoundtherubrictobeusefulduringrevision.Afterstudentsusedtherubricprescriptivelytopreparetheirownpapers,eachstudentwasgivenapeer’spaperandaskedtogradeitusingtherubric.Studentswerethengiventheopportunitytorevisetheirownpapers;therewassignificantimprovementinpaperqualityfollowingthisrevision.Overall,resultsindicatedthatrubricswerehelpfultoolsduringboththewritingandrevisingphasesofpaperproduction.Lipnevich,McCallen,Miles,andSmith(2014)comparedtheformativebenefitofusingdetailedrubricsversusexemplarpapers.Aftercompletingaroughdraft,studentswererandomlyassignedtooneofthreegroups:rubric,exemplarpapers,orrubricandexemplarpapers.Studentswereinstructedtousethesematerialstorevisetheirpapers.Resultsindicatethatallthreegroupsdemonstratedsignificantimprovementfromfirsttoseconddraft;however,therubric[only]groupshowedthemostimprovement(Cohen’sd=1.54).Therubrichas10dimensionsthatfollowthebasiccomponentsofaresearchpaper(e.g.,descriptionofresearchproject,studydesign,studymaterials).Eachdimensionisscoredonascalerangingfrom1(belowexpectation)to3(exceedsexpectation).Examplesareprovidedintherubrictohelpdistinguishbetweengradelevelswithineachdimension.

59

SummativeAssessments

SummativeRubricsAlthoughAPAlistsseveralstandardizedassessmentsdesignedtomeasurevariousaspectsofcommunication(e.g.,CollegiateAssessmentofAcademicProficiencyWritingEssayTestandWritingSkillsTest,WorkKeysFoundationalSkillsAssessment,andCollegiateLevelAssessment),andtheAssociationofAmericanCollegesandUniversities(AACU)providesaWrittenCommunicationrubric(https://www.aacu.org/value/rubrics/written-communication),nonefocusoncommunicationinpsychology,ormoreglobally,onscientificwriting.Althoughwriting,specificallyscientificwriting,istypicalinnaturalandsocialsciencecurricula,fewstandardizedrubricstoevaluatesuchworkhavebeendeveloped,tested,andmadeaccessible.Inthissection,weidentifyempirically-testedrubricsforsummativeuseinscientificwritingthatcanbeeasilyaccessedandimplementedoradaptedforclassroomuseinpsychology.Stellmack,Konheim-Kalkstein,Manor,Massay,andSchmitz(2009)createdandevaluatedthereliabilityandvalidityofarubricusedtoscoreAPA-styleresearchpapers.TheresearchersfocusedononlytheIntroductionsectionoftherubric;however,updatedandexpandedrubricsforallsections(e.g.,Introduction,Method,andResults/Discussion)areavailableathttp://www.psych.umn.edu/psylabs/acoustic/rubrics.htm.StudentswereinstructedtowriteanAPA-styleIntroductionincorporatingfivesources.Therubricencompasseseightdimensionsthatarefundamentalaspectsofthistypeofwritingassignment:APAformatting,literaturereview,purposeofstudy,studydescriptionandhypothesis,overallorganization/logicalflow,sources,scientificwritingstyle,andcomposition/grammar/wordchoice.Studentsearnupto3pointsforeachdimension,andexamplesofthegradelevelswithineachdimensionareprovided.Forexample,ifeitherthestudydescriptionorthehypothesisismissing,then1pointwouldbeearnedforthestudydescriptionandhypothesisdimension.Alternatively,ifboththedescriptionandhypothesisareprovided,buteitherisunclear,then2pointswouldbeearned.ScoresfortheIntroductionsectionrangefrom0to24.Stellmacketal.(2009)evaluatedtheinterraterandintraraterreliabilityofthisrubric.Forthreegraders,interraterreliabilitywasdefinedbothliberally(i.e.,scoresforadimensionwerewithin1pointacrossthethreegraders)andconservatively(i.e.,scoresforadimensionwereequalacrossthethreegraders).Agreementrangedfrom.90(liberal)to.37(conservative).Forintraraterreliability,gradersre-evaluatedasubsetoftheiroriginalpapers2weeksaftertheinitialassessment.Agreementrangedfrom.98(liberal)to.78(conservative).Insuchconsistencyestimates,valuesatorabove.70areviewedasacceptable(Stemler,2004).Beyondpsychology-basedwriting,theBiologyThesisAssessmentProtocol(BioTAP)providesasystematicmethodtoassessscientificwritinginthebiologicalsciences,andthecomponentsoftherubricareapplicableacrossnaturalandsocialsciencedisciplinesandcouldbeadaptedforusewithinpsychology.Reynolds,Smith,Moskovitz,andSayle(2009)usedBioTAPtoevaluateundergraduatethesesinbiology.Therubrichasthreecategories:higher-orderwritingissues(e.g.,“Doesthethesismakeacompellingargumentforthesignificanceofthestudent’sresearchwithinthecontextofthecurrentliterature?”),mid-andlower-orderwritingissues(“Is

60

thethesisclearlyorganized?”),andqualityofscientificwork.Acrossthethreecategories,thereare13individualcriteria,eachscoredona3-pointscale(i.e.,no,somewhat,yes).Thecompleterubricandinstructionalmethodcanbeaccessedatwww.science-writing.org/biotap.html.Reynoldsandcolleagues(2009)evaluatedtheinterraterreliabilityforthenineitemsthatcomprisedthehigher-,mid-,andlower-orderissues.Theitemsinthequalityofscientificworkdimensionrequirespecializedknowledgeofthepapercontent,andbecausetheevaluatorshaddifferentareasofexpertisewithinthebiologicalsciences,thiscategorywasnotincludedinthereliabilityassessments.Thepercentagreementforindividualcriteriawithinthehigher,mid-,andlower-orderedissuesrangedfrom.76to.90,withCohen’skapparangingfrom.41to.67,indicatingsufficientreliability.Timmerman,Strickland,Johnson,andPayne(2011)createdauniversalrubricforassessingscientificwritingcalledtheRubricforScienceWriting.Therubricwasdesignedtoevaluatestudents’empiricalresearchreportsinagenetics,evolution,orecologycourse.Therubricisusedtoevaluate15dimensionsincludingthoserelatedtoanintroduction,hypotheses,method,results,discussion,useofprimaryliterature,andwritingquality.Eachdimensionisscoredona4-pointscalerangingfromnotaddressedtoproficient.SimilartoStellmackandcolleagues’(2009)rubric,examplesofgradelevelswithineachdimensionareprovided.Timmermanandcolleagues(2011)assessedreliabilityusinggeneralizability(g)analysis(Crick&Brennan,1984).Thisassessmentdeterminestheamountofvariationinraterscoresthatcanbeattributedtoactualdifferencesinthequalityofthepaperandseparatesoutvariationduetoindividualratersorrater-assignmentinteractions(Brennan,1992).Usingthismetric,ascoreof1.0indicatesthatallofthevariationsinthescoresareduetodifferencesinthequalityofthestudentpapersandascoreof0.0indicatesthatnoneofthevariationinthescoresisduetodifferencesinthequalityofthepaper.Generalizabilitycoefficientsatorabove.80aretypicallyconsideredacceptable(Marzano,2002).GeneralizabilityanalysisfortheRubricforScienceWritingindicatesthat85%(g=.85)ofthevariationinscoresacrossthreebiologylaboratorypaperswasattributabletoactualdifferencesinthequalityofthestudentwork.Therefore,applyingtherubricundersimilarcircumstancesshouldproducereliablescores.

StandardizedAssessmentsStandardizedassessmentsusedtoevaluatelearningneartheendofanundergraduateprograminpsychologyincludetheEducationalTestingService’s(ETS)MajorFieldTestinPsychology(MFT-P;https://www.ets.org/mft/about/content/psychology),theAreaConcentrationAchievementTestinPsychology(ACAT-P)byPACAT,Inc.(http://www.collegeoutcomes.com),andtheETSGraduateRecordExaminationSubjectTestinPsychology(GRE-P;https://www.ets.org/gre/subject/about/content/psychology).Eachoftheseassessmentsmeasuresknowledgeabouttopicsandconceptstypicallytaughtinpsychologycoursesthatareoftenpartofthemajor.Inthemostrecentdataavailableforeachtest,theMFT-Phasbeenusedby357programs(25,895individuals)andtheACAT-Phasbeenusedbyapproximately170

61

programs.Inanationalsurveyofpsychologyprograms(Stoloff,2015),50of278(18%)programsindicatedtheyusedoneofthesetwomeasures.TheMFT-Pisa2-hrtestthatincludes140multiple-choicequestionsandprovidesscoresforindividualstudentsanddepartmentalmeansonthetotaltestandfoursubtests:learning(~5–7%ofthetest),cognition(~9–11%),andmemory(~3–5%);sensoryandperception(~3–5%)andphysiology(~10–12%);clinicalandabnormal(~10–12%)andpersonality(~6–8%);anddevelopmental(~10–12%)andsocial(~10–12%).Departmentalmeansarealsoprovidedforsixassessmentindicators:memoryandcognition;perception,sensation,andphysiology;developmental;clinicalandabnormal;social;andmeasurementandmethodology.Totalscoreandsubscoresarereportedasscaledscores.Thescaledrangeforthetotalscoreis120to200andforthesubscoresis20to100.AComparativeDataGuide(https://www.ets.org/s/mft/pdf/acdg_psychology.pdf)isprovidedbyETSthatincludestablesofscaledscoresandpercentilesthatcanbeusedtocomparetotalscoresandsubscoresforindividualstudentsandtocomparemeansoftotalscores,subscores,andassessmentindicatorscoresforinstitutions.ThecomparativedatasetincludesallU.S.seniorswhoweretestedusingthemostrecentversionofthetest.ResearchershaveexaminedfactorsrelatedtoperformanceontheMFT-P.BothoverallGPAandpsychologyGPAwerecorrelatedwithMFT-Ptotalscoreandallsubscores;moststudiesfoundthatallSATscoresandthetotalnumberofpsychologycreditscompletedwerecorrelatedwiththeMFT-Ptotalscoreandmostofthesubscores(Dolinsky&Kelley,2010;Gallagher&Cook,2013;Stoloff&Feeney,2002[foranexception,seePinter,Matchock,Charles,&Balch,2014]).However,completionofonlyafewspecificcourseswasrelatedtoMFT-Pperformance.Thesespecificcoursesvariedsomewhatamongstudies,butincludedacombinationofAbnormalPsychology,CounselingPsychology,PhysiologicalPsychology,SocialPsychology,andHistoryandSystems(Dolinsky&Kelley,2010;Gallagher&Cook,2013;Stoloff&Feeney,2002).TheACAT-P(http://www.collegeoutcomes.com)canbeusedtoassess13contentareas:abnormal,animallearningandmotivation,clinicalandcounseling,developmental,experimentaldesign,historyandsystems,humanlearningandcognition,organizationalbehavior,personality,physiological,sensationandperception,social,andstatistics.Thecontentofthisexamisflexibleandallowsdepartmentstotailoritscontentbychoosingtheareastheywishtoassess.Departmentshavethechoiceofassessing10areas(in120min),eightareas(in96min),sixareas(in72min),orfourareas(in48min).TheACAT-Pcanbeusedtoassessgeneralknowledgeofseniorsattheendoftheirprogramorasapre-posttestassessmenttoexaminechangeinknowledgeovertheundergraduateprogram.Asanationalcomparisongroup,meanscoresineachcontentareawithinpsychologyareprovidedforgraduatingseniorswhocompletedthetestduringeachyearforthelast13years.TheGRE-P(https://www.ets.org/gre/subject/about/content/psychology)hasapproximately205multiple-choice,five-optionquestions.Studentsreceiveanexperimentalsubscore,asocialsubscore,andatotalscore.Thetwoscaledsubscorescanrangefrom20to99inone-point

62

increments,andthetotalscaledscorecanrangefrom200to990in10-pointincrements.Theexperimentalsubscoreisbasedon40%ofthetestquestionsandisrelatedtolearning(3-5%),language(3-4%),memory(7-9%),thinking(4-6%),sensationandperception(5-7%),andphysiological/behavioralneuroscience(12-14%).Thesocialsubscoreisbasedonapproximately43%ofthetestquestionsandisrelatedtoclinicalandabnormal(12-14%),lifespandevelopment(12-14%),personality(3-5%),andsocial(12-14%).Thetotalscoreisbasedonquestionsthatcontributetothetwosubscoresalongwiththeremainingtestquestionsthatarerelatedtootherareas,includinggeneral(4-6%)andmeasurementandmethodology(11-13%).Testadministrationtakes2hr50min.AccordingtoETS(https://www.ets.org/gre/subject/about/content/psychology),researchhasprovidedevidenceforconstruct,content,predictive,andexternalvalidityoftheGREsubjecttests.

PerceivedLearningAssessment

TwoItemsManyinstructorsandresearchershavestruggledwithhowtobestmeasurelearninginawaythatisnotbasedonthespecificcontentwithinacourse.Thisdilemmahasledsomeindividualstousemeasuresofperceivedcognitivelearning.Forexample,Richmond,Gorham,andMcCroskey(1987)createdtheirowntwo-itemmeasureofperceivedlearning.Onthismeasure,studentsareaskedtoindicateona10-pointscale,rangingfrom0(nothing)to9(morethaninanyotherclass),howmuchtheyhadlearnedintheclassandhowmuchtheythinktheycouldhavelearnedintheclassiftheyhadtheidealinstructor.Thescoreonthefirstitemmaybetakenasalearningscorewhereasa“learningloss”scorecanbederivedbysubtractingthefirst-itemscorefromthesecond-itemscore.Subsequentstudieshaveprovidedevidenceforthetest-retestreliabilityofboththelearning(.85)and“learningloss”(.88)scores(McCroskey,Sallinen,Fayer,Richmond,&Barraclough,1996)aswellasthecriterionvalidity(“learningloss”scoreswerecorrelatedwithquizgrade)(Chesebro&McCroskey,2000)ofthisperceivedlearningmeasure.

LearningIndicatorsScaleFrymier,Shulnian,andHouser(1996)developedanassessmentforlearnerempowerment,whichincludedalearningindicatorsscale(αreliability=.84)basedonnineitemsrelatedtobehaviorsthatstudentsmayengageintoenhancelearning.However,FrymierandHouser(1999)suggestedthatfourofthenineitemsontheoriginalscalewererelatedtocommunication,andthisfocusmightintroduceapotentialconfoundforstudentswhotendtobeapprehensiveaboutcommunication.Thus,FrymierandHousercreatedarevisedlearningindicatorsscalethateliminatedthisproblem.Thenewscalecontainedsevenitems(fournewitemsandthreefromtheoriginalscale)onwhichstudentsratehowoftentheyengageingivenbehaviorsona5-pointLikertscalerangingfrom0(never)to4(veryoften).Thescaleisreliable(α=.85)andshowsbothconstructvalidity(scoreswerepositivelycorrelatedwithinstructornonverbalimmediacy,learnerempowerment,andstatemotivation–allconstructstypicallyrelatedtolearning)aswellascriterionvalidity(scoreswerepositivelycorrelatedwithameasureofaffectivelearningandanassignmentgrade).

63

CourseOutcomesScaleCentraandGaubatz(2005)examinedtheCourseOutcomesScaleonETS’sStudentInstructionalReport(SIRII;https://www.ets.org/sir_ii/about/)asameasureofperceivedstudentlearning.Thisscaleincludesfiveitemsthatareratedona5-pointscalerangingfrom1(muchmorethanmostcourses)to5(muchlessthanmostcourses).Thefiveitemsaddressperceptionsoflearningofcoursecontentspecifically(2items)aswellasmoregenerallearningoutcomes(3items).OthercomponentsandscalesontheSIRII,includingscoresontheOverallEvaluation,StudentEffortandInvolvementScale,andtheAssignments,Exams,andGradingScaleweresignificantpredictorsofperceivedlearningasmeasuredbytheCourseOutcomeScale.Perceptionsoflearningmeasuredoncourseevaluationsarerelatedtooverallcoursesatisfactionandratingofcourseinstructor.

StudentEstimatesAnothermeasureofperceivedlearningisstudents’self-assessmentoftheirperformanceonanassignment.Thetypeandtimingoftheseself-reportsofperceivedlearningvary.Students’estimatesmaybemadeinresponsetoasinglequestion(e.g.,provideanoverallestimateofexpectedperformanceonagradedmeasureoflearning)oronanitem-by-itembasisonthemeasure(Schraw,2009).Estimatescanbemadepriorto(prediction)orafter(postdiction)completingtheactualknowledgeassessment.Moderatecorrelationsbetweenstudents’estimatesandinstructors’scoresonperformancemeasureshavebeenreportedintwometa-analyses;FalchikovandBaud(1989)foundameancorrelationof.39(basedon45correlationcoefficients),andSitzmann,Ely,Brown,andBauer(2010)foundameancorrelationof.34(basedon137effectsizes).Thereareanumberoffactorsthatcaninfluencethestrengthoftherelationshipbetweenperceivedandperformancemeasuresoflearningincludingcompetenceofthelearner(experts[Kruger&Dunning,1999]andhigher-performingstudents[Boletal.,2005;Hackeretal.,2000]mademoreaccurateself-reports;higher-performingstudentstendedtounderestimatetheirlevelofperformance(Boletal.,2005),whereaslower-performingstudentsweremorelikelytooverestimateit(Boletal.,2005;Hackeretal.,2000),deliverymode(strongercorrelationinface-to-faceandhybridthaninonlinecourses[Sitzmannetal.,2010]),congruenceofmeasures(strongercorrelationwhentheperceivedandperformancemeasuresweresimilar[Sitzmannetal.,2010]),andemphasisofself-assessment(strongercorrelationwhenself-reportwasbasedonlevelofknowledgethanongaininknowledge[Sitzmannetal.,2010]).Twoadditionalfactorsthatinfluencethisrelationshiparepracticeandfeedback.

FeedbackAspreviouslystated,AngeloandCross(1993)emphasizedtheimportanceofprovidingfeedbackaboutaccuracyofcontentknowledgewhenusingCATs.Feedbackregardingperformanceonsummativeassessmentsisalsoimportantforlearning.Additionally,feedbackregardingaccuracywhenindividualspracticedself-assessingknowledgestrengthenedtherelationshipbetweenperceivedandperformancemeasuresoflearning(Sitzmannetal.,2010).

64

Hackerandcolleagues(2000)foundthatself-reportaccuracyimprovedwithpracticeonlyforhigher-performingstudents;lower-performingstudentsdidnotimprovewithpractice.InthestudiesthatSitzmannandcolleagues(2010)includedintheirmeta-analysis,useofperceivedlearningmeasuresasanindicatoroflearningvariedbydiscipline,rangingfrom17%formedicaleducationto79%forcommunication.Withinthe51psychologystudiesthatwereincluded,22%usedself-assessmentasameasureoflearning.However,ofthe15psychologystudiesincludedinthismeta-analysisthatexaminedtheaccuracyofself-assessmentmeasuresbasedonsimilaritybetweenstudentestimatesandinstructor-providedscores,40%foundtheywereaccurate,13%foundtheywereinaccurate,and47%reportedmixedresults.Giventhelackofevidenceforaccuracyofthistypeofperceivedmeasureoflearning,Sitzmannetal.(2010)recommendedusinggradedworkasindicatorsoflearning.Althoughthevalueandinterpretationofperceivedlearningaccuracymeasuresareasubjectofdebate(Rovai,Wighting,Baker,&Grooms,2009),evidencesuggeststhattheiraccuracymayincreasewithtraining.Moreover,perceptionsoflearningareassociatedwithperceptionsofotherlearning-relatedconstructs.Forexample,ameta-analysisrevealedpositivecorrelationsbetweenself-assessmentsofknowledgeandmotivation(ρ=.59)andself-efficacy(ρ=.43,Sitzmannetal.,2010).

Self-EfficacySelf-efficacyreferstotheextenttowhichindividualsarecapableof,orperceivethemselvestobecapableof,achievingdesignatedoutcomes(Bandura,1977).Inthischapter,allmeasuresinvolveself-reportsofperceivedself-efficacy.GroundedinBandura’s(1977)cognitivesociallearningtheory,efficacyexpectancies(theperceivedabilitytocompleteatasksuccessfully)differfromoutcomeexpectancies(theassumptionthatifanactioniscompleted,thenanoutcomewillbeachieved).Forexample,students’outcomeexpectanciesmaybethatstudyingregularlywillleadthemtodowellinacourse.Thecorrespondingefficacyexpectancywouldbetheirperceptionofwhethertheyarecapableofstudyingregularly.Inanacademiccontext,ameta-analysisrevealedthatself-efficacywaslinkedtobothenhancedacademicperformanceandpersistence(Multonetal.,1991).Ithasalsobeenlinkedtoself-regulatedlearning,goalsetting,anduseoflearningstrategies(Zimmerman,2000).Onepowerfulpredictorofself-efficacyisperformanceaccomplishments(priorsuccessatatask).Otherpredictorsincludevicariousexperiences(watchingotherssucceed),verbalpersuasion,andinterpretationofemotionalarousal(Bandura,1977).Althoughself-efficacymaybemeasuredasaglobalconstruct(Chen,Gully,&Eden,2001),itspredictivepowerisgreaterwhenitismeasuredinaspecificdomain(Bandura,Barbaranelli,Caprara,&Pastorelli,1996).Inthissection,wedescribeinstrumentsforassessingself-efficacyineducationalcontexts.First,wediscussmeasuresthatfocusexclusivelyonacademicself-efficacy(e.g.,masteryofsubjectareacontentandrelatedskills).Second,wereviewmeasuresthatassessperceptionsofbothacademicandsocialself-efficacy,operationalizedbysuchitemsasmakingnewfriends,talkingtouniversitystaff,andaskingquestionsinclass.Finally,we

65

discussTheMotivatedStrategiesforLearningQuestionnaire(MSLQ,Garcia&Pintrich,1995;Pintrichetal.,1991;Pintrich,Smith,Garcia,&McKeachie,1993),awidely-usedinstrumentthatmeasuresself-efficacyinconjunctionwithothermotivationandlearning-relatedconstructs.Giventheconnectionsbetweenself-efficacyandmultipleconstructsrelatedtoacademicsuccessandtoacademicperformanceitself,measuringself-efficacymayservemultiplepurposesintheclassroom.Forexample,knowingstudents’currentlevelofself-efficacymayhelplaunchdiscussionsaboutconstructsrelatedtoself-efficacyandstrategiestoimproveit.Theclassroomdiscussionshouldincludecoverageofpositiveacademicbehaviorssuchasrecommendedstudyhabits(Hoigaard,Kovac,Overby,&Haugen,2015),self-regulatedlearning,andgoalsetting.Improvingself-efficacycanincreasemotivationincurrentstudentsandmayencouragestudentstobecomelife-longlearners.Tobuildrealisticself-efficacybeliefs,studentsshouldbetrainedtodevelopspecificskillsneededforeffectiveperformance(Galyon,Blondin,Yaw,Nalls,&Williams,2012).Animportantcaveatrelatestoensuringthatstudents’individualself-efficacybeliefsarewellcalibratedwiththeiractualperformance(DiBenedetto&Bembenutty,2013).

MeasuresofAcademicSelf-EfficacyAmongthescalestargetedstrictlyatacademicself-efficacy,Chemers,Hu,andGarcia(2001)developedaneight-item,reliable(α=.81)measureonwhichstudentsratedtheextenttowhichstatementsappliedtothemona7-pointLikertscalerangingfrom1(veryuntrue)to7(verytrue).Findingsindicatedthatself-efficacywaspositivelyrelatedtobothacademicexpectations(futureperformance,meetinggoals)andinstructorevaluationsofclassroomperformance.Studentswithhigherself-efficacyalsotendedtohavestrongerself-ratedcopingskillsrelativetoexpectedlevelsofpressure.EliasandLoomis(2002)developedtheAcademicSelf-EfficacyScale(ASES),whichdiffersfromothersdiscussed.Insteadofratingsofconfidenceinperformingspecifictasks,theASESincludesratingsofconfidenceinsuccessfullycompleting18specificgeneraleducationand5specificphysicaleducationcourseswithatleastagradeofB.Additionally,studentsratetheirconfidenceinachieving13academicmilestones,suchas“earnacumulativeGPAofatleast2.0(or3.0)after2yearsofstudy,”“successfullypassallcoursesenrolledinoverthenextthreesemesters,”and“graduate.”Itemsareratedona10-pointLikertscalewithansweroptionsrangingfrom0(noconfidenceatall)to9(completeconfidence).Thethreescalefactorsalldemonstratedacceptablereliability(α=.86to.94).Self-efficacyforbothgeneralcoursesandacademicmilestoneswerepositivelycorrelatedwithneedforcognition(NFC;Cacioppo,Petty,&Kao,1984).Inaddition,overallself-efficacyandNFCsignificantlypredictedGPA,withself-efficacyservingasamediatoroftherelationshipbetweenNFCandGPA.EliasandLoomisnotedthatthepatternofresultssuggestedthatenjoymentofacademics,asreflectedbyNFC,enhancedperceptionsofself-efficacy.Anothermeasurefocusingonconfidence,theAcademicBehaviouralConfidence(ABC)Scale(Sander&Sanders,2009),includes24items(α=.88).Studentsindicateconfidenceintheir

66

abilitytoaccomplisheachitemona5-pointresponsescalewithnotatallconfidentandveryconfidentastheanchorpoints.Afour-factormodel(grades,verbalizing,studying,andattendance)with17itemsemergedaspreferableinconfirmatoryfactoranalyses.UsingtheABCscale,Putwain,Sander,andLarkin(2013)investigatedtherelationshipsamongacademicself-efficacy,learning-relatedemotions,andacademicsuccess(astandardized,weightedscoreofassessmentsduringthesemester).Findingsindicatedthatconfidenceinstudyingabilitiesatthebeginningofthefirstsemesterpositivelypredictedperformanceduringthatsemesterandpositivelearning-relatedemotionsatthebeginningofthenextsemester.Theresearcherssuggestedthatself-efficacyasitappliestostudyingmayrelatetoenhancedself-regulatedlearning,which,inturn,mayrelatetoperceivingdifficulttasksaschallengesratherthanthreats.

MeasuresofAcademicandSocialSelf-EfficacyEncompassingsocialaswellasacademicself-efficacy,Solberg,O’Brien,Villareal,Kennel,andDavis(1993)developedandvalidatedtheCollegeSelf-EfficacyInventory(CSEI)usingaHispanicpopulation,butitisalsoappropriateforusewithotherstudentpopulations.Theinventoryincludedalistof20tasks.Studentsindicatedtheirconfidenceincompletingeachtaskusingascalerangingfrom0(notatallconfident)to10(extremelyconfident).Afactoranalysisyieldedthreefactorsthatencompassed19ofthetaskswithfactorloadingsgreaterthan.50.Thethreefactorsrelatedtocourses,roommates,andothersocial-typetasks.Overallreliability(α=.93)andsubscalereliability(α=.88foreachfactor)weresatisfactory.Allthreeself-efficacyfactorswerenegativelycorrelatedwithpsychologicaldistressasmeasuredbytheBriefSymptomInventory(Derogatis&Melisaratos,1983)withrsrangingfrom-.44to-.53.Gore(2006)reportedthatcorrelationsofCSEIself-efficacyscoreswithGPAwerenotablyhigherattheendofthesecondandthirdsemesters(rs=.35and.21,respectively)thantheywereatthebeginningofthefirstsemester(rsfrom.00to.13).Thisfindingsupportstheimportanceofperformanceaccomplishmentsasaninfluenceonself-efficacy.GPAwasmorecloselyassociatedwithcourse-relatedself-efficacythanwithsocialself-efficacy.Toexaminetheeffectsofbothacademicandsocialself-efficacyandstressonacademicoutcomes,Zajachova,Lynch,andEspenshade(2005)developeda27-itemlistoftasks.Studentsprovidedtworatingsforeachtask.Ontheefficacyscale,theyratedtheirconfidenceinsuccessfullycompletingthetask,usinga0(notconfident)to10(extremelyconfident)scale.Onthestressscale,studentsratedhowstressfuleachtaskwas,usinga0(notstressful)to10(verystressful)responsescale.Analysesindicatedthatbothscaleshadthesamefourfactors:interactionatschool;performanceinclass;performanceoutofclass;andmanagingwork,family,andschool.Reliabilitiesforthefourfactorsubscalesrangedfromα=.72to.87.Foreachfactorpair,self-efficacynegativelycorrelatedwithstress.Findingsrevealedthatself-efficacywasapositivepredictorofGPAbutwasunrelatedtoretentioninthesophomoreyear.Incontrast,stresswasnegatively,butnotsignificantly,relatedtoGPA.

67

TheMotivatedStrategiesforLearningQuestionnaire(MSLQ)ResearchersfrommultipleinstitutionsdevelopedtheMSLQbasedonsocialcognitiveprinciples.AsGarciaandPintrich(1995)explained,theMSLQhastwomajorsections(motivationandlearningstrategies)withthemotivationalsectionsubdividedintothreecomponents:expectancy(includingtheself-efficacyscale),value(e.g.,extrinsicandintrinsicorientations),andaffect(encompassingtestanxiety).Becausebeliefsandstrategieslikelydifferacrosscourses,studentsrespondtoall81itemsintermsofaspecificcourseusinga7-pointLikertscalewithansweroptionsrangingfrom1(notatalltrueofme)to7(verytrueofme).Theeightitemsontheself-efficacyscale(α=.93,Pintrichetal.,1991)reflectexpectancyforsuccessandself-efficacy.Regardingpredictivevalidity,scoresonthisscalepositivelycorrelatedwithfinalgrade(r=.41);thecorrelationwasstrongerfortheself-efficacyscalethanforanyoftheotherMSLQmotivationorlearningstrategyscales.Self-efficacyscoresalsocorrelatedmorestronglywithintrinsicmotivation(r=.59)thanwithanyotherscale.KomarrajuandNadler(2013)reportedasignificantcorrelation(r=.50)betweenself-efficacyandeffortregulationandfoundthatstudentswithhigherself-efficacyweremorelikelythanthosewithlowerself-efficacytobelievethatintelligencecanchangeandtoadoptmasterygoals.

ConclusionAwell-developedassessmentplanencompassesbothclassroomandprogram-levelassessmentsoflearningbasedonspecificgoals.Forexample,themeasuresdiscussedinthischaptercoincidewithgoalsintheAPAGuidelines2.0(APA,2013)relatedtotheknowledgebaseofpsychology,communication,andprofessionaldevelopment.Bothformativeandsummativeassessmentsplayadistinctroleinpromotingandmeasuringstudentlearning;although,educatorsmaynotconsistentlyrecognizedistinctionsbetweenthetwo(Taras,2002)orrealizethatthesametypesofassessments(e.g.,rubrics,quizzes)maybeusedforbothformativeandsummativepurposes.Thetoolsdiscussedinthischaptertoevaluateactualclassroomlearning(e.g.,CATsandrubrics)andprogramassessment(e.g.,standardizedexaminations)maynotbeclassifiedastraditionalscales.However,consideringthevariabilityinlearninggoalsforbothcontentandskillsacrosscourses,itwouldbechallengingtodevelopscalesofactuallearningsuitableforuniversalapplicationatthecourselevel.Moreover,multiplepossibilitiesdoexistformeaningfulclassroomassessment(e.g.,examinations,presentations,problem-solvingassignments),which,ifconstructedandevaluatedappropriately,mayserveaseffectivemeasuresoflearning.Forprogramevaluation,theuseofsuchstandardizedassessmentscanprovideacommonmetricofperformance.Giventheseexistingassessmentsforevaluatingcourseandprogram-specificknowledgeandskills,itmaynotbenecessarytocreategeneralizable,traditionalscalesforthesepurposes.Ontheotherhand,multiplescalesareavailabletomeasurestudents’perceptionsoflearningandconstructsrelatedtolearning.AsRovaiandcolleagues(2009)noted,educationaloutcomesareinfluencedbynumerousvariables,includingfactorsrelatedtocoursedesignand

68

pedagogyaswellasstudents’characteristicsandbeliefs.Perceivedself-efficacyisonesuchbeliefdiscussedinthischapter;itcontributestoacademicsuccessviaboostingengagement,commitment,andpersistence(Bandura,1993).However,asmentionedinChapter14,SoTLScales:TheCaseoftheMissingLinks(Richmond,2015),tomeaningfullyinterpretself-reportedmeasuresofperceivedskill,theyshouldbecomparedtoobjectiveperformancemeasures.Instructors’awarenessoftheassociationbetweenperceivedself-efficacyandacademicperformanceandoffactorscontributingtorealisticself-efficacybeliefsmaycontributetotheultimategoalofstudentlearning.

69

ReferencesReferencesmarkedwithanasteriskindicateascale.AmericanPsychologicalAssociation.(2013).APAguidelinesfortheundergraduatepsychology

major:Version2.0.Retrievedfromhttp://www.apa.org/ed/precollege/undergrad/index.aspx

Anderson,D.,&Burns,S.(2013).One-minutepaper:Studentperceptionoflearninggains.CollegeStudentJournal,47,219–227.Retrievedfromhttp://www.projectinnovation.com/college-student-journal.html

Angelo,T.,&Cross,K.P.(1993).ClassroomAssessmentTechniques:AHandbookforCollegeTeachers.SanFrancisco,CA:Jossey-Bass.

Bandura,A.(1977).Self-efficacy:Towardaunifyingtheoryofbehavioralchange.PsychologicalReview,84,191–215.doi:10.1037/0033-295X.84.2.191

Bandura,A.(1993).Perceivedself-efficacyincognitivedevelopmentandfunctioning.EducationalPsychologist,28,117–148.doi:10.1207/s15326985ep2802_3

Bandura,A.,Barbaranelli,C.,Caprara,G.V.,&Pastorelli,C.(1996).Multifacetedimpactofself-efficacybeliefsonacademicfunctioning.ChildDevelopment,67,1206–1222,doi:10.1111/j.1467-8624.1996.tb01791.x

Barron,A.B.,Hebets,E.A.,Cleland,T.A.,Fitzpatrick,C.L.,Hauber,M.E.,&Stevens,J.R.(2015).Embracingmultipledefinitionsoflearning.TrendsinNeurosciences,38,405–407.doi:10.1016/j.tins.2015.04.008

Bol,L.,Hacker,D.J.,O’shea,P.,&Allen,D.(2005).Theinfluenceofovertpractice,achievementlevel,andexplanatorystyleoncalibrationaccuracyandperformance.TheJournalofExperimentalEducation,73,269–290.doi:10.3200/JEXE.73.4.269-290

Brennan,R.L.(1992).AnNCMEinstructionalmoduleongeneralizabilitytheory.EducationalMeasurement:IssuesandPractice,11(4),27–34.doi:10.1111/j.1745-3992.1992.tb00260.x

*Cacioppo,J.T.,Petty,R.E.,&Kao,C.F.(1984).TheefficientassessmentofNeedforCognition.JournalofPersonalityAssessment,48,306–307.doi:10.1207/s15327752jpa4803_13

Centra,J.A.,&Gaubatz,N.B.(2005).Studentperceptionsoflearningandinstructionaleffectivenessincollegecourses.EducationalTestingService.Retrievedfromhttp://www.ets.org/Media/Products/perceptions.pdf

Chemers,M.,Hu,L.,&Garcia,B.(2001).Academicself-efficacyandfirstyearcollegestudentperformanceandadjustment.JournalofEducationalPsychology,93,55–64.doi:10.1037/0022-0663.93.1.55

Chen,G.,Gully,S.M.,&Eden,D.(2001).Validationofanewgeneralself-efficacyscale.OrganizationalResearchMethods,4,62–83.doi:10.1177/109442810141004

Chesebro,J.,L.,&McCroskey,J.C.(2000).Therelationshipbetweenstudents’reportsoflearningandtheiractualrecalloflecturematerial:Avaliditytest.CommunicationEducation,49,297–301.doi:10.1080/03634520009379217

Chiou,C.-C.,Wang,Y.-M.,&Lee,L.-T.(2014).Reducingstatisticsanxietyandenhancingstatisticslearningachievement:Effectivenessofaone-minutestrategy.PsychologicalReports:SocioculturalIssuesinPsychology,115,297–310.doi:10.2466/11.04.PRO.115c12z3

70

Cottell,P.,&Harwood,E.(1998).Doclassroomassessmenttechniques(CATs)improvestudentlearning?NewDirectionsforTeachingandLearning,75,37–46.doi:10.1002/tl.7504

Crick,J.E.,&Brennan,R.L.(1984).Generalpurposeanalysisofvariancesystem2.2.IowaCity,IA.AmericanCollegeTestingProgram.

Derogatis,L.R.,&Melisaratos,N.(1983).TheBriefSymptomInventory:Anintroductoryreport.PsychologicalMedicine,13,595–605.doi:10.1017/S0033291700048017

DiBenedetto,M.K.,&Bembenutty,H.(2013).Withinthepipeline:Self-regulatedlearning,self-efficacy,andsocializationamongcollegestudentsinsciencecourses.LearningandIndividualDifferences,23,218–224.doi:10.1016/j.lindif.2012.09.015

Dolinsky,B.,&Kelley,J.M.(2010).Forbetterorforworse:Usinganobjectiveprogramassessmentmeasuretoenhanceanundergraduatepsychologyprogram.TeachingofPsychology,37,252–256.doi:10.1080/00986283.2010.510978

Dunn,K.E.,&Mulvenon,S.W.(2009).Acriticalreviewofresearchonformativeassessment:Thelimitedscientificevidenceoftheimpactofformativeassessmentineducation.PracticalAssessment,Research,&Evaluation,14,1–11.Retrievedfromhttp://pareonline.net/getvn.asp?v=14&n=7

*Elias,S.M.,&Loomis,R.J.(2002).Utilizingneedforcognitionandperceivedself-efficacytopredictacademicperformance.JournalofAppliedSocialPsychology,32,1687–1702.doi:10.1111/j.1559-1816.2002.tb02770.x

Falchikov,N.,&Boud,D.(1989).Studentself-assessmentinhighereducation:Ameta-analysis.ReviewofEducationalResearch,59,395–430.doi:10.3102/00346543059004395

Frymier,A.B.,&Houser,M.L.(1999).Therevisedlearningindicatorsscale.CommunicationStudies,50(1),1–12.doi:10.1080/10510979909388466

Frymier,A.B.,Shulman,G.M.,&Houser,M.(1996).Thedevelopmentofalearnerempowermentmeasure.CommunicationEducation,45,181–199.doi:10.1080/03634529609379048

Gallagher,S.P.,&Cook,S.P.(2013).ThevalidityoftheMajorFieldTestinpsychologyasaprogrammeassessmenttool.PsychologyTeachingReview,19(2),59–72.Retrievedfromhttp://www.millersville.edu/~sgallagh/Publications/ptr2013.pdf

Galyon,C.E.,Blondin,C.A.,Yaw,J.S.,Nalls,M.L.,&Williams,R.L.(2012).Therelationshipofacademicself-efficacytoclassparticipationandexamperformance.SocialPsychologyofEducation,15,233–249.doi:10.1007/s11218-011-9175-x

Garcia,T.,&Pintrich,P.R.(1995).Assessingstudents'motivationandlearningstrategies:TheMotivatedStrategiesforLearningQuestionnaire.PaperpresentedattheannualmeetingoftheAmericanEducationalResearchAssociation,SanFrancisco,CA.RetrievedfromERICdatabase.(ED383770)

Gore,P.J.(2006).Academicself-efficacyasapredictorofcollegeoutcomes:Twoincrementalvaliditystudies.JournalofCareerAssessment,14,92–115.Retrievedfromhttp://jca.sagepub.com

*Greenberg,K.P.(2015).Rubricuseinformativeassessment:Adetailedbehavioralrubrichelpsstudentsimprovetheirscientificwritingskills.TeachingofPsychology,42,211–217.doi:10.1177/009868315587618

71

Hacker,D.J.,Bol,L.,&Bahbabani,K.(2008).Explainingcalibrationaccuracyinclassroomcontexts:Theeffectsofincentives,reflection,andexplanatorystyle.MetacognitionLearning,3,101–121.doi:10.1007/s11409-008-9021-5

Høigaard,R.,Kovač,V.B.,Øverby,N.C.,&Haugen,T.(2015).Academicself-efficacymediatestheeffectsofschoolpsychologicalclimateonacademicachievement.SchoolPsychologyQuarterly,30,64–74.doi:10.1037/spq0000056

Komarraju,M.,&Nadler,D.(2013).Self-efficacyandacademicachievement:Whydoimplicitbeliefs,goals,andeffortregulationmatter?LearningandIndividualDifferences,25,67–72.doi:10.1016/j.lindif.2013.01.005

Kruger,J.,&Dunning,D.(1999).Unskilledandunawareofit:Howdifficultiesinrecognizingone'sownincompetenceleadtoinflatedself-assessments.JournalofPersonalityandSocialPsychology,77,1121–1134.doi:10.1037/0022-3514.77.6.1121

*Lipnevich,A.A.,McCallen,L.N.,Miles,K.P.,&Smith,J.K.(2014).Mindthegap!Students’useofexemplarsanddetailedrubricsasformativeassessment.InstructionalScience,42,539–559.doi:10.1007/s11251-013-9299-9

Lom,B.(2012).Classroomactivities:Simplestrategiestoincorporatestudent-centeredactivitieswithinundergraduatesciencelectures.TheJournalofUndergraduateNeuroscienceEducation,11,A64–A71.Retrievedfromhttp://www.funjournal.org/

Marzano,R.J.(2002).Acomparisonofselectedmethodsofscoringclassroomassessments.AppliedMeasurementinEducation,15,249–268.doi:10.1207/S15324818AME1503.2

McCroskey,J.C.,Sallinen,A.,Fayer,J.M.,Richmond,V.P.,&Barraclough,R.A.(1996).Nonverbalimmediacyandcognitivelearning:Across-culturalinvestigation.CommunicationEducation,45,200–211.doi:10.1080/03634529609379049

Montgomery,K.(2002).Authentictasksandrubrics:Goingbeyondtraditionalassessmentsincollegeteaching.CollegeTeaching,50(1),34–39.doi:10.1080/87567550209595870

Mueller,J.(2014)Rubrics.Availableathttp://jfmueller.faculty.noctrl.edu/toolbox/rubrics.htmMulton,K.D.,Brown,S.D.,&Lent,R.W.(1991).Relationofself-efficacybeliefstoacademic

outcomes:Ameta-analyticinvestigation.JournalofCounselingPsychology,38,30–38.doi:10.1037/0022-0167.38.1.30

NationalAssociationofCollegesandEmployers(2015).Joboutlook:Thecandidateskills/qualitiesemployerswant,theinfluenceofattributes.Retrievedfromhttps://www.naceweb.org/s11122014/job-outlook-skills-qualities-employers-want.aspx

Pinter,B.,Matchock,R.L.,Charles,E.P.,&Balch,W.R.(2014).Across-sectionalevaluationofstudentachievementusingstandardizedandperformance-basedtests.TeachingofPsychology,41,20–17.doi:10.1177/0098628313514174

*Pintrich,P.R.,&DeGroot,E.V.(1990).Motivationalandself-regulatedlearningcomponentsofclassroomacademicperformance.JournalofEducationalPsychology,82,33–40.doi:10.1037/0022-0663.82.1.33

*Pintrich,P.R.,Smith,D.A.F.,Garcia,T.,&McKeachie,W.J.(1991).AmanualfortheuseoftheMotivatedStrategiesforLearningQuestionnaire(MSLQ).Technicalreport91-B-004.RetrievedfromERICdatabase.(ED338122)

Pintrich,P.R.,Smith,D.A.F.,Garcia,T.,&McKeachie,W.J.(1993).ReliabilityandpredictivevalidityoftheMotivatedStrategiesforLearningQuestionnaire.EducationalandPsychologicalMeasurement,53,801–813,doi:10.1177/0013164493053003024.

72

Putwain,D.,Sander,P.,&Larkin,D.(2013).Academicself-efficacyinstudy-relatedskillsandbehaviours:Relationswithlearning-relatedemotionsandacademicsuccess.BritishJournalofEducationalPsychology,83,633–650.doi:10.1111/j.2044-8279.2012.02084.x

Reddy,M.Y.,&Andrade,H.(2010).Areviewofrubricuseinhighereducation.Assessment&EvaluationinHigherEducation,35,435–448.doi:10.1080/02602930902862859

*Reynolds,J.,Smith,R.,Moskovitz,C.,&Sayle,A.(2009).BioTAP:Asystematicapproachtoteachingscientificwritingandevaluatingundergraduatetheses.BioScience,59,896–903.doi:10.1025/bio.2009.59.10.11

Richmond,V.P.,Gorham,J.S.,&McCroskey,J.C.(1987).Therelationshipbetweenselectedimmediacybehaviorsandcognitivelearning.InM.McLaughlin(Ed.),CommunicationYearbook10,(pp.574-590).BeverlyHills,CA:Sage.

Rovai,A.P.,Wighting,M.J.,Baker,J.D.,&Grooms,L.D.(2009).Developmentofaninstrumenttomeasureperceivedcognitive,affective,andpsychomotorlearningintraditionalandvirtualclassroomhighereducationsettings.InternetandHigherEducation,12,7–13.doi:10.1016/j.iheduc.2008.10.002

*Sander,P.,&Sanders,L.(2009).Measuringacademicbehaviouralconfidence:TheABCScalerevisited.StudiesinHigherEducation,34,19–35.doi:10.1080/03075070802457058

Schraw,G.(2009).Aconceptualanalysisoffivemeasuresofmetacognitivemonitoring.MetacognitionandLearning,4,33–45.doi:10.1007/s11409-008-9031-3

*Schraw,G.,&Dennison,R.S.(1994).Assessingmetacognitiveawareness.ContemporaryEducationalPsychology,19,460–475.doi:10.1006/ceps.1994.1033

Short,F.,&Martin,J.(2012).Whowantstobeapsychologygraduate?Impactofformativemultiple-choicereviewquestionsonsummativeassessmentperformance.PsychologyLearning&Teaching,11,218–227.doi:10.2304/plat.2012.11.2.218

Simpson-Beck,V.(2011).Assessingclassroomassessmenttechniques.ActiveLearninginHigherEducation,12(2),125–132.doi:10.1177/1469787411402482

Sitzmann,T.,Ely,K.,Brown,K.G.,&Bauer,K.(2010).Self-assessmentofknowledge:Acognitivelearningoraffectivemeasure?AcademyofManagementLearning&Education,9(2),169–191.doi:10.5465/AMLE.2010.51428542

Soetaert,E.(1998).Qualityintheclassroom:ClassroomassessmenttechniquesasTQM.NewDirectionsforTeachingandLearning,75,47–55.doi:10.1002/tl.7505

*Solberg,V.S.,O'Brien,K.,Villareal,P.,Kennel,R.,&Davis,B.(1993).Self-efficacyandHispaniccollegestudents:ValidationoftheCollegeSelf-EfficacyInstrument.HispanicJournalofBehavioralSciences,15(1),80–95.doi:10.1177/07399863930151004

Stead,D.R.(2005).Areviewoftheone-minutepaper.ActiveLearninginHigherEducation,6,118–131.doi:10.1177/1469787405054237

*Stellmack,M.A.,Kohneim-Kalkstein,Y.L.,Manor,J.E.,Massey,A.R.,&Schmitz,A.P.(2009).AnassessmentofreliabilityandvalidityofarubricforgradingAPA-styleintroductions.TeachingofPsychology,36,102–107.doi:10.1080/00986280902739776

Stemler,S.E.(2004).Acomparisonofconsensus,consistency,andmeasurementapproachestoestimatinginterraterreliability.PracticalAssessment,ResearchandEvaluation,9(4).Retrievedfromhttp://pareonline.net

73

Stoloff,M.L.,&Feeney,K.J.(2002).TheMajorFieldTestasanassessmenttoolforanundergraduatepsychologyprogram.TeachingofPsychology,29,92–98.doi:10.1207/515328023TOP2902_01

Stoloff,M.L.,Good,M.R.,Smith,K.L.,&Brewster,J.(2015).Characteristicsofprogramsthatmaximizepsychologymajorsuccess.TeachingofPsychology,42,99–108.doi:10.1177/0098628315569877

*Timmerman,B.E.C.,Strickland,D.C.,Johnson,R.L.,&Payne,J.R.(2011).Developmentofa‘universal’rubricforassessingundergraduates’scientificreasoningskillsusingscientificwriting.AssessmentandEvaluationinHigherEducation,36,509–547.doi:10.1080.02602930903540991

Taras,M.(2008).Summativeandformativeassessment:Perceptionsandrealities.ActiveLearninginHigherEducation,9,172–192.doi:10.1177/1469787408091655

Wiliam,D.,&Black,P.(1996).Meaningsandconsequences:Abasisfordistinguishingformativeandsummativefunctionsofassessment?BritishEducationalResearchJournal,22,537–548.doi:10.1080/0141192960220502

*Zajacova,A.,Lynch,S.M.,&Espenshade,T.J.(2005).Self-efficacy,stress,andacademicsuccessincollege.ResearchinHigherEducation,6,677–706.doi:10.1007/s11162-004-4139-z

Zimmerman,B.J.(2000).Self-efficacy:Anessentialmotivetolearn.ContemporaryEducationalPsychology,25,82–91.doi:10.1006/ceps.1999.1016

*Zimmerman,B.J.,&Martinez-Pons,M.(1988).Constructvalidationofastrategymodelofstudentself-regulatedlearning.JournalofEducationalPsychology,80,284–290.doi:10.1037/0022-0663.80.3.284

74

Chapter7:MeasuringCriticalThinkingSkillsR.EricLandrum1andMaureenA.McCarthy21BoiseStateUniversity, 2KennesawStateUniversity

Docriticalthinkingskillsexist--andcantheybemeasured?Clearlyarticulatingtheconstructofcriticalthinkingiscentraltomeasurementyetarticulatingacleardefinitionofcriticalthinkingremainselusive.Infact,Halpern(1999)acknowledgedthedifficultyofdefiningtheconstructandsheofferedawiderangeofpossibleinterrelateddefinitions.Shealsoreflectedonsimilaritiesoftheconstructacrossdisciplinesincluding:problemsolving,decisionmaking,andcognitiveprocesses(alsoseeHalpern,1996foracomprehensivereviewoftheconstructofcriticalthinking).Despitethecomplexityofdefiningtheconstruct,webelievethatitisbothpossibleandimportanttomeasurecriticalthinking,particularlyduringthiseraofincreaseddemandforaccountability.Criticalthinkingremainsoneofthemostimportantskillsidentifiedasanoutcomeofacollegedegree.Notonlyarecriticalthinkingskillsdesiredincollegegraduates,butthisskillsetisbeneficialtoaneducatedcitizenry.Inadditiontostudents,manyconstituencieshaveakeeninterestincollegegraduatesdemonstratingcriticalthinkingskills,includingeducators(Appleby,2009;Keeling&Hersh,2012;Yanchar,Slife,&Warne,2008),highereducationassociations(AmericanAssociationofColleges&Universities[AAC&U],2006;2010),employers(AAC&U,2008),andthegeneralpublic(AAC&U,2005;Baum&Ma,2007).Morerecently,theAmericanPsychologicalAssociation(APA)reaffirmedtheimportanceofcriticalthinkingskillsintherevisionofdisciplinespecificguidelinesfortheundergraduatemajor(APA,2013).Moregenerally,thisemphasisoncriticalthinkingasanimportantoutcomeofacollegedegreewasemphasizedwiththepublicationofAcademicallyAdriftbyArumandRoksa(2011a).IntheirresearchusingtheCollegiateLearningAssessment(CLA),theyfoundthatalargepercentageofstudentsinbothtwo-yearandfour-yearinstitutionsdidnotdemonstrateprogressincriticalthinkingskillsattheendoftheiracademicstudies.AlthoughtheeffortsofArumandRoksa(2011b)havelimitationswithregardtomethodologyandthemotivationofCLAtest-takers,thevalueoftheprocessisclear;meaningfulassessmentcanprovideinvaluablefeedbacktoeducators,administrators,andtothehighereducationcommunity.BroadPerspectivesAboutCriticalThinkingScholarshavewrittenextensivelyaboutcriticalthinking(Halpern,1996;Halpern,2010)asanimportantskill;however,acomprehensivereviewandanalysisoftheconstructexceedthescopeofthischapter.Somehavesuggestedthatcriticalthinkingisdevelopedasadisciplinespecificskill(Davies,2013;McGovern,Furumoto,Halpern,Kimble,McKeachie,1991),whereasothershavesuggestedthatcriticalthinkingisdevelopedbroadlyacrossmanycourses.Criticalthinkingcanbedescribedastheactofprocessing,evaluating,andcreatingnewinformationratherthanmerelyrecallinginformation(Butler,2012;Halpern,2010).Infact,DunnandSmith(2008)madetheargumentthatwritingisaformofcriticalthinking(seealsoPreiss,Castillo,Flotts,&SanMartin,2013)andHalpern(1987)suggestedthatthegenerationand

75

interpretationofanalogiesisanactivitythatclearlydemonstratescriticalthinking.SeeTable1foradditionaldefinitionsofcriticalthinkingdefinitions.

Table1ExamplesofCriticalThinkingDefinitions“Theconsciousprocessapersondoeswhenheorsheexploresasituationoraproblemfrom

differentperspectives”(French,Hand,Nam,Yen,&Vazquez,2014,p.275).“Challengingaclaimoranopinion(eitherone’sownoranotherperson’s)withthepurposeof

findingoutwhattobelieveordo”(O’Hare&McGuinness,2009,p.123).“Reasonableandreflectivethinkingthatisfocusedondecidingwhattobelievetodo”(Norris&

Ennis,1989,p.1).“Theuseofthosecognitiveskillsorstrategiesthatincreasetheprobabilityofadesirable

outcome.Itisusedtodescribethinkingthatispurposeful,reasoned,andgoal-directed—thekindofthinkinginvolvedinsolvingproblems,formulatinginferences,calculatinglikelihoods,andmakingdecisions,whenthethinkerisusingskillsthatarethoughtfulandeffectivefortheparticularcontextandtypeofthinkingtask”(Halpern,2003,p.6,ascitedinButler,2012).

Asecondterm—psychologicalliteracy—hasalsobeenusedinterchangeablywithhistoricaloriginsdatingtotheSt.Mary’sconferencein1991(McGovern,Furumoto,Halpern,Kimble,McKeachie,1991).Withthere-emergenceofpsychologicalliteracy(McGovernetal.,2010)emphasizedasanimportantoutcomeforthemajor,criticalthinkingcontinuestobeacentraltopicinthediscussionsofpsychologyeducators.Infact,manycomponentsofcriticalthinkingarecontainedinthedefinitionofpsychologicalliteracy:

(a)havingawell-definedvocabularyandbasicknowledgeofthecriticalsubjectmatterofpsychology;(b)valuingtheintellectualchallengesrequiredtousescientificthinkingandthedisciplinedanalysisofinformationtoevaluatealternativecoursesofaction;(c)takingacreativeandamiableskepticapproachtoproblemsolving;(d)applyingpsychologicalprinciplestopersonal,social,andorganizationalissuesinwork,relationships,andthebroadercommunity;(e)actingethically;(f)beingcompetentinusingandevaluatinginformationandtechnology;(g)communicatingeffectivelyindifferentmodesandwithmanydifferentaudiences;(h)recognizing,understanding,andfosteringrespectfordiversity;and(i)beinginsightfulandreflectiveaboutone’sownandothers’behaviorandmentalprocesses.(McGovern,etal.,2010,p.11)

Thisconceptualizationdovetailsnicelywithrecentnationaleffortsdevotedtovalidatingundergraduateeducationinpsychologyasaliberalartsdegreethataffordsstudentsopportunitiestothinkcriticallyacrossmultiplecareeropportunities.TheAmericanPsychologicalAssociationrevisedtheAPAGuidelinesfortheUndergraduatePsychologyMajorin2013,referredtoasGuidelines2.0,whichcontinuedtoemphasize

76

complexthinkingasanimportantoutcomeofthemajorinpsychology.Infact,Goal2oftheGuidelinesincludesfivespecificstudentoutcomes:

• usescientificreasoningtointerpretpsychologicalphenomena• demonstratepsychologyinformationliteracy• engageininnovativeandintegrativethinkingandproblemsolving• interpret,design,andconductbasicpsychologicalresearch• incorporatesocioculturalfactorsinscientificinquiry

Ifwecomparetheseoutcomestothedefinitionsofcriticalthinkingabove,itseemsapparentthatthereisoverlapbetweenthedefinitionsofpsychologicalliteracyandcriticalthinking.

MeasuresofCriticalThinkingOurreviewofcriticalthinkingmeasuresistwofold:First,wewanttomakesurethatthemainstreammeasuresofcriticalthinkingarereviewed(albeitbriefly)inthischapter.Wereviewcriticalthinkingmeasuresthatarespecifictopsychologyaswellasbroad-basedgeneralmeasures.Second,ourreviewisnottobeinterpretedascomprehensive.Insteadwewanttoshareinformationaboutthemostcommonmeasuresofcriticalthinking.Ifthereaderdesiresadditionaldetailsaboutthemeasures,wehaveincludedanappendixwithreferencesforadditionalinformation.

GeneralMeasuresForeachofthegeneralmeasures,weprovide“quicksnippets”abouthowthemeasurehasbeenusedinpublishedresearch;thisismeanttoprovideasamplingofthecurrenteffortsandisnotmeanttobecomprehensive.Forexample,theWatson-GlaserCriticalThinkingAppraisaltestisoftencitedasoneofthemostfrequentlyusedgeneralmeasuresofcriticalthinking.MorerecentlyBurke,Sears,Kraus,andRoberts-Cady(2014)usedtheWatson-GlaserCriticalThinkingAppraisal(WGCTA;Watson&Glaser,1980)inabetween-groupscomparisonofcriticalthinkingscoresacrossdifferentdisciplines.TheyfoundthatstudentsinaphilosophycourseimprovedtheircriticalthinkingwhenmeasuredbytheWGCTA.However,thissameimprovementwasnotfoundinthepsychologycoursespecificallydesignedtoimprovecriticalthinkingskills.Thesefindingsmaybeareflectionofdifferencesincourses,orquitepossiblythedifficultyingenerallymeasuringtheconstructofcriticalthinking.MacphersonandOwen(2010)alsousedtheWGCTAinatest-reteststudytoexaminedevelopmentofcriticalthinkingbetweentwocohorts.TheyexperienceddifficultyinusingthetesttodetectdifferencesincriticalthinkingthatwerenotalreadyexplainedwiththesubtestsoftheWGCTA.Thesefindingsmayreflectthecomplicatednatureoftheconstruct.Further,whenMagno(2010)examinedtheroleofmetacognitionincriticalthinking,heusedastructuralequationmodeltolinkmetacognitiontotheWGCTA.TheconstructisfurthercomplicatedbyfindingsfromClifford,Boufal,andKurtz(2004).UsingtheWGCTA,theyfoundthatcriticalthinkingskillswererelatedtopersonalitycharacteristics,inparticulartoopennesstoexperience.Thustheconstructofcriticalthinking,andthegeneralmeasuresofcriticalthinking,makeitdifficulttoaccuratelymeasuretheimportantskill.

77

Similardifficultiesinaccuratelymeasuringcriticalthinkingarepresentacrossothermeasures.Forexample,theCornellCriticalThinkingTest(CCTT)hasbeenusedinavarietyofresearchstudies.Recently,Stark(2012)comparedtheCCTTtoapsychologyspecifictestofcriticalthinkingandfoundincreasesinthepsychologyspecifictest,butthattheseincreaseswerenotreflectedinthemoregeneralmeasureusingtheCCCT.O’HareandMcGuiness(2009)administeredasubsetoftestsfromtheCaliforniaCriticalThinkingSkillsTest(CCTST)(Facione,Facione,Blohm,Howard,&Giancarlo,1998)andRaven’sAdvancedProgressiveMatrices(Set1;Raven,1965)topsychologyundergraduatesatQueen’sUniversityinBelfast.Usingthesemeasures,theyfoundthatreasoningskillsimprovedasstudentsprogressedfromthefirsttothirdyearincollege.TheCCTSTwasalsoutilizedbyFeroandcolleagues(2010)inacomparisonofasmallnumberofnursingstudents’criticalthinkinglevelstoperformanceonsimulatedclinicalsituationsinnursing.However,theydidnotfindacorrelationbetweencriticalskills-basedperformanceandperformanceontheCCTST.ForamoreoverarchingperspectiveaboutthechallengesfacingresearchersusingtheWGCTAandtheCCTST,seeSchrawandGutierrez(2012).TheHalpernCriticalThinkingAssessment(HCTA;Halpern,2010)isuniqueinthatitreliesbothonrecognitionmemory(suchascompletingmultiplechoiceitems)aswellasrecallmemory(providinganswerstoshortessays).AnotherimportantcontributionthatresearchershavemadewiththeHCTAisthatthesecriticalthinkingscoreshavebeencomparedwithreal-worldoutcomes,suchasasignificantnegativecorrelationbetweenHCTAscoresandnegativelifeevents(Butler,2012;Butleretal.,2012).TheEnnis-WeirTestofCriticalThinking(EWTCT;Ennis&Weir,1985)isafree-responseinstrumentwhichrequiresawrittenargumentinresponsetoastimulus.TheEWTCTwasusedbySzaboandSchwartz(2011)toexaminepotentialpre-semestertopost-semestergrowthincriticalthinkingscoresusingonlinediscussiontoolsinaface-to-facecourse.Usingthisinstrument,theyconcludedthattheonlinesupplementalinstructionimprovedthecriticalreasoningofthepre-serviceteachersparticipatinginthestudy.Pascarellaandcolleagues(2014)assessedcriticalthinkingincollegestudentsusingtheCriticalThinkingTest(CTT;AmericanCollegeTestingProgram,1990)toexaminehowdiversityexperiencesmayaffectcriticalthinkingattheconclusionofthecollegeexperience.Theyconcludethatexposuretodiversityincreasescriticalthinkinginthestudentswhoparticipatedinthestudy.

Psychology-SpecificMeasuresThePsychologicalCriticalThinkingExam(PCTE)developedbyLawson(1999)wasutilizedbyMcLeanandMiller(2010)asabetweengroupsmeasuretodemonstratecriticalthinkingdifferencesbetweencourses.ThismeasurealsoprovedusefulforHaw(2011)whenheadministeredthePCTEtostudentsintheirsecondandfourthyearsofinstructiontocompareadvancesincriticalthinking.UsingthePCTE,heconcludedthatpsychology-specificcriticalthinkingskillsdoimprovewithadditionalinstruction.Lawson,Jordan-Fleming,andBodle(2015)

78

recentlypublishedanupdatetothePCTE.Similarly,Muehlenkamp,Weiss,andHansen(2015)testedtheefficacyofproblem-basedlearninginstructionaltechniques,andusedscoresonthePCTEaspre-andpost-outcomemeasures,demonstratingthatstudentsintheproblem-basedlearningconditionexhibitedhighercriticalthinkingscoresattheendofthesemester.Asecondpsychologyspecificcriticalthinkingtesthasalsobeenusedinanumberofstudies.TheCriticalThinkinginPsychologyTest,developedbyBensleyandBaxter(2006),containsanargumentanalysistest,amethodologicalreasoningtest,andacausalreasoningtest;however,thistestisunpublishedandisnotwidelyavailable.Ithas,however,beenusedinmultipleresearchcontexts,suchasaninstrumentusedtomeasuregainsafterspecificcriticalthinkinginstruction(Bensley,Crowe,Bernhardt,Buckner,&Allman,2010)inspecificresearchstudies.TeachingCriticalThinkingSkillsDespitethedifficultieswithdefiningtheconstructandmeasuringcriticalthinking,researcherscontinuetorecommendteachingtheseskills.Morespecifically,severalresearchers(Frantz&McCarthy,inpress;Lilienfeld,Lohr,&Olatunji,2008;Wesp&Montgomery,1998)haverecommendedthatpsychologycoursesofferopportunitiesforhelpingstudentsdeveloptheseskillsbyquestioningcommonmythsaboutpsychology.Forexample,WespandMontgomery(1998)wereabletodemonstrateanincreaseincriticalthinkingaftertakingacoursedesignedtodecreasebeliefsaboutparanormalactivities.Similarly,Lilienfeldandcolleagues(2008)designedacoursetohelpstudentstothinkcriticallyaboutpsychotherapyeffectiveness;inotherwords,whetherthetreatmenthelpsmorethandoingnothingorwhethertheoutcomeisduetotheplaceboeffect.Theywereabletodemonstrateimprovementinthecriticalthinkingskillsofthestudentsenrolledinthecourse.Inadditiontoresearchstudiessupportingtheuseofteachingcriticalthinkingasaprimaryobjectiveofpsychologycourses,twokeytextstoaidindesigningcoursesincludeTheCriticalThinkingCompanionforIntroductoryPsychology(Halonen,1995)andThinkingCriticallyaboutCriticalThinking(Halpern,1996).Botharefilledwithideasforhands-onexercisesforengagingstudentsintaskswhichmayhelptosupportthedevelopmentofcriticalthinkingskills.Moreimportantly,withtheavailabilityofthesedevelopingmeasures,psychologyeducatorsdonotneedtoguessabouttheeffectivenessoftheseexercises.Utilizingthetechniquesavailablefromthescholarshipofteachingandlearning(SoTL)literature,scholarscanmeasureanddocumenttheeffectivenessofplannedinterventionstoenhancecriticalthinking.RecommendationsDespitetheimportanceofteachingcriticalthinkingandtheattemptstomeasurethisconstruct,theconstructremainsdifficulttomeasureefficiently.Ku(2009)identifiedseveralkeypointstoconsider,includingwhetheranobjectivemultiple-choiceformatcanbeusedtoaccuratelymeasurecriticalthinking.Kualsoindicatedthatitisdifficulttomeasurehigherlevelsofcomplexreasoningusingamultiple-choiceformat.Althoughmultiplechoicetestingiscertainlyanefficientmethodofmeasurement,itmaybedifficulttoconvinceresearchersthatamultiple-choiceformatprovidesanaccurateandcompletemeasureofcriticalthinking.

79

Howdowebalancetheneedforefficientmeasurementagainstthecomplexityoftheconstruct?OnesolutionistoadaptHalpern’s(2013)generalrecommendationsformeasuringstudentlearningingeneral.Specifically,measuringstudentlearningshouldincludethefollowingelements(adaptedforcriticalthinking):

1. Multiple,variedmeasuresforcriticalthinkingarenecessarybecausenosinglemeasurecancaptureitscomplexity.

2. Facultyinvolvementinallaspectsofthemeasurementofcriticalthinkingandtheutilizationofcriticalthinkingoutcomesisessentialforsuccess.

3. Departmentsshouldberewardedforconductingmeaningfulassessmentsofcriticalthinkingskills,evenwhentheoutcomesofthatassessmentdemonstrateroomforimprovement.

4. Facultymembersandinstitutionsshouldusetheoutcomesofcriticalthinkingassessmentstoimprovetheirteachingandtheirstudents’learning,whetherthatinvolvescurriculumchanges,individualfacultychangingpedagogicalapproachesifneeded,andsoon.

5. Departmentsshouldtakeavalue-addedapproachtothemeasurementofcriticalthinkingscoresovertime;thatis,strivetounderstandthecriticalthinkinggrowthwithineachstudentratherthanacomparisonofdifferentgroupsofstudents.Usingthisapproach,allstudentscandemonstrateenhancedcriticalthinkingskillsovertime.

6. Seektoutilizemultiplesourcesofinformationaboutcriticalthinkingfromdifferingperspectives;byidentifyingoverlappingefforts,aconvergenceofeffortsthroughpurposefulcoordinationmayleadtorichersourcesofdataaswellasmorecompleteandrepresentativeoutcomes.

Althoughaneducatormighthavesomeindicationaboutthecriticalthinkingskillsthataredevelopedduringacourse,amorethoroughunderstandingisneeded.Forinstance,therearepre-coursetopost-coursestudieswhereresearchersexaminedwhethercriticalthinkingchangedmeasurablyoverthesemester,withmixedresults(e.g.,Stark,2012).However,webelievethatmoreresearchisneededregardingcriticalthinkingskillsatcommencement,andhowthoseskillsrelatetosuccessafterthebachelor’sdegree.Infact,usingtheHalpernCriticalThinkingAssessment(Butler2012;Butler,etal.,2012),researchershavereportedpromisingoutcomesrelatingcriticalthinkingmeasurestoreal-worldoutcomes.PerhapsthemostintegrativemeasuresofcriticalthinkingarereportedintheassessmentplanofJamesMadisonUniversity(Apple,Serdikoff,Reis-Bergan,&Barron,2008).Multipleassessmentsofcriticalthinkingoccurnotonlyacrosscoursesbutalsoattheconclusionofthepsychologymajor’sundergraduatecareer.Psychologydepartmentsshouldemploytheavailablecriticalthinkingmeasuresmoreoften,andcoordinatedresearcheffortsonanationalscopeareneededtomaximizetheutilityofsuchmeasuresininstitution-specificdomains.ThemodelprovidedbyAppleandcolleagues(2008)isaverygoodstartingpointformanydepartmentstoconsider.

80

Afullyimplementedmulti-modalmulti-methodapproachincludesembeddedassessment,nationallystandardizedtests,cross-sectionalandlongitudinalstudies,andthecreationofanationaldatabaseoftestresultsthatmaybeusefulforprogramreviewpurposesaswellastheidentificationofbestpractices.

Withoutinformationaboutlearning,thereislesslearning.Facultyculturesandincentiveregimesthatsystematicallydevalueteachinginfavorofresearchareallowedtopersistbecausethereisnobasisforfixingthemandnoirrefutableevidenceofhowmuchstudentsarebeingshortchanged.(Carey,2010,p.A72)

Inouropinion,animportantcomponentofassessmentisusingtheinformationtoinformandreviseeducationalpractice.Theultimategoaloftestingisthepredictionofnon-testbehavior.Theultimategoalofanundergraduateeducationinpsychologyistoimpactbehaviors,attitudes,andopinionsofourstudentsfollowinggraduationsothattheycancreaterealchangeintheworld,whetherthatbethroughtheirownbehaviororthroughtheinfluenceofothers.Theabilitytothinkcriticallyisakeyskillinreachingthesegoals.

81

ReferencesReferencesmarkedwithanasteriskindicateascale.AmericanCollegeTestingProgram.(1990).ReportonthetechnicalcharacteristicsofCAAP:

Pilotyear1:1988-89.IowaCity,IA:Author.AmericanPsychologicalAssociation.(2013).APAguidelinesfortheundergraduatepsychology

major:Version2.0.Retrievedfromhttp://www.apa.org/precollege/undergrad/index.aspx

Apple,K.J.,Serdikoff,S.L.,Reis-Bergan,M.J.,&Barron,K.E.(2008).Programmaticassessmentofcriticalthinking.InD.S.Dunn,J.S.Halonen,&R.A.Smith(Eds.),Teachingcriticalthinkinginpsychology(pp.77-88).Malden,MA:BlackwellPublishing.

Appleby,D.C.(2009).Essentialskillsforpsychologymajors:Thosewesaytheycandevelopandthoseemployerssaytheyvalueinjobapplicants.PaperpresentedattheannualmeetingoftheAmericanPsychologicalAssociation,Toronto,Canada.

Arum,R.,&Roksa,J.(2011a,January28).Areundergraduatesactuallylearninganything?ChronicleofHigherEducation,57(21),A30-A31.

Arum,R.,&Roksa,J.(2011b).Academicallyadrift:Limitedlearningoncollegecampuses.Chicago,IL:UniversityofChicagoPress.

AssociationofAmericanCollegesandUniversities.(2005).Liberaleducationoutcomes:Apreliminaryreportonstudentachievementincollege.Washington,DC:Author.

AssociationofAmericanCollegesandUniversities.(2006).Howshouldcollegespreparestudentstosucceedintoday’sglobaleconomy?Washington,DC:HartResearchAssociates.

AssociationofAmericanCollegesandUniversities.(2008).Howshouldcollegesassessandimprovestudentlearning?Washington,DC:HartResearchAssociates.

AssociationofAmericanCollegesandUniversities.(2010).Raisingthebar:Employers’viewsoncollegelearninginthewakeoftheeconomicdownturn.Washington,DC:HartResearchAssociates.

Baum,S.,&Ma,J.(2007).Educationpays:Thebenefitsofhighereducationforindividualsandsociety.Washington,DC:CollegeBoard.

Bensley,D.A.,&Baxter,C.(2006).TheCriticalThinkinginPsychologyTest.Unpublishedmanuscript,FrostburgStateUniversity,Frostburg,MD.

Bensley,D.A.,Crowe,D.S.,Bernhardt,P.,Buckner,C.,&Allman,A.L.(2010).Teachingandassessingcriticalthinkingskillsforargumentanalysisinpsychology.TeachingofPsychology,37,91-96.doi:10.1080/0098628100326656

Burke,B.L.,Sears,S.R.,Kraus,S.,&Roberts-Cady,S.(2014).Criticalanalysis:Acomparisonofcriticalthinkingchangesinpsychologyandphilosophyclasses.TeachingofPsychology,41,28-36.doi:10.1177/0098628313514175

Butler,H.A.(2012).HalpernCriticalThinkingAssessmentpredictsreal-worldoutcomesofcriticalthinking.AppliedCognitivePsychology,26,721-729.doi:10.1002/acp.2851

Butler,H.A.,Dwyer,C.P.,Hogan,M.J.,Franco,A.,Rivas,S.F.,Saiz,C.,&Almeida,L.S.(2012).TheHalpernCriticalThinkingAssessmentandreal-worldoutcomes:Cross-nationalapplications.ThinkingSkillsandCreativity,7,112-121.doi:10.1016/j.tsc.2012.04.001

Carey,K.(2010,December17).Studentlearning:Measureorperish.ChronicleofHigherEducation,57(17),A72.

82

Clifford,J.S.,Boufal,M.M.,&Kurtz,J.E.(2004).Personalitytraitsandcriticalthinkingskillsincollegestudents.Assessment,11,169-176.doi:10.1177/1073191104263250

Davies,M.(2013).Criticalthinkingandthedisciplinesreconsidered.HigherEducationResearch&Development,32,529-544.doi:10.1080/07294360.2012.697878

Dunn,D.S.,&Smith,R.A.(2008).Writingascriticalthinking.InD.S.Dunn,J.S.Halonen,&R.A.Smith(Eds.),Teachingcriticalthinkinginpsychology(pp.163-173).Malden,MA:BlackwellPublishing.

*Ennis,R.H.,&Millman,J.(2005a).CornellCriticalThinkingTest,LevelX.PacificGrove,CA:MidwestPublications.

*Ennis,R.H.,&Millman,J.(2005b).CornellCriticalThinkingTest,LevelZ.PacificGrove,CA:MidwestPublications.

*Ennis,R.H.,&Weir,E.(1985).TheEnnis-WeirCriticalThinkingEssayTest.PacificGrove,CA:MidwestPublications.

*Facione,P.A.,Facione,R.N.,Blohm,S.W.,Howard,K.,&Giancarlo,C.A.F.(1998).CaliforniaCriticalThinkingSkillsTest:Manual(revised).Millbrae,CA:CaliforniaAcademicPress.

Fero,L.J.,O’Donnell,J.M.,Zullo,T.G.,Dabbs,A.D.,Kitutu,J.,Samosky,J.T.,&Hoffman,L.A.(2010).Criticalthinkinginnursingstudents:Comparisonofsimulation-basedperformancewithmetrics.JournalofAdvancedNursing,66,2182-2193.doi:10.1111/j.1365-2648.2010.05385.x

Frantz,S.,&McCarthy,M.A.(inpress).Challengingthestatusquo:Evidencethatintroductorypsychologycandispelmyths.TeachingofPsychology.

French,B.F.,Hand,B.,Nam,J.,Yen,H.-J.,&Vazquez,V.(2014).DetectionofdifferentialitemfunctioningintheCornellCriticalThinkingTestacrossKoreanandNorthAmericanstudents.PsychologicalTestandAssessmentModeling,56,275-286.

Halonen,J.(1995).Thecriticalthinkingcompanionforintroductorypsychology.NewYork,NY:Worth.

Halonen,J.S.(2008).Measureformeasure:Thechallengeofassessingcriticalthinking.InD.S.Dunn,J.S.Halonen,&R.A.Smith(Eds.),Teachingcriticalthinkinginpsychology(pp.61-75).Malden,MA:BlackwellPublishing.

Halpern,D.F.(1987).Analogiesasacriticalthinkingskill.InD.E.Berger,K.Pezdek,&W.P.Banks(Eds.),Applicationsofcognitivepsychology:Problemsolving,education,andcomputing(pp.305-313).Hillsdale,NJ:Erlbaum.

Halpern,D.F.(1996).Thinkingcriticallyaboutcriticalthinking.Mahwah,NJ:Erlbaum.Halpern,D.F.(1999).Teachingforcriticalthinking:Helpingcollegestudentsdeveloptheskills

anddispositionsofacriticalthinker.NewDirectionsforTeachingandLearning,80,69-74.SanFrancisco,CA:Jossey-Bass.

Halpern,D.F.(2003a).Thinkingcriticallyaboutcreativethinking.InM.A.Runco(Ed).Criticalcreativeprocesses.Perspectivesoncreativityresearch(pp.189-207).Cresskill,NJ:HamptonPress.

Halpern,D.F.(2003b).Thoughtandknowledge:Anintroductiontocriticalthinking(4thed.).Mahwah,NJ:Erlbaum.

Halpern,D.F.(2013).Aisforassessment:Theotherscarletletter.TeachingofPsychology,40,358-362.doi:10.1177/0098628313501050

83

*Halpern,D.F.(2010).HalpernCriticalThinkingAssessment.ViennaTestSystem.Modling,Austria:SchuhfriedGmbH.Retrievedfromhttp://www.schuhfried.at/

Haw,J.(2011).ImprovingpsychologicalcriticalthinkinginAustralianuniversitystudents.AustralianJournalofPsychology,63,150-153.doi:10.1111/j.1742-9536.2011.00018.x

Keeling,R.P.,&Hersh,R.H.(2012,July/August).Where’sthelearninginhigherlearning?Trusteeship,20(4),16-21.

Ku,K.Y.L.(2009).Assessingstudents’criticalthinkingperformance:Urgingformeasurementsusingmulti-responseformat.ThinkingSkillsandCreativity,4,70-76.doi:10.1016/j.tsc.2009.02.001

*Lawson,T.J.(1999).Assessingpsychologicalcriticalthinkingasalearningoutcomeforpsychologymajors.TeachingofPsychology,26,207-209.doi:10.1207/S15328023TOP260311

*Lawson,T.J.,Jordan-Fleming,M.K.,&Bodle,J.H.(2015).Measuringpsychologicalcriticalthinking:Anupdate.TeachingofPsychology,42,248-253.doi:10.1177/0098628315587624

Lilienfeld,S.O.,Lohr,J.M.,&Olatunji,B.O.(2008).Encouragingstudentstothinkcriticallyaboutpsychotherapy:Overcomingnaïverealism.InD.S.Dunn,J.S.Halonen,&R.A.Smith(Eds.),Teachingcriticalthinkinginpsychology(pp.267-271).Malden,MA:BlackwellPublishing.

Macpherson,K.,&Owen,C.(2010).Assessmentofcriticalthinkingabilityinmedicalstudents.Assessment&EvaluationinHigherEducation,35,45-58.doi:10.1080/02602930802475471

Magno,C.(2010).Theroleofmetacognitiveskillsindevelopingcriticalthinking.MetacognitionLearning,5,137-156.doi:10.1007/s11409-010-9054-4

McGovern,T.V.,Furumoto,L.,Halpern,D.F.,Kimble,G.A.,&McKeachie,W.J.(1991).Liberaleducation,studyindepth,andtheartsandsciencesmajor–Psychology.AmericanPsychologist,46,598-605.

McGovern,T.V.,Corey,L.,Cranney,J.,Dixon,W.E.,Jr.,Holmes,J.D.,Kuebli,J.E.,…&Walker,S.J.(2010).Psychologicallyliteratecitizens.InD.F.Halpern(Ed.),Undergraduateeducationinpsychology:Ablueprintforthefutureofthediscipline(pp.9-28).Washington,DC:AmericanPsychologicalAssociation.

McLean,C.P.,&Miller,N.A.(2010).Changesincriticalthinkingskillsfollowingacourseonscienceandpseudoscience:Aquasi-experimentalstudy.TeachingofPsychology,37,85-90.doi:10.1080/0098628100462714

Muehlenkamp,J.J.,Weiss,N.,&Hansen,M.(2015).Problem-basedlearningforintroductorypsychology:Preliminarysupportingevidence.ScholarshipofTeachingandLearninginPsychology.Advanceonlinepublication.http://dx.doi.org/10.1037/stl0000027

Norris,S.P.,&Ennis,R.(1989).Evaluatingcriticalthinking.PacificGrove,CA:MidwestPublications.

O’Hare,L.,&McGuinness,C.(2009).Measuringcriticalthinking,intelligence,andacademicperformanceinpsychologyundergraduates.TheIrishJournalofPsychology,30,123-131.doi:10.1080/03033910.2009.10446304

84

Pascarella,E.T.,Martin,G.L.,Hanson,J.M.,Trolian,T.L.,Gillig,B.,&Blaich,C.(2014).Effectsofdiversityexperiencesoncriticalthinkingskillsover4yearsofcollege.JournalofCollegeStudentDevelopment,55,86-92.doi:10.1353/csd.2014.0009

Preiss,D.D.,Castillo,J.C.,Flotts,P.,&SanMartin,E.(2013).Assessmentofargumentativewritingandcriticalthinkinginhighereducation:Educationalcorrelatesandgenderdifferences.LearningandIndividualDifferences,28,193-203.doi:10.1016/j.lindif.2013.06.004

*Raven,J.(1965).Progressivematrices.London,England:H.K.Lewis.Schraw,G.,&Gutierrez,A.(2012).Assessmentofthinkingskills.InM.F.Shaughnessy(Ed.),

Criticalthinkingandhigherorderthinking:Acurrentperspective(pp.191-203).NewYork,NY:NovaSciencePublishers.

Stark,E.(2012).Enhancingandassessingcriticalthinkinginapsychologicalresearchmethodscourse.TeachingofPsychology,39,107-112.doi:10.1177/0098628312437725

Szabo,Z.,&Schwartz,J.(2011).Learningmethodsforteachereducation:Theuseofonlinediscussionstoimprovecriticalthinking.Technology,PedagogyandEducation,20,79-94.doi:10.1080/1475939X.2010.5344866

*Watson,G.,&Glaser,E.M.(1980).Watson-GlaserCriticalThinkingAppraisal.SanAntonio,TX:PsychologicalCorporation.

Wesp,R.,&Montgomery,K.(1998).Developingcriticalthinkingthroughthestudyofparanormalphenomena.TeachingofPsychology,25,275-278.doi:10.1080/00986289809709714

Yanchar,S.C.,Slife,B.D.,&Warne,R.(2008).Criticalthinkingasdisciplinarypractice.ReviewofGeneralPsychology,12,265-281.doi:10.1037/1089-2680.12.3.265

85

Appendix:ACompendiumofGeneralCriticalThinkingMeasures,withBriefDescriptionsMeasure BriefDescription

CaliforniaCriticalThinkingSkillsTests

Basedoninformationprovided,taskswithincreasingdifficultyarepresented.Separatescalescoresavailableforanalysis,interpretation,evaluation,explanation,deductivereasoning,inductivereasoningandatotalcriticalthinkingskillsscore.

CambridgeThinkingSkillsAssessment

Presents50multiplechoicequestionsmeasuringcriticalthinkingandproblemsolvingskills,includingnumericalandspatialreasoning,criticalthinking,understandingargumentsandeverydayreasoning.Availableonlineandpaperandpencilforms.

CollegiateAssessmentofAcademicProficiency(CAAP)CriticalThinkingTest

Fourpassagesarepresentedfollowedbya32-itemmultiplechoicetestwhichstudentsclarify,analyze,evaluate,andextendarguments.Totalscoreisgenerated.

CollegiateLearningAssessment(CLA)CriticalThinking,AnalyticReasoning,andProblemSolving

Performanceandanalyticwritingtasksarepresentedthatmeasureastudent’sabilitytoevaluateevidence,analyzeandsynthesizeevidence,drawconclusions,andacknowledgealternativeviewpoints.

CornellCriticalThinkingTest Studentsaretestedondeduction,credibility,andidentificationofassumptions;appropriateforgrade5togrades12-14.

Ennis-WeirCriticalThinkingEssayTest

Testinginvolvesgettingthepoint,reasoningandassumptions,offeringalternativepossibilitiesandexplanations.Usedforgrade7throughcollege.Assessesproblemsolving,criticalthinking,andcommunication.

HalpernCriticalThinkingAssessment

Respondentsarepresentedwith25everydayscenarios,andfreeresponsesareconstructed;then,thescenariosarepresentedagainrequiringaforcedchoiceresponse.Thisprocedurehelpstoseparategenerationandrecognitionprocesses.

iCriticalThinking

Presentedwith14tasksbasedonreal-worldsituations,thisinstrumentiscompletedin60minutesandyieldsadigitalliteracycertificationspecifictocriticalthinkinginatechnology-enableddigitalenvironment.

InternationalCriticalThinkingEssayTest

Involvesanalysisofawritingprompt(identifytheelementsofreasoning)worth80possiblepoints,andassessmentofawritingprompt(usinganalysisandevaluation)worth20possiblepoints.

MeasureofAcademicProficiencyandProgress(MAPP)

Addressesreading,writing,mathematics,andcriticalthinking.Thecriticalthinkingsub-scorerangesfrom100to130.Studentsrespondtomultiplechoicequestionsrequiringevaluation,relevance,andrecognition.Studentperformanceisclassifiedasproficient,marginal,ornotproficient.

86

ProficiencyProfile

ThismultiplechoiceinstrumentequatestotheformerAcademicProfile,andyieldsacriticalthinkingproficiencylevel(LevelI,II,orIII).Availableinstandardform(108questions)orabbreviatedform(36questions).

Watson-GlaserCriticalThinkingAppraisal

Studentsareassessedondecision-makingskillsandjudgment;testtakersclassifiedaslow,average,orhighincriticalthinkingability.UsingFormS,40self-reportitemsareused;higherscoresindicategreatercriticalthinkingabilities.

87

Chapter8:StudentEngagementTowardCoursework:Measures,Considerations,andFutureDirectionsKevinL.ZabelandAmyHeger

UniversityofTennessee

Studentengagementisthefuelthatdrivesthepotentialforsuccessincollegecourses.Justasacarcannotoperatewithoutfuel,alackofstudentinterestorengagementhindersthebeneficialimpactofclass-facilitatedexperiences.Althoughmultipleoperationaldefinitionsofstudentengagementexist,studentengagementisbroadlydefinedasthequantityandqualityofphysicalandpsychologicalenergythatstudentsinvestinthecollegeexperience(Astin,1999)ortheextenttowhichstudentstakepartineducationallyeffectivepractices(Kuh,2003).Regardlessofconstructdefinition,studentengagementrelatestoacademicachievementandseveralimportantlearningoutcomes.Forexample,studentengagementislinkedtopersistenceamongfirst-andsecond-yearcollegestudents(Kuh,Cruce,Shoup,Kinzie&Gonyea,2008),retentionandreduceddrop-outrates(Finn,1989),achievement(Newmann,1992),gradepointaverage(Carini,Kuh,&Klein,2006),andaplethoraofotherpositiveoutcomes(seeJanosz,2012forareview).Theimportanceofstudentengagementincourseswasdramaticallyshowcasedinafieldexperimentwhereanengagementactivitythatconnectedsciencetostudents’personallivesincreasedstudentinterestinclass,aswellasclassperformance,especiallyamongstudentswithlowexpectationsofsuccess(Hulleman&Harackiewicz,2009).Althoughtheimportanceofstudentengagementandinterestinclassroommaterialseemsestablished,lessfoundedarepsychometricallysoundandwell-validatedmeasurestoaccuratelyandreliablyassessstudentengagementandinteresttowardclassroommaterials. Studentengagementisanattitudethat,likeallattitudes,variesamongindividualsintermsofstrengthandvalence.Pastscholarsandresearchershaveutilizedamultifacetedoperationalization(Fredricks,Blumenfeld,&Paris,2004;Furlong,Whipple,St.Jean,Simental,Soliz,&Punthuna,2003;Jimerson,Campos,&Greif,2003)todefinestudentengagement.Specifically,studentengagementconsistsofaffective(e.g.,relationshipswithpeersandteachers,emotions),behavioral(e.g.,effort,studentparticipation),andcognitive(e.g.,investment,personalgoals,autonomy)components(Appleton,Christenson,&Furlong,2008;Fredricksetal.,2004;Furlongetal.,2003;Jimersonetal.,2003).Thedistinctionbetweenaffective,behavioral,andcognitivecomponentsofstudentengagementbecomesclearwhenexamininghoweachmayimportantlyimpactoneanotherandleadtospecifictypesoflong-termconsequences.Forinstance,areducedsenseofbelongingwithinaschoolorclassroom(affectivecomponentofstudentengagement)mayleadtowithdrawingfromschoolactivities(behavioralcomponent),whichinturnleadstocognitiveperceptions(“schoolisnotimportanttomyself-concept”)thathavenegativelong-termconsequences. Themultidimensionalnatureofstudentengagementhasledtodivergencesinthemeasurementandoperationalizationoftheconstruct.Indeed,alackofaunifyingthemeindefiningandthusmeasuringtheconstructisaproblemelaboratedonbymany(e.g.,Reschly&Christenson,2012).Someeducationresearchershavecalledforamoreprecise

88

operationalizationformeasuringstudentengagementtoallowforarefinedunderstandingofhowandunderwhatcircumstancesstudentengagementpredictslearning(Axelson&Flick,2010).Theseconsiderationsaside,muchextantresearchhasmeasuredstudentinteresttowardteachersandinclusionwithintheclassroom.However,adearthofresearchhasexaminedthemeasurementofstudentengagementasitpertainstoclassroommaterialinparticular,especiallyregardingstudentengagementinpostsecondaryeducationsettings.Inwhatfollows,wecriticallyreviewwidely-utilizedmeasuresrelevanttoassessingstudentengagement,broadlydefined,towardclassmaterial,focusingontheircommonalitiesanddistinctionsbothfrompragmaticandpsychometricperspectives. Researcherswhostudyengagementhavemeasureditatbothamacro-andamicro-level.Themacro-levelformofengagementfocusesonmeasuringelementsrelatedtoinvestmentandeffortinschool-relatedactivities(e.g.,Marks,2000;Newmann,Wehlage,&Lamborn,1992;Skinner,Wellborn,&Connell,1990)andidentificationorconnectionbroadlywithschoolandacademics(Finn,1993;Finn&Rock,1997).Themicro-levelformofengagementfocusesonmeasuringengagementinoneparticularschool-relatedaspect,suchasaparticularcourseorstudentactivity(Handelsman,Briggs,Sullivan,&Towler,2005).Althoughmacro-andmicro-levelengagementmeasuresarepositivelyassociated,micro-levelmeasuresmayallowforamorenuancedunderstandingofengagementtowardaparticularclassandtherelevantbehavioral,cognitive,andaffectivefactorsinthatcontext.Forexample,facultycanallrecalltheexampleofastudentwhoispurportedtobeengagedinfellowfacultymembers’classes,butclearlyisnotinone’sownclass.Thisexampleissimplebutillustratesanimportantpoint:macro-levelmeasuresofstudentengagementmaynotbeasvalidinassessingstudentengagementtowardparticularaspectsofclassmaterialoraparticularclass.However,macro-levelstudentengagementmeasurescanbeespeciallyusefulinuncoveringastudent’sengagementwithschoolmoregenerally,whichitselfcanhaveseveralpositiveoutcomes.Aswithanymeasure,trade-offsexistinusingmicro,relativetomacro,measures.Givenourfocusonstudentengagementtowardcoursework,thebulkofthischapterfocusesonmicro-levelmeasures. Macro-LevelMeasure

NationalSurveyofStudentEngagement(NSSE)Themainmacro-levelmeasureutilizedtoassessstudentengagementistheNSSE(Kuh,2001).TheNSSEisusedextensivelybycollegesanduniversitiestoevaluateengagementamongfreshmanandseniorstudents.TheNSSEconsistsofaseriesofcollegeactivity(behavior)itemsdirectlymeasuringstudentengagement,educationalandpersonalgrowthitems,anditemsregardingopinionsaboutone’sschool,aswellasavarietyofotherquestions.TheNSSEpositivelypredictsavarietyoflearning-relatedoutcomes,includinggradepointaverage,criticalthinking,andstandardizedtestscores(Carinietal.,2006;Ewell,2002;Pascarella&Terenzini,2005).Nevertheless,theNSSEisdesignedtoassessself-engagementbroadlytowardthecollegeexperience,andnotparticularlytowardcourseworkfromspecificclasses.Furthermore,theNSSElacksarigoroustheoreticalorientationthatdrivestheorganizationanduseof

89

particularitems.Indeed,apointofemphasisinthecurrentchapteristhatmeasuresofstudentengagementcanbeimprovedbybeinggroundedintheoreticalrationale. Micro-LevelMeasures

StudentInterestStudentengagementhasbeenoperationalizedandmeasuredinavarietyofmanners.Wefocustheremainderofthischapteronseveralmicro-levelmeasuresutilizedinpreviousresearchtoassessstudentengagement,broadlydefined.Studentinterestisonesuchmannerinwhichengagementtowardcollegecoursesandmaterialhasbeenassessed.

13-ItemStudentInterestMeasureOneusefulmeasureofstudentinterestisHarackiewicz,Barron,Tauer,Carter,andElliott’s(2000)13-itemmeasure.Theitemswithinthismeasurefocusoninteresttowardaparticularclass,thelectureswithintheclass,andtheprofessorteachingtheclass.Thismeasurehassomeoverlapwithaprevious7-itemmeasureofstudentinterest(Harackiewicz,Barron,Carter,Lehto,&Elliot,1997).However,this13-itemmeasure,unlikeitspredecessor(Harackiewiczetal.,1997),differentiatesbetween“catch”and“hold”interestfactors.Catchinterestfactorsinitiallytriggerstudentinterest,andmayconsistofflashyPowerpointslides,grippingexamples,orstimulatingteachingmethodsthatleadtoinitialclassenjoyment.Threeitemsassesscatchinterest(α=.93),including“Idon’tlikethelecturesverymuch(reverse-coded),”“Thelecturesinthisclassseemtodragonforever(reverse-coded)”,and“Ilikemyprofessor.”Holdinterestfactorsincludecourseelementsthatmaintainstudentinterestoverasustainedperiodoftimebyactivatingintrinsicmotivation(Harackiewiczetal.,2000).Tenitems(α=.72)assessholdinterest,suchas“Ithinkthecoursematerialinthisclassisusefulformetolearn”and“Ithinkthefieldofpsychologyisveryinteresting.”Participantsrespondtoitemsusinga1(stronglydisagree)to7(stronglyagree)responserange.The3-itemmeasureofcatchinterestislabeledasEnjoyment,andthe10-itemmeasureofholdinterestislabeledasanInterestScale(Harackiewiczetal.,2000). Interestingly,theholdmeasureofstudentinterest(i.e.,theInterestScale)wasabetterpredictorofperformance(i.e.,learning)inclassesandofthenumberofsubsequentpsychologyclassesstudentstook,suggestingthatitmightbeanespeciallyimportantdimensiontofacilitateamongstudentstopromotelearningoutcomes.Thecatchandholdmeasuresarepositivelycorrelated(r=.58),buttheiroperationaldefinitionssuggestthattheyareconceptuallydistinctdimensionsofstudentinterestthatshouldbetreatedassuchinfutureresearch(Harackiewiczetal.,2000).Psychometrically,itisimportanttonotethatthe“catch”interestscaleonlyhasthreeitems,andlikemanyothermeasuresofstudentinterestandengagement,lackspropervalidation.Futureresearchshouldexamineuniquewaysthatcatchandholdinterestpredictstudent-relatedoutcomes,aswellasmeansbywhichtheycanbeobtainedthroughteachingmethodsandotherfeaturesofparticularclasses,keepinginmindthatthefactorsthatenhancecatchandholdinterestmayvaryasafunctionclasstypesubjectmatterandlevel.

90

InterestasanEmotionStudentinterestcanalsobeoperationalizedasanemotion(Izard,1977)thatfocusesattentionandopennesstoreceivinginformation(Dougherty,Abe,&Izard,1996).Fromthisperspective,interestisastableemotionthatshouldinfluenceintrinsicmotivationtolearnandinterestinacademicendeavors.TheInterestSubscaleoftheDifferentialEmotionsScale(Izard,Libero,Putnam,&Haynes,1993)hasbeenutilizedbypreviousresearchers(Bye,Pushkar,&Conway,2007)tooperationalizeinterestbroadlyasadispositionalemotionrelevanttoacademicoutcomes.TheInterestSubscale(α=.75;Byeetal.,2007)isa3-itemmeasureofthedegreetowhichindividualsgenerallyexperiencetheemotionofinterest,andconsistsoftheitems“Howoftendoyoufeelsointerestedinwhatyou’redoingthatyou’recaughtupinit?,”“Howoftendoyoufeellikewhatyou’redoingorwatchingisinteresting?,”and“Howoftendoyoufeelalert,curious,andkindofexcitedaboutsomething?”Participantsrespondusinga1(rarelyornever)to5(veryoften)responserange.Interestconceptualizedthiswayispositivelycorrelatedwithintrinsicmotivationtolearn(Byeetal.,2007).Itisimportanttonote,however,thatthismeasureofinterestismacro-level,anddoesnotpertaintostudentinterestorengagementtoparticularaspectsofclassroommaterial,thoughitemscouldeasilybeadaptedtoassessthedegreetowhichstudentsexperiencetheemotionofinterestspecificallytowardclasses.

StudentEngagement

StudentCourseEngagementQuestionnaire(SCEQ)Handelsmanandcolleagues(2005)createdtheSCEQtoassessstudentengagementspecifictoacollegecourse.Contrarytoothermeasuresofstudentengagement(e.g.,NSSE),thismeasureassessesmoremicro-levelclassengagement.Participantsratewhethereachof23itemsaredescriptiveofthemusinga1(notatallcharacteristicofme)to5(verycharacteristicofme)responserange.Handelsmanandcolleagues’exploratoryfactoranalysisrevealedthatstudentcourseengagementwascomposedoffourfactors:1)askillsengagementfactor(α=.82)thatcapturesengagementbymeansofpracticingskills(exampleitemsare“Takinggoodnotesinclass”and“Makingsuretostudyonaregularbasis”),2)anemotionalengagementfactor(α=.82)thatcapturesengagementbymeansofemotionalinvolvementwithclassmaterial(exampleitemsare“Findingwaystomakethecourseinterestingtome”and“Applyingcoursematerialtomylife”),3)aparticipation/interactionengagementfactor(α=.79)thatcapturesengagementbymeansofclassparticipationandinteractionswithteachersandotherstudents(exampleitemsare“Participatingactivelyinsmall-groupdiscussions,”and“Helpingfellowstudents”),and4)aperformanceengagementfactor(α=.76)thatcapturesengagementbymeansofperformancelevel(exampleitemsare“Gettingagoodgrade”and“Doingwellonthetests”). Furthermore,Handelsmanandcolleagues(2005)demonstratedthepredictivevalidityoftheSCEQbyfindingthatparticularfactorsofthemulti-facetedengagementconstructwerepositivelyrelatedtomidtermandfinalexamgradesinamathematicscourse.Additionalstudies

91

employingtheSCEQhaveshownothersignificantrelationshipswithacademicinitiativesandoutcomes.Forexample,participationinstudentmanagementteams(SMTs)increasescourseengagement,anditisthroughthisengagementthatSMTspositivelyinfluencecourseperformance(Troisi,2014).Also,differenttypesofengagedlearningactivitiesincollege(e.g.internships,undergraduateresearch)promoteparticularfactorsofengagementfromtheSCEQ(Miller,Rycek&Fritson,2011). Overall,givenitscharacteristics,theSCEQmaybeausefulmeasureofstudentengagementwhenaparticulardomainorclassisthechieftargetofinterest.Handelsmanandcolleagues(2005)mentioncategoriesfromthemacro-levelNSSEthattheyanticipatedwouldparallelparticularSCEQfactors,buttheydidnotcollectdatatocomparethetwoscales.FutureresearchshouldusealongitudinaldesigntoinvestigateifstudentscoresontheSCEQacrosscoursesconvergetopredictoverallcollegeengagementontheNSSE.Worrisomeaspectsofthescale,however,needtobehighlighted.Specifically,notheoreticalfoundationwasarticulatedinSCEQquestionnaireitemdevelopmentorregardingamulti-dimensionalfactorstructure.Reasoningwasprovidedpost-hocforeachfactorandreferencesofsupportiveresearchweresomewhatsparse.Thedependentvariableschosentotestpatternsforconvergentanddiscriminantvaliditywereone-itemmeasures,whicharenotoriousforbeingunreliable(e.g.,Churchill,1979;Guilford,1954),soadditionalrigoroustestingofthescale’spsychometricpropertiesshouldbepursued.Indeed,thelackofrigorouspsychometrictestingofteaching-relatedmeasuresisanissuediscussedinChapter3ofthise-book(Christopher,2015).

UtrechtWorkEngagementScale-Student(UWES-S).Anadditionalandmorepsychometricallyrigorousassessmentofstudentengagementhasarisenoutoftheorganizationalpsychologyliterature.TheUWES-Sevolvedoutofresearchonengagementintheworkplace(e.g.,Schaufeli&Bakker,2003;Schutte,Toppinen,Kalimo,&Schaufeli,2000),andconsistsofathree-dimensional,17-itemmeasureofstudentengagement(Schaufeli,Salanova,González-Romá,&Bakker,2002). Engagementisdelineatedbythreecore-dimensions:vigor,dedication,andabsorption.Vigorisoperationalizedashighlevelsofenergy,devotingefforttoone’swork,andmentalresilienceandperseverance.Sixitemsareusedtoassessvigor(e.g.,“WhenI’mdoingmyworkasastudent,Ifeelburstingwithenergy”).Dedicationisoperationalizedasasenseofdedication,significance,enthusiasm,inspiration,pride,andchallengeinone’swork.Dedicationisassessedbyfiveitems(e.g.,“I’menthusiasticaboutmystudies”).Absorptionisoperationalizedasbeingfullyconcentratedanddeeplyengrossedinone’swork(Schaufelietal.,2002,pp.74).Absorptionisassessedbysixitems(e.g.,“WhenIamstudying,Iforgeteverythingelsearoundme”).ItemresponsesarerecordedonaLikertscalefrom0(never)to6(always).Recentcross-nationalresearchhasconfirmedthesoundpsychometricpropertiesoftheUWES-S(Schaufeli,Martínez,Marques-Pinto,Salanova,&Bakker,2002;Schaufelietal.,2002),aswellasthegeneralityofthemeasurementmodelacrossculturesfortheabsorptionandvigorsubscales.Futureworkshouldexamineifthedimensionsstructureisconsistentacrossdifferentacademiclevels(i.e.,elementary,secondary,post-secondary)aswell.

92

Overall,theUWES-Sdoesasolidjobofassessinginvolvementandmorepersonalseemingaspectsoftheengagementexperiencewithitsitemsaddressinginvestment,persistence,dedication,lossoftime,etc.Thenon-educationoriginoftheengagementscaleshouldnotbeofconcernbecauseitspsychometricpropertieshavebeeninvestigatedrigorously,moresothanthoseoftheSCEQ,forinstance.Communicationacrossdisciplinesisoftenlessthansatisfactory(Christopher,2015)andtheUWES-Smeasureofstudentengagementisaninstanceinwhichteachingofpsychologyresearcherscouldgainfromexistingmeasuresintheorganizationalpsychologyliterature.

AncillaryMeasures

GritSeveralmeasuresthatmaynotbeconsideredtraditionalmeasuresofstudentengagementmayholdutilityintappingintotheconstruct.Gritisonesuchmeasure,andisdefinedasperseveranceandpassionforlong-termgoalsandinterests(Duckworth,Peterson,Matthews,andKelly,2007,pp.1087).TheprominentgritmeasureisDuckworthetal.’s(2007)12-item,two-factormeasure(α=.85).ThetwofactorsincludeConsistencyofInterests(6itemsα=.84)andPerseveranceofEffort(6items;α=.78).Consistencyitemsinclude“Ioftensetagoalanddecidetopursueanewone”(reverse-coded)and“Myinterestschangefromyeartoyear”(reverse-coded).PerseveranceofEffortitemsinclude“Iamahardworker”and“Ihaveachievedagoalthattookyearsofwork.”Ashorter8-itemversion(Duckworth&Quinn,2009)withthesamefactorstructureastheoriginalgritscale(Duckworthetal.,2007)hasalsobeenwell-validated(α=.83). Gritconceptuallysharessimilaritieswiththestudentengagementandinterestmeasurespreviouslyreviewed,itisdistinctinitsfocusonlong-term,consistenteffortandperseverance.Althoughnotyetusedintheteachingofpsychologyrealm,webelievethatmeasuresofgritmayholdparticularrelativeadvantagesoverstudentinterestandengagementmeasuresinassessingengagementtowardcoursematerialinwhichconsistent,long-termengagementisrequired(e.g.,multi-stageassignmentssuchasempiricalresearchpapers).Indeed,utilizingmeasuresofgritincombinationwithmeasuresofstudentengagementandinterestmayholdadvantagessuchasaidingindistinguishingstudentswholackengagementwithaparticularactivityoraspectofaspecificcourse,incomparisontothosestudentswhoconsistentlylackengagementtowardaspectsofcoursesingeneraloverthelong-term.

BoredomAnotherworthwhileperspectiveonengagementistoexaminetheopposite,thetendencytowithdrawfromacademicactivities.Boredomisdescribedasanunpleasantemotionalstatecharacterizedbylowarousalandalackofstimulation(Mikulas&Vodanovich,1993).Boredomisunderstoodasarelevantemotionforacademicexperiences,andhasbeeninvestigatedas

93

oneofeightachievementemotionsmeasuredbytheachievementemotionsquestionnaire(AEQ;Pekrun,Goetz,&Perry,2005;Pekrun,Goetz,Titz,&Perry,2002).

AchievementEmotionsQuestionnaire(AEQ)Achievementemotionscanariseindifferentacademicsettings.Boredom,therefore,ismeasuredwithtwoscalesintheAEQ;onespecifiesitasalearning-relatedemotionandtheotherasaclass-relatedemotion.InstructionsdifferbyaskingparticipantstorateonaLikertscalefrom1(completelydisagree)to5(completelyagree)howtheitemspertaintofeelingstheymayexperienceduringstudying(learning-related)orduringclass(class-related;each11items).Learning-relatedboredomscaleitems(α=.93)include“IfindmymindwanderingwhileIstudy”and“BecauseI’mboredIhavenodesiretolearn.”Class-relatedboredomscaleitems(α=.92)include“Ifindthisclassfairlydull”and“I’mtemptedtowalkoutofthelecturebecauseitissoboring.” Pekrun,Goetz,Daniels,Stupnisky,andPerry(2010)classifyboredomasanegativedeactivatingachievementemotionbecauseitdiminishesmotivationandcanhavenegativeeffectsonperformance.Inparticular,boredommeasuredwiththeAEQispositivelyrelatedtoattentionproblemsandnegativelyrelatedtointrinsicmotivationtolearn,studyeffort,academicperformancescoresandotherdetrimentaleducationaloutcomes(Pekrunetal.,2010).Pekrun,Hall,Goetz,andPerry(2014)foundthat,withalongitudinalstudyacrosstheacademicyear,areciprocalcycleexistsbetweencourse-relatedboredomandexamperformance,suchthatboredomnegativelyimpactssubsequentexamperformanceandexamperformancenegativelyimpactssubsequentboredom.Unlikealackofinterestorenjoyment—neutralstatescharacterizedbyanabsenceofapproachmotivation—boredominstigatesanavoidancemotivation(Pekrunetal.,2010).Adesiretoescapethesituationoccurs,whichmightleadstudentstodisengagefromtheirlearningandcourse-work.

AcademicBoredomSurveyThe10-itemAcademicBoredomSurvey(ABS-10)isanotherscalecreatedtoassessboredom(Acee,Kim,Kim,Kim,Hsiang-Ning,Kim,Cho,Wicker,&theBoredomResearchGroup,2002).TheABS-10takesintoaccountwhetherboredomoriginatedfromataskbeingtoohardortooeasybyaskingparticipantstorecallasituationinwhichtheywereover-challengedorunder-challenged.InstructionsspecifytorespondonaLikertscalefrom1(notatall)to9(extremely)theextenttowhichanitemistrueforeachsituation.Sampleitemsare“Wanttodosomethingelse”and“Findtheactivitydull.”Confirmatoryfactoranalysisofthescalerevealeda2-factorsolutionforinstancesofbeingover-challenged(i.e.self-focusedfactorandtask-focusedfactor)anda1-factorsolutionforinstancesofbeingunder-challenged.Thesethreefactorshavedemonstratedexcellentlevelsofreliability(i.e.,α’s=.86,.80,&.90,respectively;Aceeetal.,2010).However,futureresearchshouldaimtofurthervalidatethemeasure,duetodrawbacksinanalysesandassumptionsmadeininitialmeasuredesign.Additionally,theresultsoftheABS-10comparedtotheAEQ,whilefoundtobesignificantlyrelatedtoboredom,werealsorelatedtoagreaterextenttootherachievementemotions(i.e.anger,hopelessness)forself-focusedscaleitems.Overall,boredomresearchusingtheAEQhasbeenmostproductivein

94

validatingameasureofboredomandestablishingitsharmfulrelationshipforacademicperformanceoutcomes(e.g.Pekrunetal.,2010;Pekrunetal.,2014).However,theABS-10givesscholarsanothermeasurementtoolthatisrelevantforunderstandingdifferencesinboredomasaresultoftaskcharacteristics(i.e.,beingtoochallengedornotchallengedenough). RecentDevelopments

StudentEngagementScale(SES)OnerecentmeasureofcollegestudentengagementthatisencouragingwithregardtoitstheoreticalbaseandcomponentsistheSES(Gunuc&Kuzu,2015).TheSESisasix-factormeasureofstudentengagementthatexaminesbothcampusandclassengagementandspecificallytheaffective,cognitive,andbehavioralcomponentsofclassengagement.Thus,itisrootedinthetripartiteattitudeconceptualizationofstudentengagement,andalsoconsidersthemacro(i.e.,campus)andmicro(i.e.,class)factorsofstudentengagement.TheitemsoftheSESwerederivedthroughpreviousmeasuresofengagementandqualitativepilotinterviews,aswellasbasedinanunderlyingtheoreticalperspective.ReliabilityfortheoverallSESisexcellent(α=.96),asitisforeachofthesixsmallerfactorsaswell(α’srangefrom.81to.91).Furthermore,severalaspectsofthedesignandimplementationofthismeasurearelaudable.Specifically,GunucandKuzu(2015)groundedthemeasuredevelopmentintheoryandlinkedittopreviousresearchconsideringthemacrovs.microdistinctionsandthemultidimensional(e.g.,affective,behavioral,andcognitive)natureofengagement.AlthoughresearchusingtheSESisinitsinfancyandfurtherempiricalresearchisneededtovalidatethemeasure,webelieveittoholdgreatpromiseasastudentengagementmeasureforfutureresearch. DiscussionandFutureDirectionsOurhopeistohaveprovidedarangeofempirically-validatedmeasuresrelevanttoassessmentofstudentinterestandengagementinclassmaterial.Itisimportanttohighlight,however,thatwithalackofconsensusonanoperationaldefinitionofinterestandengagementincoursematerial,manyresearchershavecreatedtheirownmeasuresforthepurposeofmeasuringinterestandengagementrelevanttotheirparticularresearchquestions.Forexample,someresearcherscreatescaleswheretheyselectonlycertainitemsforinclusionfrompreexistingmeasures(e.g.Gasiewski,Eagan,Garcia,Hurtado,&Chang,2012;Langley,2006;Svanum&Bigatti,2009),orcreatetheirownitems(e.g.Marks,2000;Milleretal.,2011;Taneja,Fiore&Fischer,2015).Weexpressconcernoversuchpracticesduetocriticaldeficitsininvestigationofthepsychometricqualitiesinmanyoftheseinstances. Specializedmeasuresshouldnotbeoverlooked,nonetheless.Thewaysinwhichstudentsengagewithengineeringcourseworkversuspoetrymightvarygreatly.Limitationsofthegeneralizabilityofthecatchallengagementmeasureswereviewherearelikely.Infact,someresearchershaveadaptedversionstoaddressspecifictopicssuchasthemathversionoftheAEQ(AEQ-M;Pekrun,Goetz,&Frenzel,2005).Wedidnotdiscussindepthinterestindomain-specificmaterial,butfeelitisnecessarytoemphasizetheimportanceofsuchapproaches.

95

Recentefforts(Saddler,Sonnert,Hazari,&Tai,2012)toinvestigatefemales’interestandinvolvementintheSTEMfields(orlackthereof)isjustoneexampleoftheimportanceofspecialization. Whendiscussingstudentinterestandengagement,itisvitaltocallattentiontonewopportunitiesformeasurementwiththeconstantlychanginglandscapeofthetraditionallearningenvironment.Thedevelopmentofanewdigitalage,whereelectronicsandmediahavebecomemoreversatileandprevalent,hasgeneratedanewareaofstudentengagementresearch(beyondthescopeofscalesdiscussedhere).Empiricalinvestigationshavebeguntoexploretechnology’srolefromconcernsofmaintainingengagementvianewonlineclassplatforms,toenhancingengagementviaintegrationofsuchnewtechnologiesduringclass,toinhibitingengagementviaelectronicdistractionsinthelecturehall.Onewidespreadexampleofhowtechnologyadvanceshavepermeatededucationalsettingsistheusageofaudienceresponsesystemsatuniversities(Abrahamson,2006).Oneversioniswirelessremotescalledclickersthatenablestudentstoinstantaneouslyrespondtoquestionsduringlectures.Clickersareuniquefromtraditionalhand-raisingasaresponsesysteminthattheyprovideanonymity,oftentimespromptinggreaterparticipation.Indeed,researchhasuncoveredincreasesininterest,understanding,andattendanceinbiologycourseswiththeuseofclickers(Preszler,Dawe,Shuster,&Shuster,2007). Theroleofvideogames,virtualreality,andpodcastsinacademiccourseshavealsobeguntobediscussed(Bouta,Retalis&Paraskeva,2012;Peden&Domask,2011;Shernoff,2013).Socialmediaadditionallypresentsanotherprospectiveavenueinwhichstudentsmightbemorelikelytointeractwithcoursematerial.Althoughusingsocialmedia(e.g.,Twitter)asaneducationaltoolcanbebeneficialinsomecontexts(Junco,Heiberger,&Loken,2011),communicativeactivitiesonsocialmedia(e.g.,Facebookcommenting)arenegativelyassociatedwithcollegeengagement(Junco,2012).Indeed,constantconnectivelymighthavedetrimentaleffectsforlearningenvironments,particularlyconceivableasadistractionfromengagingwiththecoursematerialtobelearned(Sana,Weston,&Cepeda,2013;Tanejaetal.,2015).Theeffectsoftechnologiesonengagementarearipeareaforfutureinvestigation. Asishighlightedinthisreview,diversemeasurescanbeusedtoexaminebothmicroandmacrolevelsofstudentengagement.Futureresearchersshouldbemindfultochoosemeasuresforusebasedonthetheoreticalperspectiveoftheirresearchquestion.Thus,ifaresearcher’stheoreticalperspectiveisbasedonanaffectiveconceptualizationofstudentengagementtowardaparticularclassoraspectofclass,thenastudentinterest(Harackiewiczetal.,2000)orboredom(Aceeetal.,2010;Pekrunetal.,2005)measuremaybeideal.Ifamultidimensionalconceptualizationofstudentengagementtowardaclassisrequired,thentheSES(Gunuc&Kuzu,2015)ortheSCEQ(Handelsmanetal.,2005)maybeidealmeasures.Researchersinterestedinpredictinglong-termengagementmaywanttoconsidertheGritScale(Duckwothetal.,2007)asaviablemeasure.Furthermore,thevalidationofdomain-specificmeasuresofstudentengagementmaybeespeciallyfruitfultopredictdomain-specificoutcomes.Finally,measuresofstudentengagementthatfocuson“catch”elementsofstudentengagementthatmayleadto“hold”factorsofstudentengagementthatmediatestudent-relatedoutcomes

96

couldbeparticularlyproductivegivenenhancementsinteachingtechnologyandteachingmethods.Overall,aneedformultidimensionalmeasuresofstudentengagementtobemoretheoretically-basedandthoroughlyvalidatedpresentsmanyavenuesforfutureresearchonthetopic.

97

ReferencesReferencesmarkedwithanasteriskindicateascale.Abrahamson,L.(2006).Abriefhistoryofnetworkedclassrooms:Effects,cases,pedagogy,and

implications.InD.A.Banks(Ed.),Audienceresponsesystemsinhighereducation:Applicationsandcases(pp.1-25).Hershey,PA:InformationSciencePublishing.

*Acee,T.W.,Kim,H.,Kim,H.l.,Kim,1.,Hsiang-Ning,R.C,Kim,M.,Cho,Y.,Wicker,F.W.,&TheBoredomResearchGroup(2010).Academicboredominunder-andover-challengingsituations.ContemporaryEducationalPsychology,35,17-27.doi:10.1016/j.cedpsych.2009.08.002

Appleton,J.J.,Christenson,S.L.,&Furlong,M.J.(2008).Studentengagementwithschool:Criticalconceptualandmethodologicalissuesoftheconstruct.PsychologyintheSchools,45,369-386.doi:10.1002/pits.20303

*Appleton,J.J.,Christenson,S.L.,Kim,D.,&Reschly,A.L.(2006).Measuringcognitiveandpsychologicalengagement:ValidationoftheStudentEngagementInstrument.JournalofSchoolPsychology,44,427-445.doi:10.1016/j.jsp.2006.04.002

Astin,A.W.(1999).Studentinvolvement:Adevelopmentaltheoryforhighereducation.JournalofCollegeStudentDevelopment,40,518-529.

Axelson,R.D.,&Flick,A.(2010).Definingstudentengagement.Change:TheMagazineofHigherLearning,43,38-43.doi:10.1080/00091383.2011.533096

Bouta,H.,Retalis,S.,&Paraskeva,F.(2012).Utilizingacollaborativemacro-scripttoenhancestudentengagement:Amixedmethodstudyina3Dvirtualenvironment.Computers&Education,58,501-517.doi:10.1016/j.compedu.2011.08.031

Bye,D.,Pushkar,D.,&Conway,M.(2007).Motivation,interest,andpositiveaffectintraditionalandnontraditionalundergraduatestudents.AdultEducationQuarterly,57,141-158.doi:10.1177/0741713606294235

Carini,R.M.,Kuh,G.D.,&Klein,S.P.(2006).Studentengagementandstudentlearning:Testingthelinkages.ResearchinHigherEducation,47,1-36.doi:10.1007/s11162-005-8150-9

Christopher,A.N.(2015).Selectingtherightscale:Aneditor’sperspective.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

ChurchillJr.,G.A.(1979).Aparadigmfordevelopingbettermeasuresofmarketingconstructs.JournalofMarketingResearch,16,64-73.doi:10.2307/3150876

Dougherty,L.M.,Abe,J.,&Izard,C.E.(1996).Differentialemotionstheoryandemotionaldevelopmentinadulthoodandlaterlife.InC.Magai&S.H.McFadden(Eds.),Handbookofemotion,adultdevelopment,andaging(pp.27-41).SanDiego,CA:AcademicPress.

*Duckworth,A.L.,Peterson,C.,Matthews,M.D.,&Kelly,D.R.(2007).Grit:Perseveranceandpassionforlong-termgoals.JournalofPersonalityandSocialPsychology,92,1087–1101.doi:10.1037/0022-3514.92.6.1087

*Duckworth,A.L.,&Quinn,P.D.(2009).DevelopmentandvalidationoftheShortGritScale(Grit-S).JournalofPersonalityAssessment,91,166–174.doi:10.1080/00223890802634290

Ewell,P.T.(2002).AnAnalysisofRelationshipsbetweenNSSEandSelectedStudentLearningOutcomesMeasuresforSeniorsAttendingPublicinstitutionsinSouthDakota,NationalCenterforHigherEducationManagementSystems,Boulder,CO.

98

Finn,J.D.(1993).Schoolengagementandstudentsatrisk.Washington,D.C.:NationalCenterforEducationalStatistics.

Finn,J.D.(1989).Withdrawingfromschool.ReviewofEducationalResearch,59,117-143.doi:10.3102/00346543059002117

Finn,J.D.,&Rock,D.A.(1997).Academicsuccessamongstudentsatriskforschoolfailure.JournalofAppliedPsychology,82,221-234.doi:90/10.1037/0021-9010.82.2.221

Fredricks,J.A.,Blumenfeld,P.C.,&Paris,A.H.(2004).Schoolengagement:Potentialoftheconcept,stateoftheevidence.ReviewofEducationalResearch,74,59-109.doi:10.3102/00346543074001059

Fredericks,J.A.,&McColskey,W.(2012).Themeasurementofstudentengagement:Acomparativeanalysisofvariousmethodsandstudentself-reportinstruments.InS.L.Christenson,A.L.Reschly,&C.Wylie(Eds.),Handbookofresearchonstudentengagement(pp.763–782).NewYork,NY:Springer.

Furlong,M.J.,Whipple,A.D.,St.Jean,G.,Simental,J.,Soliz,A.,&Punthuna,S.(2003).Multiplecontextsofschoolengagement:Movingtowardaunifyingframeworkforeducationalresearchandpractice.TheCaliforniaSchoolPsychologist,8,99-113.doi:10.1007/BF03340899

Gasiewski,J.A.,Eagan,M.K.,Garcia,G.A.,Hurtado,S.,&Chang,M.J.(2012).Fromgatekeepingtoengagement:Amulticontextual,mixedmethodstudyofstudentacademicengagementinintroductorySTEMcourses.ResearchinHigherEducation,53,229-261.doi:10.1007/s11162-011-9247-y

González-Romá,V.,Schaufeli,W.B.,Bakker,A.,&Lloret,S.(2006).Burnoutandengagement:Independentfactorsoroppositepoles?JournalofVocationalBehavior,68,165-174.doi:10.1016/j.jvb.2005.01.003

Guilford,J.P.(1954).Psychometricmethods.NewYork,NY:McGraw–Hill.*Gunuc,S.,&Kuzu,A.(2015).Studentengagementscale:Development,reliability,andvalidity.

AssessmentandEvaluationinHigherEducation,40,587-610.doi:10.1080/02602938.2014.938019

*Handelsman,M.M.,Briggs,W.L.,Sullivan,N.,&Towler,A.(2005).Ameasureofcollegestudentcourseengagement.TheJournalofEducationalResearch,98,184-192.doi:10.3200/JOER.98.3.184-192

*Harackiewicz,J.M.,Barron,K.E.,Carter,S.M.,Lehto,A.T.,&Elliot,A.J.(1997).Predictorsandconsequencesofachievementgoalsinthecollegeclassroom:Maintaininginterestandmakingthegrade.JournalofPersonalityandSocialPsychology,73,1284-1295.doi:90/10.1037/0022-3514.73.6.1284

*Harackiewicz,J.M.,Barron,K.E.,Tauer,J.M.,Carter,S.M.,&Elliot,A.J.(2000).Short-termandlong-termconsequencesofachievementgoals:Predictinginterestandperformanceovertime.JournalofEducationalPsychology,92,316-330.doi:10.1037//0022-0663.92.2.316

Hulleman,C.S.,&Harackiewicz,J.M.(2009).Promotinginterestandperformanceinhighschoolscienceclasses.Science,326,1410-1412.doi:10.1126/science.1177067

Izard,C.E.(1977).Humanemotions.NewYork:Plenum.*Izard,C.E.,Libero,D.Z.,Putman,P.,&Haynes,O.M.(1993).Stabilityofemotionexperiences

andtheirrelationstotraitsofpersonality.JournalofPersonalityandSocialPsychology,

99

64,847-860.doi:10.1037/0022-3514.64.5.847Janosz,M.(2012).Outcomesofengagementandengagementasanoutcome:Someconsensus,

divergences,andunansweredquestions.InS.L.Christenson,A.L.Reschly,&CWylie(Eds.),Handbookofresearchonstudentengagement(pp.695-706).NewYork,NY:Springer.

Jimerson,S.R.,Campos,E.,&Greif,J.L.(2003).Towardanunderstandingofdefinitionsandmeasuresofschoolengagementandrelatedterms.TheCaliforniaSchoolPsychologist,8,7-27.doi:10.1007/BF03340893

Junco,R.(2012).TherelationshipbetweenfrequencyofFacebookuse,participationinFacebookactivities,andstudentengagement.Computers&Education,58,162-171.doi:10.1016/j.compedu.2011.08.004

Junco,R.,Heiberger,G.,&Loken,E.(2011).TheeffectofTwitteroncollegestudentengagementandgrades.JournalofComputerAssistedLearning,27,119-132.doi:10.1111/j.1365-2729.2010.00387.x

*Kuh,G.D.(2001).Assessingwhatreallymatterstostudentlearninginsidethenationalsurveyofstudentengagement.Change:TheMagazineofHigherLearning,33,10-17.doi:10.1080/00091380109601795

Kuh,G.D.(2003).Whatwe’relearningaboutstudentengagementfromtheNSSE:Benchmarksforeffectiveeducationalpractices.Change:TheMagazineofHigherLearning,35,24-32.doi:10.1080/00091380309604090

Kuh,G.D.,Cruce,T.M.,Shoup,R.,Kinzie,J.,&Gonyea,R.M.(2008).Unmaskingtheeffectsofstudentengagementonfirst-yeargradesandpersistence.TheJournalofHigherEducation,79,540-563.doi:10.1353/jhe.0.0019

Langley,D.(2006).Thestudentengagementindex:Aproposedstudentratingsystembasedonthenationalbenchmarksofeffectiveeducationalpractice.UniversityofMinnesota:CenterforTeachingandLearningServices.

Marks,H.M.(2000).Studentengagementininstructionalactivity:Patternsintheelementary,middle,andhighschoolyears.AmericanEducationalResearchJournal,37,153-184.doi:10.3102/00028312037001153

Mauno,S.,Kinnunen,U.,&Ruokolainen,M.(2007).Jobdemandsandresourcesasantecedentsofworkengagement:Alongitudinalstudy.JournalofVocationalBehavior,70,149-171.doi:10.1016/j.jvb.2006.09.002

Mikulas,W.L.,&Vodanovich,S.J.(1993).Theessenceofboredom.ThePsychologicalRecord,43,3-12.doi:10.3389/fpsyg.2014.01245

Miller,R.L.,Rycek,R.F.,&Fritson,K.(2011).Theeffectsofhighimpactlearningexperiencesonstudentengagement.Procedia-SocialandBehavioralSciences,15,53-59.doi:10.1016/j.sbspro.2011.03.050

Newmann,F.M.(1992).StudentengagementandachievementinAmericansecondaryschools.NewYork,NY:TeachersCollegePress.doi:10.5860/CHOICE.30-3945

Newmann,F.M.,Wehlage,G.G.,&Lamborn,S.D.(1992).Thesignificanceandsourcesofstudentengagement.InF.M.Newman(Ed.),StudentengagementandachievementinAmericansecondaryschools(pp.11–39).NewYork,NY:Teacher’sCollegePress.doi:10.5860/CHOICE.30-3945

Pascarella,E.T.,&Terenzini,P.T.(2005).Howcollegeaffectsstudents:Athirddecadeof

100

research(Vol.2).SanFrancisco:Jossey-Bass.Peden,B.F.,&Domask,B.Z.(2011).Dopodcastsengageandeducatecollegestudents?InR.L.

Miller,E.Amsel,B.M.Kowalewski,B.C.Beins,K.D.Keith,&B.F.Peden(Eds.),PromotingStudentEngagement(pp.170-177).SocietyfortheTeachingofPsychology(e-book).

Pekrun,R.,Goetz,T.,Daniels,L.M.,Stupnisky,R.H.,&Perry,R.P.(2010).Boredominachievementsettings:Control-valueantecedentsandperformanceoutcomesofaneglectedemotion.JournalofEducationalPsychology,102,531–549.doi:10.1037/a0019243

Pekrun,R.,Goetz,T.,&Frenzel,A.C.(2005).AchievementEmotionsQuestionnaire—Mathematics(AEQ-M):User’smanual.Unpublishedmanual,UniversityofMunich,DepartmentofPsychology.

Pekrun,R.,Goetz,T.,&Perry,R.P.(2005).AcademicEmotionsQuestionnaire(AEQ):User’smanual.Munich,Germany:UniversityofMunich,DepartmentofPsychology.doi:10.1016/j.cedpsych.2010.10.002

*Pekrun,R.,Goetz,T.,Titz,W.,&Perry,R.P.(2002).Academicemotionsinstudents'self-regulatedlearningandachievement:Aprogramofqualitativeandquantitativeresearch.EducationalPsychologist,37,91-105.doi:10.1207/S15326985EP3702_4

Pekrun,R.,Hall,N.C.,Goetz,T.,&Perry,R.P.(2014).Boredomandacademicachievement:Testingamodelofreciprocalcausation.JournalofEducationalPsychology,106,696-710.doi:10.1037/a0036006696

Preszler,R.W.,Dawe,A.,Shuster,C.B.,&Shuster,M.(2007).Assessmentoftheeffectsofstudentresponsesystemsonstudentlearningandattitudesoverabroadrangeofbiologycourses.CBE-LifeSciencesEducation,6,29-41.doi:10.1187/cbe.06-09-0190

Reschly,A.L.,&Christenson,S.L.(2012).Jingle,jangle,andconceptualhaziness:Evolutionandfuturedirectionsoftheengagementconstruct.InS.L.Christenson,A.L.Reschly,&C.Wylie(Eds.),Handbookofresearchonstudentengagement(pp.3-20).NewYork,NY:Springer.

Sadler,P.M.,Sonnert,G.,Hazari,Z.,&Tai,R.(2012).StabilityandvolatilityofSTEMcareerinterestinhighschool:Agenderstudy.ScienceEducation,96,411-427.

Salanova,M.,Schaufeli,W.,Martínez,I.,&Bresó,E.(2010).Howobstaclesandfacilitatorspredictacademicperformance:Themediatingroleofstudyburnoutandengagement.Anxiety,Stress&Coping,23,53-70.doi:10.1080/10615800802609965

Sana,F.,Weston,T.,&Cepeda,N.J.(2013).Laptopmultitaskinghindersclassroomlearningforbothusersandnearbypeers.Computers&Education,62,24-31.doi:10.1016/j.compedu.2012.10.003

*Schaufeli,W.B.,&Bakker,A.B.(2003).UWES–UtrechtWorkEngagementScale:TestManual.Utrecht,TheNetherlands:DepartmentofPsychology,UtrechtUniversity.

Schaufeli,W.B.,Leiter,M.P.,Maslach,C.,&Jackson,S.E.(1996).MaslachBurnoutInventory–GeneralSurvey(MBI-GS).InC.Maslach,S.E.Jackson,&M.P.Leiter(Eds.),MBImanual(3rded.),(pp.22–26).PaloAlto,CA:ConsultingPsychologistsPress.

Schaufeli,W.B.,Martínez,I.,Marques-Pinto,A.,Salanova,M.,&Bakker,A.(2002).Burnoutandengagementinuniversitystudents:Across-nationalstudy.JournalofCross-CulturalPsychology,33,464-481.doi:10.1177/0022022102033005003

101

*Schaufeli,W.B.,Salanova,M.,González-Romá,V.,&Bakker,A.(2002).Themeasurementofburnoutandengagement:Aconfirmatoryfactoranalyticapproach.JournalofHappinessStudies,3,71-92.doi:10.1023/A:1015630930326

Schutte,N.,Toppinen,S.,Kalimo,R.,&Schaufeli,W.(2000).ThefactorialvalidityoftheMaslachBurnoutInventory‐GeneralSurvey(MBI‐GS)acrossoccupationalgroupsandnations.JournalofOccupationalandOrganizationalPsychology,73,53-66.doi:10.1348/096317900166877

Shernoff,D.J.(2013).OptimalLearningEnvironmentstoPromoteStudentEngagement.NewYork,NY:Springer.

Skinner,E.A.,Wellborn,J.G.,&Connell,J.P.(1990).WhatittakestodowellinschoolandwhetherI’vegotit:Aprocessmodelofperceivedcontrolandchildren’sengagementandachievementinschool.JournalofEducationalPsychology,82,22-32.doi:0022-0663/90/S00.75

Svanum,S.,&Bigatti,S.M.(2009).Academiccourseengagementduringonesemesterforecastscollegesuccess:Engagedstudentsaremorelikelytoearnadegree,doitfaster,anddoitbetter.JournalofCollegeStudentDevelopment,50,120-132.doi:10.1353/csd.0.0055

Taneja,A.,Fiore,V.,&Fischer,B.(2015).Cyber-slackingintheclassroom:Potentialfordigitaldistractioninthenewage.Computers&Education,82,141-151.doi:10.1016/j.compedu.2014.11.009

Troisi,J.D.(2014).Makingthegradeandstayingengaged:Theinfluenceofstudentmanagementteamsonstudentclassroomoutcomes.TeachingofPsychology,41,99-103.doi:10.1177/0098628314530337

102

Chapter9:MeasuringService-LearningandCivicEngagementLoriSimons

WidenerUniversity

Duringthepastthreedecades,thedistributionofacademic-basedservice-learningcourseshasexpandedinundergraduateliberalartsprogramsinpublicandprivatepost-secondaryinstitutions(Giles&Eyler,2013;Marxen,2003).Institutionsofhighereducationhaveincorporatedacademic-basedservice-learningcoursesinliberalartscurriculaasawaytoteachundergraduatestudentstothinkcriticallyabouttheconditionsthatleadtosocialandracialdisparitiesinthecommunityandtodevelopintoresponsiblecitizens(Quaye&Harper,2007).Academic-basedservice-learning(ABSL)isapedagogicalapproachthatrequiresstudentstoconnectthecoursecontenttotheservicecontextthroughapplication,reflectionanddiscussion(Eyler&Giles,1999). InvestigationsonABSLhaveassessedtheimpactofservice-learningonstudentlearning,communitypartnerships,andfacultyengagement.Infact,themajorityofthesestudieshavefocusedonthebenefitsfromABSLonstudentlearningoutcomes(i.e.,criticalthinking,civicengagement).Forexample,studentswhoparticipatedinABSLreportedadeeperunderstandingofthecoursecontentcomparedtothosestudentswhodidnotparticipateinABSL(Litke,2002,Strage,2000).StudieshavealsodocumentedtheimpactsofABSLonthedevelopmentofleadershipattributes(Moely,McFarland,Miron,Mercer,&Ilustre,2002;Vogelesgang&Astin,2000),interpersonalskills(Eyler,2000;Moore,2000),diversityattitudes(Boyle-Baise&Kilbane,2000;Rockquemore&Shaffer,2000),andsocialresponsibilityandcivicengagement(Reinke,2003). Muchofthisresearchindicatesthatcivicengagementisaservice-learningoutcome(Eyler&Giles,1999).Civicengagementabroadtermwithmultipledefinitions(Hatcher,2010).Civicengagementactivitiesaredesignedtopromotesocially-responsibleleadershipinstudentsbyhavingthemworkwithcommunityrecipientstosolveproblems.Examplesoftheseactivitiesincludevolunteeringinasoupkitchenorwritingalettertoanelectedofficial(APA,2015).Inaddition,civicengagementoftenreferstocivicprofessionalism,socialresponsibility,andcommunityengagement(Hatcher,2010;Steinberg,Hatcher,&Bringle,2011).Academicendeavorsrangingfromservice-learning,internships,andotherformsofexperientiallearningareusedtoinstillthevaluesandcharacteristicsassociatedwithcivicengagement(AssociationforExperientialEducation,2011). ThescholarshiponABSLhasevaluatedtheimpactofservice-learningfromoneoftworesearchmethodologies.Quantitativemethodshaveassessedstudentattitudesbeforeandafterservicewithsingle,convenientsamples.Withthismethod,itisnotpossibletodetectwhetherthereportedchangesareattributedtotheserviceexperiencebecausethestudieslackacomparisongroup(Payne,2000;Reinke,2003;Root,Callahan,&Sepanski,2002)andmeasureattitudesinsteadofskillswitheithersingle-itemsurveys(Rockquemore&Schaffer,2000),reflectiveessays(Green,2001)orethnographictechniques(Boyle-Base&Kilbane,2000).Surveysarealsousedtomeasurechangesinstudentskillsfromthebeginningtotheendofthe

103

course.Pretestandposttestsurveysareusuallyadministeredtostudentsinacoursewithaservicecomponentandcomparedtothoseinacoursewithoutaservicecomponenttomeasurechangesbeforeandafterservice.Thispre-postmethodologymayglossoverwhatactuallyhappenswithintheserviceexperience,becauseitdoesnotmeasurechangesordetectprocessesthatoccurwhileengagedinservice(Boyle-Baise,2002;Wang&Jackson,2005). Incontrast,qualitativemethodshaveidentifiedtheprocessesassociatedwiththedevelopmentofculturalawareness(Rockquemore&Shaffer,2000;Rootetal.,2002)andsocial,communityandcivicresponsibility(Battistoni,2002;Eyler,2009).QualitativeresearchhasbeencriticizedforrelyingonsmallsamplesofWhite,middle-classstudentsandanalytictechniques(i.e.,ethnographic,focusgroups)thatmakereplicationdifficult(Boyle-Baise,2002).Theuniquenessoftheservicecontextfurtherlimitstheresearchers’abilitytogeneralizethefindings(Simonsetal.,2012).Itisimportanttorecognizethatthereisatrade-offbetweenexternalandinternalvalidityinassessmentmethodsofservice-learning(Wang,Kelly,&Hritsuk,2003).However,oneareathathasreceivedsubstantiallylessattentionintheliteratureisservice-learningandcivicengagementmeasures(i.e.,scales).Althoughscholarshavedevelopedmeasurestoassessservice-learningimpacts,mostofthemhaveconstructedsingleitemstoassessstudentlearning.Infact,Bringle,Phillips,andHudson(2004)suggestthatthedevelopmentandimplementationofmultiple-itemmeasuresarenecessaryifthefieldofservice-learningistoadvance.Theuseofreliableandvalidmeasurestoassessserviceimpacts(i.e.,civicengagement)maybethenextstepintheadvancementofthisareaofscholarship.Thepurposeofthischapteristodescribeservice-learningandcivicengagementmeasuresanddistinguishbetweenthosewithandwithoutpsychometricevidence.Method

SearchStrategyAcomprehensivesearchofservice-learningmeasureswasconductedusingelectronicdatabasessearchsuchasEbscohostandERIC.Keywordsincludingservice-learning,civicengagement,socialresponsibility,civicattitudes,knowledgeandskills,andcommunityengagementsurveys,measures,instruments,andscaleswereusedinthissearch.Thesearchledtoasmallnumberofscholarlypublications.AbroadersearchwasconductedusingGoogleScholar.Thissearchidentifieddifferentpublicationsources,includingconferenceproceedings,books,andjournals(i.e.,TeachingandLearning,HigherEducation).Fewpeer-revieworscholarlyjournalsweredevotedsolelytoservice-learningandcivicengagement.ThepublicationsourcesidentifiedfromtheGooglesearchwascomparedtothoseidentifiedontheNationalService-LearningClearinghouse(NSLC)website(2015).TheNSLCwebsitelisted11sources,butonlysevenofthemincludeapeerreviewprocessforpublication.Thepeerreviewprocessdiffersandcanbemorerigorousforjournalsthanforconferenceproceedingsoreditedvolumes.Tolimitthescopeofthereviewprocess,onlyarticleswerereviewedthatwerepublishedintheMichiganJournalofCommunityService-Learning(MJCSL).TheMJCSLisapremierjournalintheareaofservice-learningandcommunityengagement.Thegoalsofthisjournalaretoencouragescholarshiponcommunityengagement,contributetotheintellectual

104

vigorandacademiclegitimacyofservice-learningandcommunity-engagedscholarship,andexpandtheknowledgebaseonacademicservice-learning(Howard,n.d.).

SampleTheMJCSLpublished274articlesfromfall1994tospring2015.Eacharticlewasexaminedusinganopencodingprocedure(Creswell,1994&1998)todetermineifthearticleincludedatleastoneofthefollowingcomponents:(a)researchorpedagogicalstudy;(b)studentorprogramoutcomes;(c)quantitativedatacollectionmethods;or(d)items,scalesorothertoolstomeasureservice-learningorcivicengagement.Afinalsetof67articleswasusedintheanalysesasshowninAppendixA.

ArticleCodingData(i.e.,67articles)underwentanitemanalysisthroughwhichpatternswereidentifiedandcodedusingtheconstantcomparativemethod(Creswell,2005).Opencodingconsistedofcategorizingthemeasuresaccordingservice-learningconstructs(Eyler&Giles,1999)asshowninTable1.Axialcodingconsistedofsystematicallyanalyzingthemeasuresaccordingtomeasurementconceptstodetermineiftheconstructhadacceptablepsychometriccharacteristics(i.e.,reliability)(Bringleetal.,2004;Cohen&Swererdlik,2009).Thirty-eightarticlesreportedonstudiesthatusedmeasureswithpsychometricevidence.Selectivecodingconsistedofanalyzingthedata(i.e.,38articles)accordingtoscaleconstructionandadvancedpsychometrictheories,asalsoshowninAppendixA(Cohen&Swererdlik,2009;Furr,2010).Finally,sevenarticleswithredundantmeasures,singleitems,orstandardizedmeasuresnotdesignedspecificallytoassessservice-learningwereremovedfromthefinalanalysis.Table1CategoriesofService-LearningandCivicEngagementMeasures Service-LearningCategories n CivicAttitudes 4CivicMindedness 1CommunityPreferences 4CourseEvaluations 5CommunityImpacts 2CulturalAwareness(Intercultural,Diversity) 8Education(Tutoring,AttitudestowardLearning) 4Engagement 5Integrity 1InterpersonalSkills 3Leadership 3Motivation 9 MoralDevelopment 5

105

Perceptions 7Problem-Solving 1Satisfaction 10Self-Esteem(Self-Efficacy,Competence) 8ServiceImpacts(Career) 4SocialJustice 4SocialResponsibility 3Stereotypes(Racism,Ageism) 5 StudentLearning 10Tension 1Volunteering(Helping) 3PopulationthatthemeasurewasintendedStudents 62Alumni 2Faculty 5CommunityPartners 5Institutions(Administration) 1 *Note.nreferstothenumberofitems,scales,andmeasuresreportedintheMJCSLfrom1994-2015.Somemeasuresassessmorethanoneconstructorareusedwithmorethanonegroup.ResultsTwenty-fourpercentofthearticlespublishedinMJCSLbetween1994and2015includedquantitativemeasuresasdatacollectionmethods.Outofthe67articlesthatmettheinclusioncriteria,38ofthemincludedpsychometricevidenceaboutthemeasure.Ofthesearticles,25ofthemreportedonmeasuresdesignedtoassessservice-learning.AsshowninTable2,21measuresdesignedtoassessserviceimpactswithsupportingpsychometricevidencederivedfrom18articlesrepresentedthefinalset.Table2MeasuresUsedtoAssessService-LearningandCivicEngagementMeasure Scale Response

FormatPsychometricEvidenceReportedintheArticle

ScaleofService-LearningInvolvement

Single-item4subscales

5-pointLikertscale

Modesttostrong

PersonalSocialValuesandCivicAttitudes

Single-item2subscales

4-point,Likert-typescale;5-pointLikertscale

Modesttostrong

CommunityServiceself-Efficacy Single-item 10-point Fairtostrong

106

Scale ratingCBOSupervisorRatingsofCbSLStudentPerformanceEvaluations

Single-item 5-pointratingscale

Modesttostrong

CommunityServiceInvolvementPreferenceInventory

Single-item4subscales

5-pointLikertscale

Fairtomodest

StudentLearningfromCommunityService

Single-item(usedwithothermeasures)

Index(meanscores)

Fair

FeedbackDispositionInstrument

Single-item(usedwithothermeasures)

5-pointratingscale

Modest

GoalOrientedInstrument Single-item(usedwithothermeasures)

9-pointratingscale

Strong

CivicAttitudesandSkillsQuestionnaire

Multi-item6scales

5-pointLikertscale

Fairtomodest

CourseSatisfactionMeasures Multi-item4subscales

5-pointLikertscale

Modesttostrong

ThreeAspectsofEngagement Multi-item4subscales

5-pointLikertscale

Strong

Integrity(thisscalewaspartofalargerquestionnaire)

Single-item2subscales

6-pointLikertscale

Fair

AssessmentofCommunityAgencyPerceptions

Multi-item4subscales

5-pointInterview

FairtoStrong

TheCommunityServicePreferenceScale

Single-itemTwo-scales

5-pointLikertScale

Strong

TheQualityoftheLearningEnvironment

Single-item 5-pointratingScale

Strong

Measure Scale ResponseFormat

PsychometricEvidenceReportedintheArticle

Web-BasedFacultyService-LearningBeliefsInventory

Multi-item4subscales

5-pointLikertScale

Fairtostrong

TheCivicMindedGraduateScale

Multi-item10subscales

6-pointLikertScale

Strong

CivicAttitudesMeasuresValuingCommunityEngagementCulturalAwarenessSeeksKnowledgeaboutPoliticalSocialIssuesKnowledgeofNewOrleans

Multi-item6subscales

5-pointLikertScale

Modesttostrong

107

CultureandIssuesKnowledgeofCurrentEventsCulturalSkillsCommunityEngagedCompetenciesScale

Single-item 5-pointratingscale

Strong

Service-LearningCourseQuality Single-Item 5-pointratingScale

Modesttostrong

TheCommunityImpactScale Multi-item8subscales

6pointratingscale;Interview

Modesttostrong

Note.Psychometricevidencereflectsinternalconsistencyestimates.Weak=<.5,fair=.51<.69,modest=.7<.79,andstrong=.80<.MeasuresofServiceImpactsonCommunity

AssessmentofCommunityAgencyPerceptionsInterviewDevelopedbyMironandMoely(2006),thisinstrumentmeasurescommunitypartnerperceptionsonfourinterviewscales:(a)AgencyVoicegathersinformationaboutthesupervisor’sinvolvementinplanningserviceactivities(i.e.,towhatextentdoyoufeelthatyouragencyandtheUniversitywereequalpartners);(b)AgencyBenefitassessestheperceivedbenefittotheagencyworkingwiththeservice-learningprogram(i.e.,howweretheagencyneedsmetbythestudent);(c)InterpersonalRelationsassessesdifferencesbetweentheagencymembersandthestudents(i.e.,doyoufeelthestudentenjoyedworkingwithothersofadifferentrace,socialclass,orculture);and(d)PerceptionoftheUniversityidentifiesthesupervisor’sviewoftheUniversity(i.e.,istheUniversitysensitivetotheneedsofthesurroundingcommunity).Cronbachalphacoefficientsrangefrom.66to.77,andinter-correlationsforthefourscalesrangefrom.33to.36.

CommunityImpactScaleDevelopedbySrinivas,Meenan,Drogin,andDePrince(2015)tomeasurethepotentialimpactofschool-communitypartnershipsoncommunitypartners,this46-iteminstrumentyieldsscoresonaneightsubscales:(a)OverallExperiences(i.e.,thecommunity-schoolpartnershipwassuccessful);(b)SocialCapital(i.e.,accesstomentorsand/orfutureemployers);(c)SkillsandCompetencies(i.e.,abilitytoworkaspartofateam);(d)MotivationsandCommitments(i.e.,commitmenttoengagingcommunities);(e)PersonalGrowthandSelf-concept(i.e.,compassionandcaringforothers);(f)Knowledge(i.e.,knowledgeaboutrelevantsocialissues);(g)OrganizationalOperations(i.e.,workloadanddemandsonyourtime);and(i)OrganizationalResources(i.e.,finances).Cronbachalphacoefficientsrangefrom.70to.94.Anitemanalysiswasconductedbetweensurveyandinterviewmethods.

108

Community-BasedOrganizationsSupervisorRatingScaleDevelopedbyFerrariandWorrall(2000),thisnine-itemsurveyassessesstudentfieldwork.Communitysupervisorsevaluatestudents’fieldworkontwosubscales,Service(i.e.,appearance,sitesensitivity)andWorkSkills(i.e.,attendance,punctuality).TheCronbach’salphacoefficientforServiceSkillsis.91andforWorkSkillsis.87.Contentvaliditywasassessedbetweensupervisors’writtencommentsandsurveyratings.

MeasuresofFacultyPerceptionsofService

CommunityEngagedScholarship(CES)DevelopedbyJameson,Jaeger,Clayton,andBringle(2012),this25-itemself-reportmeasureassessesfacultyknowledgeofandskillsinconducingcommunity-engagedscholarship(i.e.,skillsforfosteringcommunityandsocialchange,abilitytocollaboratewithcommunitymembersincommunitycapacitybuildingendeavors).Cronbachalphacoefficientsrangefrom.90to.95.

Web-BasedFacultyService-LearningBeliefsInventory(wFSLBI)DevelopedbyHou(2010)toassessfacultyperceptionsofthebenefitsandbarriersassociatedwithcommunity-basedservice-learning,thismeasureyieldsscoresonfoursubscales:(a)PROS-CLS(BenefitsClassroom)(i.e.,service-learningenrichesclassroomdiscussionsinmycourse);(b)PROS-COM(BenefitCommunity)(i.e.,theservicemystudentscompletedwasbeneficialtothecommunity);(c)CONS-CLS(BarriersClassroom)(i.e.,timeconstraintsinterferewithmyabilitytoteachaservice-learningcourse);and(d)CONS-INST(BarriersInstitution)(i.e.,facultypromotionandtenurepoliciesdonotsupportmyservice).Cronbachalphacoefficientsrangefrom.65to.91.Confirmatoryfactoryanalysis,itemdiscriminantvalidity,andgroupcomparisonswerealsoconductedtovalidatethemeasure.

MeasuresofServiceImpactsonStudents

CivicAttitudesMeasuresDevelopedbyMoelyandIllustre(2011),thisinstrumentassessesaspectsofcivicattitudesandengagementonsixindividualscales:(a)TheValuingofCommunityEngagementandService(i.e.,Ienjoyengagingincommunityservice);(b)CulturalAwareness(i.e.,Ithinkitisimportantforapersontothinkabouthis/herracialidentity);(c)SeeksKnowledgeaboutPolitical/SocietalIssues(i.e.,Thereisnopointinpayingattentiontonationalpolitics);(d)KnowledgeofNewOrleansCultureandIssues(i.e.,IamabletodescribecommunitiesofcolorintheNewOrleansarea);(e)KnowledgeofCurrentEvents(i.e.,Iamwellinformedaboutcurrentnewsevents);and(f)CulturalSkills(i.e.,Ifinditdifficulttorelatetopeoplefromadifferentraceorculture).Cronbachalphacoefficientsrangefrom.77to.90.Afactoranalysiswasconductedwithothermeasuresofcivicattitudesandresponsibility.

109

CivicAttitudesandSkillsQuestionnaire(CASQ)DevelopedbyMoely,Mercer,Ilustre,Miron,andMcFarland(2002)tomeasurecivicattitudesandskills,this84-itemself-reportquestionnaireyieldsscoresonsixscales:(a)CivicAction(i.e.,futureserviceintentions);(b)InterpersonalandProblem-solvingSkills(i.e.,workcooperativelywithotherstosolveproblems);(c)PoliticalAwareness(i.e.,awarenessoflocalandnationalevents);(d)LeadershipSkills(i.e.,abilitytolead);(e)SocialJusticeAttitudes(i.e.,attitudestowardpovertyandsocialproblemscanbesolved);and(f)DiversityAttitudes(i.e.,attitudestowarddiversityandtheirinterestsininteractinginculturallydifferentpeople).Cronbachalphacoefficientsrangefrom.69to.88andtest-retestreliabilitiesrangefrom.56to.81.Afactoranalysisandinter-correlationsofthesubscaleswereconducted.ConstructvaliditywasalsoconductedwiththeModernRacismScale(McConahay&Hough,1976).

CivicMindedGraduate(CMG)ScaleDevelopedbySteinberg,Hatcher,andBringle(2011)toassesscivic-mindedness(i.e.,adispositiontobeinvolvedinthecommunityorsenseofsocialresponsibility)incollegegraduates,this30-itemself-reportmeasureyieldsscoreon10subscales:(a)Knowledge-Volunteer(i.e.,Iknowtherearealotofopportunitiestobecomeinvolvedinthecommunity);(b)Knowledge-Academic(i.e.,IamconfidentIcanapplywhatIlearnedinmyclasses);(c)Knowledge-SocialIssues(i.e.,Iampreparedtowritealettertothenewspaperaboutacommunityproblem);(d)Skills-Listening(i.e.,Iamagoodlistener);(e)Skills-Diversity(i.e.,IprefertoworkinsettingsinwhichIworkwithpeoplewhodifferfromme);(f)Skills-ConsensusBuilding(i.e.,Otherstudentscandescribemeasapersonwhocandiscusscontroversialissues);(g)Dispositions-CommunityEngagement(i.e.,Iliketobeinvolvedincommunityissues);(h)Dispositions-Self-Efficacy(i.e.,Icancontributetoimprovinglifeinmycommunity);(i)Dispositions-SocialTrusteeofKnowledge(i.e.,IwantacareerinwhichIcanimprovesociety);and(j)BehavioralIntentions(i.e.,IplantostaycurrentwithlocalandnationalnewsafterIgraduate).Cronbachalphacoefficientsrangefrom.85to97andtest-retestreliabilitiesrangefrom.43to.62.Aprincipalcomponentfactoryanalysisandinter-itemcorrelationswereconductedtofurtherassessreliability.Contentvaliditywasassessedwithinterviewsandconvergentvaliditywasassessedwithintegrityandsocialdesirabilityscales.

CommunityServiceSelf-EfficacyScale(CSSES)DevelopedbyReeb,Katsuyama,Sammon,andYoder(1998),this10-itemquestionnaireassessesstudentefficacyforparticipatingincommunityservice(i.e.,ifIchoosetoparticipateincommunityserviceinthefuture,Iwillmakeameaningfulcontribution).Cronbach’salphacoefficientis.92,inter-itemcorrelationsrangefrom.65to.78,andtest-retestreliabilityis.62.ConstructvaliditywasassessedwiththeSocialResponsibilityInventory(Bringleetal.,2004;Reebetal.,1998).

CommunityServiceInvolvementPreferenceInventory(CSIPI)DevelopedbyPayne(2000)toassessstudentpreferenceforbecominginvolvedincommunityservice,this48-iteminventoryyieldsscoresonfourpreferences:(a)ExplorationInvolvement

110

Preference(i.e.,commitmenttoshorttermservicethatisconvenientforthehelper);(b)AffiliationInvolvementPreference(i.e.,commitmenttendstobeinfrequentandshorterinduration);(c)ExperimentationInvolvementPreference(i.e.,desiretomakeadifferenceinthelivesofothersandtolearnmoreaboutthecommunity);and(d)AssimilationInvolvementPreference(i.e.,careerandlifestyledecisionsbasedonservicetobearesponsiblecitizen).Cronbachalphacoefficientsrangefrom.63to.74.

CommunityServicePreferenceScaleDevelopedbyMoely,Furco,andReed(2008)andadaptedfromitemscreatedbyMoelyandMiron(2005)toassessstudentpreferencesfortypicalserviceactivities,this16-itemself-reportquestionnaireyieldsscoresonfourscales:(a)CharityOrientedExperience(i.e.,helpingthoseinneed);(b)SocialChange-OrientedExperience(i.e.,changingpublicpolicyforthebenefitofpeople);(c)CharityOrientation(i.e.,aserviceplacementwhereyoucanreallybecomeinvolvedinhelpingothers);and(d)SocialChangeOrientation(i.e.,aserviceplacementwhereyoucancontributetosocialchangethataffectsusall).Cronbachalphacoefficientsrangefrom.83to.90.(Moely&Illustre,2014).

CourseSatisfactionMeasuresDevelopedbyMoely,McFarland,Miron,Mercer,andIlustre(2002),thisinstrumentassessesstudentviewsoftheircoursesonfoursubscales,including;(a)CourseValue(i.e.,howusefulwasthematerialcoveredinclass);(b)LearningaboutAcademicField(i.e.,applicationofthecourseconcepts,interestinthefield,);(c)LearningabouttheCommunity(i.e.,workingwithotherseffectivelyandseeingsocialproblemsinanewway);and(d)ContributiontotheCommunity(i.e.,howusefulwereserviceactivities).Cronbachalphacoefficientsrangefrom.74to.82(Moely,etal.,2002).

ItemsUsedtoMeasureIntegrityDevelopedbyBringle,Hatcher,andMcintosh(2006),theseitemsassessintegritycomponents(i.e.,whenIaminvolvedinservice,Ifocusonmeetingtheimmediateneed)basedonMorton’sconceptofintegrityinstudentsinvolvedincommunityservice.Factoranalyseswithvarimaxrotationwereconductedonthe10items.Sevenofthe10factorsloadedontotwofactors,identityandlong-termcommitments.TheCronbachalphacoefficientforintegrityis.67andforlong-termcommitmentsis.66.

PersonalSocialValuesScaleDevelopedbyMarby(1998),thisnine-itemquestionnaireyieldsscoresontwosubscales:(a)PersonalSocialValues(i.e.,helpingotherswithdifficulty);and(b)CivicAttitudes(i.e.,adultsshouldgivesometimeforthegoodoftheircommunity).Cronbachalphacoefficientsrangefrom.63to.81.AfactoranalysiswasconductedwiththeCivicAttitudessubscaleandacademicbenefitquestionsestablishingasingle-factorofcivicattitudes(Bringle,etal.,2004;Marby,1998).

111

ScaleofService-LearningInvolvement(SSLI)DevelopedbyOlneyandGrande(1995),this60-itemquestionnaireyieldsscoresonthreesubscales:(a)Exploration(i.e.,initialreasonsforvolunteering);(b)Realization(i.e.,continualreasonsforvolunteering);and(c)Internalization(i.e.,volunteeringtohelpsolvesocialproblems).Cronbachalphacoefficientsrangefrom.70to.84.ConvergentanddivergentvaliditywasestablishedwiththeIntellectualDevelopmentScaleandtheMoralDevelopmentMeasure,respectively.

Service-LearningCourseQualityDevelopedbyMoelyandIlustre(2014)andderivedfromanearlierversiondevelopedbyFurcoandMoely(2006)toassessstudentviewsofservice-learningcoursesintermsofhavingattributesthatareconsideredimportant.This12-itemself-reportmeasureassessesservice-learningattributesonthreesubscales:(a)ValueofService(i.e.,Ifeelthatmyservice-learningactivitywasworthwhile);(b)FocusonService(i.e.,thecommunityorganizationinwhichIworkedwasreadytoreceiveservice-learningstudents);and(c)OpportunitiesforReflection(i.e.,Ihadopportunitiestoreflectonmyservice-learningthroughdiscussionswithfaculty,studentsandcommunitymembers.Cronbach’salphacoefficientforthe12-itemsis.93andinternalconsistencyforeachscalerangesfrom.72to.90.

StudentLearningfromCommunityServiceInstrumentDevelopedbySubramony(2000),this10-itemself-reportmeasureassessesstudents’perceivedeffectivenessinmeetingservice-learninggoals.ThismeasureisusedinconjunctionwiththeFeedbackDispositionInstrumentandtheGoalOrientedInstrument.TheFeedbackDispositionInstrumentmeasuresstudentpropensitytoseekoravoidfeedback,andtheGoalOrientedInstrumentmeasuresstudentsaslearningorperformancegoal-oriented.Cronbachalphacoefficientsrangefrom.68to.81.

ThreeAspectsofEngagementDevelopedbyGalliniandMoely(2003),this27-itemself-reportquestionnaireyieldsscoresonthreescales:(a)CommunityEngagement(i.e.,feelingconnectedtothecommunity);(b)AcademicEngagement(i.e.,satisfactionwiththeacademiccourseanduniversity);and(c)InterpersonalEngagement(i.e.,thecourse'sinfluenceonstudents’abilitytoworkwithotherseffectively).Cronbachalphacoefficientsrangefrom.85to.98.

QualityofLearningEnvironmentDevelopedbyBringle,Hatcher,andMuthiah(2010),this24-itemself-reportquestionnairemeasuresstudentperceptionsoflearningenvironmentcomponents,includingtheextenttowhichstudentsexperiencepeerorfacultyinteraction,coursesatisfaction,perceivedlearning,activelearning,andpersonalrelevancecomponentsthatcontributetohighqualitylearningenvironments(i.e.,Ihavedevelopedasignificantrelationshipwithatleastonestudentinthis

112

class).Itemsareaddedtogethertoproduceacompositeindex.Thealphacoefficientforthecompositeindexis.89. DiscussionEducatorsproposethatservice-learninghasapositiveeffectonstudentlearning(Eyler,&Giles,1999).PedagogicalscholarshipandresearchstudiespublishedintheMJCSLhaveevolvedfromcourseassessmentstolongitudinal,cross-sectionalinvestigations.Datacollectionmethodologiesusedtoassessservice-learninghavealsobecomemoresophisticatedovertime,despitethefactthatfewresearchersevaluateserviceimpactswithquantitativemeasures.Theinstrumentsusedtomeasureservice-learningimpactssuchassocialresponsibilityandcivicengagementhavealsoevolvedfromsingleitemstomulti-itemscales.Theresultsfromthecurrentstudysuggestthatcivicengagementisdirectlyandindirectlymeasuredasaservice-learningoutcome.Verylittleresearchmeasuredcivicengagementdirectly.Civicengagementistypicallymeasuredwithscalesthatwereconstructedtoassesscivicattitudes,socialresponsibility,andcommunityengagement.Civicengagementisalsomeasuredwithitemsdesignedtoassessattributesorcharacteristicsofresponsiblecitizenshipsuchasdiversityawarenessandsocialjustice.Diversitywasalsocitedasastudentlearningoutcome.Studentmotivationsforandperceptionsofservicescaleswereusedtomeasureserviceimpacts,butwerecitedlessoftencomparedtoitemsthatwereusedtomeasurestudentlearning(i.e.,questionsthatassesscourseobjectives)andcoursesatisfaction(i.e.,studentsatisfactionwithcourseandfield).Educatorsinterestedinmeasuringcivicengagementasastudentlearningoutcomeshouldbeginwithconstructingorrefiningexistingcourseobjectives.Facultywillneedtodecidewhattheywantstudentstolearnbytheendofthecourse.One’sdisciplinewillinfluencethisdecisionmakingprocess.Forexample,apoliticalscienceprofessormaywantstudentstodemonstrateresponsiblecitizenship;while,apsychologyprofessormaywantthemtodemonstrateculturalcompetence.Thenfacultywillneedtomakesurethatthecourseobjectivesarealignedwiththecourseoutcomes(i.e.,studentlearning).Courseobjectivesshouldalsobeexplicit.Inotherwords,ifdemonstratingculturalcompetenceisanoutcomethenlearningaboutculturaldiversityshouldbeanexplicitcourseobjective(i.e.,compareandcontractracial-ethnic-culturalidentitymodels).Facultywillneedtoensurethatboththecoursecontentandserviceactivitiesarealignedwithcourseobjectivesandoutcomes.Forinstance,writingalettertoanelectedofficialmayincreasecivicresponsibilitybutnotculturalcompetence.Oncefacultyalignthecoursecontent,objectivesandoutcomes,theywillneedtofocusonassessmentofstudentlearning.Table2listsservice-learningandcivicengagementmeasures.FacultyshoulddecidewhichoutcomestheywanttomeasureandthenreviewTable2toidentifyexistingservice-learningandcivicengagementmeasures.Facultyshouldselectservice-learningorcivicengagementmeasuresthatalignwiththeircourseoutcomestodemonstratestudentlearning.Inaddition,facultymaywanttoseekothermeasuresthatalignwiththecourseoutcomes.Psychologicalmeasuresmaybehelpfulinassessingserviceimpactsonstudentlearning;therefore,facultymaywanttoconductasearchtoobtaintheparticularmeasure(s).Facultyshoulddevelopamulti-itemsurveythatincludesservice-learning,civic

113

engagementandpsychologicalmeasuresthatassessstudentlearning.Facultymayalsowanttoincludecoursesatisfactionmeasuresinthesurvey.CoursesatisfactionmeasuresarealsoincludedinTable2.Finally,facultymaywanttouseamulti-methodapproachusingsurveysandreflectionssotheycanidentifychangesinstudentsbefore,during,andafterservice.Themeasuresusedtoassessservice-learningimpactsonstudentlearningoutcomesdifferedintermsoftheconstructs(i.e.,civicattitudes),measurements(single-itemquestionsvs.scales)andpsychometricevidence(i.e.,reliability,validity).Whilesomeresearchersusedstandardizedinstrumentsorpsychologicalmeasures,othersdevelopedsingleitemorscaleswithandwithoutsuchevidence.Veryfewscalesdesignedtomeasureservicelearninghadacceptablepsychometricdata.Outofthe21service-learningmeasuresthathadsupportingpsychometricevidence,onlynineofthemwerefurtherassessedwithadvancedpsychometrictechniques(i.e.,confirmatoryfactoryanalysis).Therefore,researchersandpractitionerswhouseservice-learningmeasurestoevaluatepedagogicalimpactsmaywanttoalsoconductpsychometricanalysesonthespecificmeasuretocontrolformeasurementerrorandtobeassuredthattheyaremeasuringwhattheyintended(seeLehan&Hussey’s(2015)primeronscaledevelopmentvalidationinthise-book).Additionalservice-learningmeasuresshouldalsobeconstructedandvalidatedusingadvancedmeasurementtheoriesandpsychometricperspectives.Thecurrentstudyexpandstheknowledge-basedonmeasuresusedtoassessservice-learningandcivicengagement.Thisisoneofthefirststudiestosystematicallyanalyzeservice-learningandcivicengagementmeasures.Althoughthisstudyonlyanalyzedquantitativemeasures,qualitativemethodsofinquiryarecomplimentaryandshouldbeexamined.Qualitativeandquantitativedatacollectionmethodsshouldalsobeusedtomeasureserviceimpacts,butonlywhencivicengagementisanexplicitcourseobjective.Greateralignmentbetweencourseobjectivesandoutcomesmaybethefirststepinmeasuringcivicengagementasaservice-learningoutcome.

114

ReferencesReferencesmarkedwithanasteriskindicateascale.AmericanPsychologicalAssociation.RetrievedJuly25,2015from

http://www.apa.org/education/undergrad/civic-engagement.aspxAssociationforExperientialEducation.(2011).Howitworksinhighereducation.Retrieved

October2,2011fromhttps://web.archive.org/web/20110113103425/http://www.aee.org/applications/highered/

Battistoni,R.M.(2002).Whatisgoodcitizenship?Conceptualframeworksacrossdisciplines.Introductiontoservice-learningtoolkit,2ndedition(pp.179-186).Providence,RI:CampusCompact.

Boyle-Baise.M.(2002).Multiculturalservicelearning.NewYork:TeachersCollegePress.Boyle-Baise,M.,&Kilbane,J.(2000).Whatreallyhappens?Alookinsideservice-learningfor

multiculturalteachereducation.MichiganJournalofCommunityServiceLearning,7,54-64.

*Bringle,R.G.,Hatcher,J.A.,&Muthiah,R.N.(2010).Theroleofservice-learningontheretentionoffirst-yearstudentstosecondyear.MichiganJournalofCommunityServiceLearning,16(2),38-49.

*Bringle,R.G.,Hatcher,J.A.,&McIntosh,R.E.(2006).AnalyzingMorton’stypologyofserviceparadigmsandintegrity.MichiganJournalofCommunityServiceLearning,13(1),5-15.

Bringle,R.G.,Phillips,M.A.,&Hudson,M.(2004).TheMeasureofservice-learning.Washington,DC:AmericanPsychologicalPress.

Cohen.R.J.,&Swerdlik,J.(2009).Psychologicaltestingandassessment:Anintroductiontotestsandmeasurement,7thedition.NewYork:McGraw-Hill.

Creswell,J.W.(2005).Educationalresearch,2ndedition.UpperSaddleRiver,NJ:PearsonPrenticeHall.

Creswell,J.W.(1998).Qualitativeinquiryandresearchdesignchoosingamongfivetraditions.ThousandOaks,CA:Sage.

Creswell,J.W.(1994).ResearchDesign:Qualitative&QuantitativeApproaches.ThousandOaks,CA:Sage.

Eyler,J.(2009).Thepowerofexperientialeducation.LiberalEducation,Fall,24-31.Eyler,J.S.(2000).Whatdowemostneedtoknowabouttheimpactofservice-learningon

studentlearning?MichiganJournalofCommunityServiceLearning,SpecialIssue,11-17.Eyler,J.S.,&Giles,D.E.(1999).Where’sthelearninginservice-learning?SanFrancisco:Jossey-

Bass.*Ferrari,J.R.,&Worrall,L.(2000).Assessmentsbycommunityagencies:How“theotherside”

seesservice-learning.MichiganJournalofCommunityServiceLearning,7,35-40.Furr,R.M.(2010).Scalecontractionandpsychometricforsocialandpersonalitypsychology.Los

Angeles:Sage.Furco,A.&Moely,B.E.(2006,April).Acomparativeanalysisoftheimpactsofservice-learning

onstudents.PaperpresentedattheannualmeetingoftheAmericanEducationalResearchAssociation,SanFrancisco,CA

*Gallini,S.M.,&Moely,B.E.(2003).Service-learningandengagement,academicchallenge,andretention.MichiganJournalofCommunityServiceLearning,10(1),1-14.

115

Giles,D.E.,&Eyler,J.(2013).Theendlessquestforscholarlyrespectabilityinservice-learningresearch.MichiganJournalofCommunityServiceLearning,20(1),53-64.

Green,A.E.(2001).“Butyouaren’twhite:”Racialperspectivesandservicelearning.MichiganJournalofCommunityServiceLearning,8(1),18-26.

Hatcher,J.A.(2010).Definingthecatchphrase:understandingcivicengagementofcollegestudents.MichiganJournalofCommunityServiceLearning,16(2),95-100.

*Hou,S-I.(2010).Developingafacultyinventorymeasuringperceivedservice-learningbenefits andbarriers.MichiganJournalofCommunityServiceLearning,16(2),78-89.

Howard,J.(Ed)(n.d.).MichiganJournalofCommunityServiceLearning,RetrievedJune6,2015fromhttp://www.unich.ed/~mjcsl/index.html

*Jameson,J.K.,Clayton,P.H.,Jaeger,A.J.,&Bringle,R.G.(2012).Investigatingfacultylearninginthecontextofcommunity-engagedscholarship.MichiganJournalofCommunityServiceLearning,18(2),40-55.

Hussey,H.D.,&Lehan,T.J.(2015).Aprimeronscaledevelopment.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Litke,R.A.(2002).Doallstudents“getit?””Comparingstudents’reflectionstocourseperfomance.MichiganJournalofCommunityServiceLearning,8(2),27-34.

*Mabry.J.B.(1998).Pedagogicalvariationsinservice-learningandstudentoutcomes.Howtime,contact,andreflectionmatter.MichiganJournalofCommunityServiceLearning,5,32-47.

Marxen,C.E.(2003).MeetingNCATEstandardthroughservice-learning:Diversity.AmericanAssociationofCollegeforTeacherEducation,IssueBrief3,Winter,1-4.

McConhay,&Hough,J.Jr.(1976).Symbolicracism.JournalofSocialIssues,32(2),23-45.doi:10.1111/j.1540-4560.1976.tb02493.x

*Miron,D.,&Moely,B.E.(2006).Communityagencyvoiceandbenefitinservicelearning.MichiganJournalofCommunityService,12(2),27-37

*Moely,B.E.,&Ilustre,V.(2014).Theimpactofservice-learningcoursecharacteristicsonuniversitystudents’learningoutcomes.MichiganJournalofCommunityServiceLearning,21(1),5-16.

*Moely,B.E.,&Ilustre,V.(2011).Universitystudents’viewsofpublicservicegraduationrequirement.MichiganJournalofCommunityServiceLearning,17(2),43-58.

*Moely,B.E.,Furco,A.,&Reed,J.(2008).Charityandsocialchange:Theimpactofindividualpreferencesonservice-learningoutcomes.MichiganJournalofCommunityServiceLearning,15(1),37-48.

*Moely,B.E.,McFarland,M.Miron,D.,Mercer,D.,&Illustre,V(2002).Changesincollegestudents’attitudesandintentionsforcivicinvolvementasafunctionofservice-learningexperiences.MichiganJournalofCommunityServiceLearning,9(1),18-26.

*Moely,B.E.,Mercer,S.H.,Ilustre,V.,Miron,D.,&McFarland,M.(2002).Psychometricpropertiesandcorrelatesofthecivicattitudesandskillsquestionnaire(CASQ):Ameasureofstudent’sattitudesrelatedtoservice-learning.MichiganJournalofCommunityServiceLearning,8(2),15-26.

Moely,B.E.,&Miron,D.(2005).Collegestudents’preferredapproachestocommunityservice:Charityandsocialchangeparadigms.InS.Root,J.Callahan,andS.H.Billig(Eds.),

116

Improvingservice-learningpractice:Researchonmodelstoenhanceimpacts.Greenwich:CT:InformationAgePublishing.

Moore,D.T.(2000).Therelationshipbetweenexperimentallearningresearchandservice-learningresearch.MichiganJournalofCommunityServiceLearning,SpecialIssue,124-128.

NationalService-LearningClearinghouse.RetriedJune6,2015fromhttps://gsn.nylc.org/clearinghouse#sthash.1RCNI3Kh.dpuf

*Olney,C.,&Grande,S.Validationofascaletomeasuredevelopmentofsocialresponsibility.MichiganJournalofCommunityServiceLearning,2,43-53.

*Payne,C.A.(2000).Changesininvolvementpreferencesasmeasuredbythecommunityserviceinvolvementpreferenceinventory.MichiganJournalofCommunityServiceLearning,7,41-53.

Quaye,S.J.,&Harper,S.R.(2007).Facultyaccountabilityforculturallyinclusivepedagogyandcurricula.LiberalEducation,32-39.

*Reeb,R.N.,Katsuyama,R.M.,Sammon,J.A.,&Yoder,D.S.(1998).Thecommunityserviceself-efficacyscale:Evidenceofreliability,constructvalidity,andpragmaticutility.MichiganJournalofCommunityServiceLearning,5,48-57.

Reinke,S.J.(2003).Makingadifference:Doesservice-learningpromotecivicengagementinMPAstudents?JournalofPublicAffairsEducation,9(2),129-137.

Root,S.,Callahan,J.,&Sepanski,J.(2002).Buildingteachingdispositionsandservice-learningpractice:Amulti-sitestudy.MichiganJournalofCommunityServiceLearning,8(2),50-59.

Rockquemore,K.A.,&Schaffer,R.H.(2000).Towardatheoryofengagement:Acognitivemappingofservicelearning.MichiganJournalofCommunityServiceLearning,7,14-23.

Simons,L.,Fehr,L.,Blank,N.,Connell,H.,Fernandez,D.,Georganas,D.,Padro,J.,&Peterson,V.(2012).Lessonslearnedfromexperientiallearning:Whatdostudentslearnfromapracticumandinternship?InternationalJournalofTeachingandLearninginHigherEducation,24(3),325-334.

*Srinivas,T.,Meenan,C.E.,Drogin,E.,&DePrince,A.P.(2015).Developmentofthecommunityimpactscalemeasuringcommunityorganizationperceptionsofpartnershipbenefitsandcosts.MichiganJournalofCommunityServiceLearning,21(2),5-21.

*Steinberg,K.S.,Hatcher,J.A.,&Bringle,R.G.(2011).Civic-mindedgraduate:Anorthstar.MichiganJournalofCommunityServiceLearning,18(1),19-33.

Strage,A.A.(2000).Service-learning:Enhancinglearningoutcomesinacollege-levellecturecourse.MichiganJournalofCommunityServiceLearning,7,5-13.

*Subramony,M.Therelationshipbetweenperformancefeedbackandservice-learning.MichiganJournalofCommunityServiceLearning,7,46-53.

Vogelgesang,L.J.,&Astin,A.W.(2000).Comparingtheeffectsofcommunityserviceandservicelearning.MichiganJournalofCommunityServiceLearning,7,25-34.

Wang,O.T.,Kelly,J.M.,&Hritsuk,B.(2003).Doesonesizefitall?:Challengeofsocialcognitivedevelopment.MichiganJournalofCommunityServiceLearning,9(2),5-14.

Wang,Y.&Jackson,G.(2005).Formsanddimensionsofcivicinvolvement.MichiganJournalofCommunityServiceLearning,11(2),39-48.

117

AppendixA:ResearchMeasuresforMeasuringService-LearningandCivicEngagementArticleAuthor(s) Publication

DateDescription

Hammond,C. 1994 -24itemsthatmeasurefacultymotivationandsatisfactiononthreesubscales:1.Personal;2.Co-curricula,and3.Curricularmotivations

Miller,J. 1994 -11itemsthatinquiresaboutstudentopinionsofplacementsite,personalexperience,andacademicexperiencerelatedtoservice

Hesser,G. 1995 -Facultyratetheimpactofserviceon11LiberalArtsLearningoutcomes(i.e.,writtencommunicationskills)

Olney,C.&Grande,S.

1995 -¹,³TheScaleofService-Learningisa60-itemscalethatmeasuresdevelopmentalphasesoftheservice-learningmodel-²TheDefiningIssuesTestisanobjectivemeasureofmoralreasoningasdefinedbyKohlberg’stheory-²TheScaleofIntellectualDevelopmentisa102-itemthatmeasuresintellectualanddevelopmentalstagesofDualism,Relativism,andCommitment-²MeasuresofMoralOrientationassessestwoorientationstowardmoraldecisionmakingasdescribedbyGilligan

Greene,D.&Diehm,G.

1995 -6itemsthatmeasurethedegreetowhichstudentsstereotypeolderadultnursinghomeresidents

Myers-Lipton,S. 1995 -²TheModernRacismScalemeasuresracismbasedonthetheoryofmodernorsymbolicracism

Kendrick,J.R. 1996 -15itemsthatmeasuresocialresponsibilityandpersonalefficacy;-9itemsthatmeasurevaluesorbeliefsfromhelpingothers

Hudson,W.E. 1996 -18itemsthatmeasurepublicpolicybeliefs;9itemsthatmeasurecourseimpact

Eyler,J.,Giles,D.E.,&Braxton,J.

1996 -²TheNationalStudyofService-LearningImpactsonStudentsincludescitizenshipconfidencescales,citizenshipvalues,citizenshipscales,andperceptionsofsocialjustice.ThissurveywasdevelopedaspartoftheMeasuringCitizenshipProjectatRutgersUniversity

Miller,J. 1997 -Twoquestionsthatmeasurestudents’perceptionofpowerWade,R.C.,&Yarbrough,D.B.

1997 -PreceptorteachersurveymeasurestheirlevelofsatisfactionwiththestudentteacherandtheUniversity;-Studentteachersurveymeasureshissatisfactionwiththepreceptorteacherandserviceexperience

Osborne,R.E.,Hammerich,S.,Hensley,C.

1997 -²Asurveyofmeasuresusedtoassessself-esteeminstudents.ThesemeasuresincludedtheRosenbergSelf-esteemscale,theCognitiveComplexityScale,theTexasSocialBehaviorInventory,theSpontaneousSelf-Conceptmeasure,andtheRemoteAssociationsTest

Mabry,B. 1998 -¹,³22itemsthatmeasurepersonalsocialvalues,civicattitudes,perceivedcourseimpactoncivicattitudes,andperceivedacademicbenefitofservice-learning

118

Reeb,R.N.,Katsuyama,R.M.,Sammon,J.A.,Yoder,D.S.

1998 -¹,³TheCommunityServiceSelf-efficacyScaleisa10-itemself-reportmeasuresthatassessesstudents’confidenceinengaginginservice;-²TheSocialResponsibilityInventoryisa15-itemmeasureassessesstudentresponsibilityforsolvingsocialproblems

Korfmacher,K.S. 1999 -10itemsthatmeasurealumnisatisfactionwiththeEnvironmentalScienceServicesProgramatBrownUniversity

Vernon,A.,Ward,K.

1999 -Asurveyofcommunitypartnerstomeasuretheirsatisfactionwiththeprogram

Rockquemore,K.A.,Schaffer,R.H.

2000 -26itemsthatmeasureservice-impactsonservice-learners

Veogelgesang,L.J.,Astin,A.W.

2000 -Itemsmeasuredthedegreeofcommitmenttopromotingracialunderstandingandactivism,academicskills,leadership,andfutureserviceweredrawnfromthelongitudinalCollegeStudentSurvey

Ferrari,J.R.,Worrall,L.

2000 -¹,³Communitybasedorganizationsupervisorratingofcommunitybasedservice-learniners

Payne,C.A. 2000 -¹ThedescriptionoftheCommunityServiceInvolvementPreferenceInventory(CSIPI);Asurveythatmeasuresstudents’levelofengagementinserviceonfoursubscales

Subramony,M. 2000 -¹TheStudentLearningfromCommunityServiceInstrumentisa10-itemmeasurethatassessesservice-learners’perceivedeffectivenessinmeetingservicegoals;-¹FeedbackDispositionmeasuresstudentpropensitytoseekandavoidfeedback;-¹GoalOrientedInstrumentmeasuresthedegreetowhichstudentsarelearningorperformancegoaloriented;-²AgencyFeedbackInstrumentmeasurestheamountoffeedbackthatwasgiventostudents

Jeffers,C.A. 2000 -34itemsmeasuringstudentattitudestowardartandlearningofartintheclassroomandgalleries

Abes,E.S.,Jackson,G.,Jones,S.R.

2002

-Asurveythatmeasuresfactorsthatmotivateanddeterfacultyuseofservice-learning

Moely,B.E.,McFarland,M.,Miron,D.,Mercer,S.H.,Ilustre,V.

2002 -TheCivicAttitudesandSkillsQuestionnaire(CASQ)measuresstudentcivicattitudesandskillsonsixsubscales;-¹Coursesatisfactionmeasuresstudentsatisfactionwiththecourseandcommunityon4subscales.

Moely,B.E.,Mercer,S.H.,Ilustre,V.,Miron,D.,McFarland,M.

2002 -¹,³TheCivicAttitudesandSkillsQuestionnaire(CASQ)measuresstudentcivicattitudesandskillsonsixsubscales.Psychometricevidenceisprovided.

*Schmidt,A., 2002 -¹71-itemsmeasuringtutoringsatisfaction;

119

Robby,M.A. -CASQ-SAT/9Scores-¹Childandteachevaluations

Prins,E.S. 2002 -Asurveymeasuringcommunitycolleges’purpose,serviceinitiatives,andreasonsforengaginginservice-learning

*Root,S.,Callahan,J.,Sepanski,J.

2002 -²Questionsmeasuringteacherefficacy,commitmenttoteaching,attitudestowarddiversity,serviceethic,andcommitmenttofutureservice;-¹Aspectsofservice-learningexperiencescalemeasuresstudentviewsofservice

Sperling,R.,Wang,V.O.,Kelly,J.M.,Hritsuk,B.

2003 -Anattributionalquestionnairemeasuringstudents’reasonforengaginginserviceandviewsoftheAmericanEducationalsysteminrelationtosocialinequity

*Evangelopous,N.,Sidorova,A.,Riolli,L.

2003 -¹Asurveywereusedtomeasurestudentattitudesandviewsofusefulnessofbusinessstatistics.

Gallini,S.M.,Moely,B.E.

2003 -¹TheThreeAspectsofEngagementmeasurescommunity,academic,andinterpersonalengagement;-¹Itemsmeasuringacademicchallengeandretentioninstudents

Marchel,C.A. 2003 -²Itemsmeasuringsocioculturalawareness,careerplansandfutureservice

Brody,S.M.,&Wright,S.C.

2004 -²VolunteerFunctionInventory;-²Self-ExpansionQuestionnaire

Hatcher,J.A.,Bringle,R.G.,Muthiah,R.

2004 -¹Aquestionnairecontainingitemsmeasuringactivelearning,coursesatisfaction,facultyinteraction,peerinteraction,perceivedlearning,andpersonalrelevance,qualitiesofservice-learningclass,academiccontent,andreflection-¹TheQualityoftheLearningEnvironmentmeasuresstudentviewsoftheacademiccontentandtheservicecontext

Fensel,L.M.,&Peyrot,M.

2005 -²Asurveymeasuringthequalityofserviceasanundergraduate,currentcommunityservice,service-relatedjob,andpersonalresponsibility;ItemsfromtheHigherEducationResearchInstituteCollegeStudentSurvey;SocialandPersonalResponsibilityScaleinalumniatcommunitycolleges

Wang,Y.,Jackson,G.

2006 -²TheStudentService-LearningCourseSurveymeasurersstudentperceptionsofcivicinvolvement,socialjusticeandcharitablereasonsforcivicinvolvementanddimensionsofCivicInvolvement

Bainger,N.,Bartholomew,K.

2006 -13itemsmeasuredorganizations’motivestoengageinservice-learningandtheirsatisfactionwithservice-learners

Miron,D.,Moely,B.E.

2006 -¹Aninterviewmeasuringcommunitypartnerperceptionsonfoursubscales

Bringle,R.G.,Hatcher,J.A.,

2006 -¹Aquestionnairecontainingitemsmeasuringintrinsicandextrinsicmotives,leadership,preferencefordifferenttypesofcommunity

120

McIntosh,R.E. service,andintegrity;-¹,³9-itemswereusedtomeasureintegrity.

Schnaubelt,T.,Stratham,A.

2007 -Fiveitemsassessedfacultyperceptionsofserviceasimportantattheinstitutionandconsideredinthetenureandpromotionprocess

Banerjee,M.,Hausafas,C.O.

2007 -Asurveymeasuringfacultyperceptionsofservice-learningasateachingstrategy

Moely,B.E.,Furco,A.,Reed,J.

2008 -¹TheCommunityServicePreferenceScalemeasuresstudents’preferenceforservicethatemphasizescharityorsocialchangeactivities;-²HES-LSquestionnairemeasuringcivicattitude,responsibilityandcareerdevelopment;-CASQ-SubscalesfromtheThreeAspectsofEngagementScale

Bernacki,M.,L.,Jaeger,E.

2008 -²TheDefiningIssuesTestmeasuresmoralreasoningaccordingtoKohlberg;-²TheMoralJustificationScaleisusedtomeasuremoralorientation;-²TheService-LearningOutcomeScalemeasuresstudentperceptionsofhowtheircoursework,understandingsocialproblems,problemsolvingability,leadership,efficacy,andpassion.

Clayton,P.H.,Bringle,R.G.,Senor,B.,Huq,J.,Morrison,M.

2010 -TheTransformationalRelationshipEvaluationScaleassessestherelationshipbetweencommunitypartnersandUniversityresearchers

Bringle,R.G.,Hatcher,J.A.,McIntosh,R.E.

2010 -¹QualityoftheLearningEnvironmentis24-itemmeasuringcoursesatisfaction,facultyinteraction,peerinteraction,perceivedlearning,activelearning,andpersonalrelevance

Barney,S.T.,Corseer,G.C.,White,L.H.

2010 -²TheCommunityAttitudestoMentalIllnessScaleisa40-itemmeasureassessingattitudestowardsthosewithamentalillness;

Hou,S-I. 2010 -¹,³TheWeb-basedFacultyService-LearningBeliefsInventoryassessesfacultyviewsofthebenefitsfromandbarrierstoimplementingserviceonfoursubscales

Seider,S.C.,Gillmor,S.C.,Rabinowicz,S.A.

2010 -²TheProtestantEthicMeasurestudentbeliefsthatindividualhardworkleadstosuccess.

Bowman,N.,Brandenberger,J.W.,Mick,C.S.,Smedley,C.T.

2010 -²SituationAttributionsforPovertyScalemeasuresbeliefsaboutpoverty;-²OpennesstoDiversityitemsmeasurethedegreetowhichstudentsareinterestedinlearningaboutthosewhodifferfromthem;-²ResponsibilityforImprovingSocietyassesshowmuchpersonalresponsibilityoneperceivesfortakingactiontohelpothers;-²EmpowermentViewofHelpingmeasuresbeliefsaboutwhetherpeoplecanovercometheirproblemswithassistancefromothers;

121

-²BeliefinaJustWorldmeasuresthebeliefsthatgoodthingshappentogoodpeople;-²SocialDominanceOrientationmeasurespreferencesforandacceptanceofinequalityacrosssocialgroups;-²Self-generatingViewofHelpingmeasuresthebeliefsthatindividualsareonlyabletohelpthemselvesovercometheirproblems

Steinberg,K.S.,Hatcher,J.A.,Bringle,R.G.

2011 -¹,³TheCivicMindedGraduateScaleisa30-itemself-reportmeasuringcivic-mindednesson10subscales

Moely,B.E.,Ilustre,V.

2011 -¹,³TheCivicAttitudeMeasuresassessedstudentattitudes,knowledgeandskillsforcommunityengagement;-CASQ

Mills,S.D. 2012 -TheTensionSurveyasksstudentsandcommunitypartnerstoindicatetheextenttowhichtheyhaveexperiencedanyofthebrieflydescribedtension

Jameson,J.K.,Jaeger,A.J.,Clayton,P.H.,Bringle,R.G.

2012 -¹TheCommunityEngagedScholarshipCompetenciesScaleisdesignedtoassessnewfacultyviewsofandcomfortwithimplementingcommunityengagedscholarship

Neihaus,E.,Crain,L.K.

2013 -²TheNationalSurveyofAlternativeBreakmeasuresserviceengagement,communityengagement,community/staffinteraction,studentlearning,emotionalandphysicalchallenged,socialissues,reflection,journalingandorientation

Moely,B.E.,Ilustre,V.

2013 -¹TheService-LearningCourseQualityIndicatorsmeasurescharacteristicsofhighqualityservice-learning;-CASQ;CivicAttitudes,KnowledgeandSkills

Soria,K.M.,Thomas-Card,T.

2014 -²TheStudentExperienceintheResearchUniversitySurveycontains600itemsthatmeasurestudentsatisfaction,academicengagement,campusclimate,researchexperiences,andcivic/communityengagement

122

*Moely,B.E.,Ilustre,V

2014 -TheService-LearningCourseQualityIndicators;-PreferencesforCharity-andSocialChange-orientedService;-CourseSatisfactionMeasures:LearningabouttheCommunity,AcademicLearning,SatisfactionwiththeUniversity.

DeLeon,N. 2014 -²TheCulturalQuotientScaleisa20-itemmeasurethatassessesculturalintelligence,knowledge,strategy,actionanddrive;-²TheInterculturalsensitivityScalesisa24-itemmeasurethatassessesinterculturalcommunicationandcompetence.

Srinivas,T.,Meenan,C.E.,Drogin,E.,DePrince,A.P.

2015 -¹TheCommunityImpactScalemeasurestheimpactofserviceoncommunitypartners.

*Russell-Stamp,M.

2015 -Web-basedFacultyService-LearningBeliefsInventorymeasuresfacultyperceptionsofthebenefitsandbarriersassociatedwithservice.

*Mitchell,T.D.,Richard,F.D.,Battistoni,R.M.,Rost-Banik,C.Netz,R.Zakoske,C.

2015 -TheCivicMindedProfessionalscalemeasuresthedegreetowhichalumniareinvolvedinservice-relatedworkorcareers.

Note.¹Denotesscaleswithpsychometricevidencethatwerespecificallydesignedtomeasureserviceimpacts,²denotesscaleswithpsychometricevidencethatwerenotspecificallydesignedtomeasureserviceimpacts,³denotesscaleswithadvancedpsychometricdata,and*indicateseitherarticleswithredundantmeasuresorsingleitemmeasures(i.e.,onequestion)thatwereexcludedfromthefinalanalysis.

123

Chapter10:MeasuringIndividualDifferencesinEpistemologicalBeliefsKellyL.Y.Ku

HongKongBaptistUniversity

Isknowledgenon-changingandabsolute,orisiteverevolvingandtentative?Studentsinthesameclassroommayholddifferentviewsofknowledge.Suchviewsmayaffecthowtheyaccept,evaluate,andinterpretwhatisbeinglearned.Understandinghowlearnersviewknowledgeisnowfundamentaltoeducation,becausetherehasbeenarisingbodyofevidenceontheimportantroleofmatureepistemologicalbeliefstowardsacademicandintellectualperformances(seeforexample,Inglis&Mejia-Ramos,2009;Ku,Lai&Hau,2014;Kuhn&Park2005;Qian&Alvermann,1995;Rodríguez&Cano,2006;Schommer-Aikins&Hutter,2002).Derivedandsubsumedunderthebroadphilosophicalstudyofepistemology(thestudyofwhatknowledgeis),thetermepistemologicalbeliefisusedtodescribeanindividual’srepresentationofthenatureofknowledge.SincetheoriginalworkofPerry(1970),differentteamsofresearchershaveputforthsomewhatdifferentemphasesinconceptualizingandmeasuringepistemologicalbeliefs.Twomajorapproacheshaveemerged;theseincludethedevelopmentalapproachrepresentedbytheworkofKuhn’s(seeKuhn&Park,2005;Kuhn,1991)andKingandKitchener’s(1994),aswellasthemulti-facetmodelassignifiedbySchommer’s(1990,2004)work.Thischapterbeginswithanoutlineonthetheoreticalframeworksofepistemologicalbeliefs.Itgoesontoreviewalistofinstrumentsgroundedinthediscussedframeworksandcloseswithissuesandsuggestionsinexaminingepistemologicalbeliefs. TheoreticalConceptionsofEpistemologicalBeliefsEpistemologyreferstothebroadphilosophicaltheoryofknowledge.Psychologistsandeducatorsadoptanempiricalapproachinstudyingpersonalepistemologyontheotherhand,aimedatassessingindividuals’subjectiveunderstandingofwhatknowledgeislike.Itisgenerallyheldthatindividualswithamorematureunderstandingofknowledgetendtoviewknowledgeasindependentlyconstructedandcomplexinitsstructure,andrealizethatconclusionstoanyproblemmightbetentativeinnature.Theserepresentationsarecalledepistemologicalbeliefs,orepistemicbeliefs.ExistingresearchonepistemologicalbeliefsismostlyderivedfromthelongitudinalstudiesofPerryconductedinthe70s.Itwasobservedthatcollegestudentsmovedthroughasequenceofepistemicdevelopmenttypicallyfromtakingadualistic(e.g.onlyoneviewcanbecorrect)toarelative(e.g.bothviewscanbecorrect)stanceofknowledge(Perry,1970).Thisprocessindicatedanintellectualgrowthfromasimpleandabsoluteunderstandingofknowledgetoanintegrativeonewhereconflictingviewsareallowedandvalued.Itwasbelievedthatstudentseventuallygrewtodevelopacommitmenttouseevidenceinjustifyingalternativepointsofviewandthusunderstoodthatnotallviewscarryequalstrength.

124

Theconstructofepistemologicalbeliefsisnotastraightforwardone.Overtheyears,tocontinuetheworkofPerry’s,researchershaveadoptedtwomajorapproaches.Theseincludeexaminingthedevelopmentalsequenceofepistemologicalbeliefsaswellasidentifyingasystemofindependentbeliefsaboutknowledgethatanindividualmightholdsimultaneously. TheDevelopmentalApproachofEpistemologicalBeliefsKuhn(Kuhn&Park,2005;Kuhn,1991)definedalearner’sepistemologicalbeliefsasfallingintooneoffourlevels:realist,absolutist,multiplist,andevaluativist(seeTable1).Intheearliestlevelofrealist,childrenattainknowledgethroughdirect“copying”ofthereality.Forexample,teachersmayobserveachildofage3believingeveryoneexperiencesthesameeventinthesameway.Ifthechildenjoyedabirthdayparty,heorshewouldbelieveeveryoneelseenjoyedtheparty.Whilechildrenbegintoobservetheexistenceofdifferentviewsintheabsolutiststage,theyresolveconflictsofunderstandingthroughsimpleassertionofacorrectviewoverincorrectones.Childrenatthisageoftenargueaboutthingstheydisagreeoninaneither-or-logic(e.g.,“Iamright,thereforeyoumustbewrong”).Adolescents,ontheotherhand,believeeveryonehastherighttoholddifferentopinions.Thischaracterizesthethirdlevel,thusallopinionsareequallycorrectandcriticalthinkingisunnecessary.Inthefinallevel,thematurelearnerrecognizesthatsomeopinionsarebettersupportedbyevidenceandarethereforemorecorrectthanothers.Atthisstage,thelearnerrecognizestheimportanceofcriticalthinking.Children’sdevelopmentofthetheoryofmindunderliesthecoreofthedevelopmentofepistemologicalunderstanding(Kuhn&Park,2005).Thatistosay,children’srepresentationofknowledgeadvancesastheybegintorealizehowtheyandotherscometoknowandreasondifferently.Kuhn’smodelthereforedescribesagenericsequentialdevelopmentaltrendassociatingageandlevelsofepistemologicalunderstanding. Table1DevelopmentofEpistemologicalUnderstandingLevel AgeGroup ConceptofKnowledgeRealist Childrenbefore

age4

Knowledgeisareplicaofexternalreality.

Absolutist Latechildhoodtopre-adolescence

Knowledgeisdiscretepiecesoffactthatareeithercorrectorincorrectreplicaofreality.

Multiplist Adolescence Knowledgeisindividual’sself-chosenopinions,whicharecompetingclaimswithoutdiscrimination.

Evaluativist Adulthood Knowledgeisclaimswithdifferentstrengthsdeterminedbyreasonsandevidence.

SharingsimilarfeatureswithKuhn’sdevelopmentalmodel,butdistinctiveinitsstrongeremphasisonthemeansofjustificationusedbylearners,KingandKitchener(1994)proposedtheReflectiveJudgmentModel(seeTable2).Themodelcomprisesthreelevels:pre-reflective,quasi-reflective,andreflective.Learnersatearlystagesdictatethatjustificationofanopinionis

125

eitherunnecessaryorisreferencedsolelytoanexternalagentofauthority.Atlaterstages,thestrengthofvarioustypesofevidence(e.g.,anecdotes,scientific,historical,etc.)andcriteria(e.g.,meaningfulness,utility,consistence,etc.)areusedtovalidateanopinionintheprocessofjustification.UnlikeKuhn’smodel,KingandKitchener(1994)putanemphasisonepistemologicalgrowthofadolescencetoadulthoodintheirmodel.Thoughwithoutclearagespecification,itisbelievedthatstartingtheperiodofadolescenceallowslearnerstograsptheimportancetouseevidenceinsupportingtheirbeliefsabouttheworld,andthatthevariousmeansofjustificationareacquiredinclinationsthatcanbecultivatedthroughlearningandexperience.Thus,itislooselyassumedforthosewithhighereducationbackgroundstooperateatmoreadvancedepistemologicalstages. Table2StageofReflectiveJudgmentinEpistemologicalUnderstandingStage ConceptofKnowledge RoleofJustificationPre-reflective

Knowledgeisfirstviewedasbasedonpersonalexperience,andlaterasobtainedfromexternalauthorities.

Justificationiseitherunnecessaryorismadewithreferencetoanauthority.

Quasi-reflective

Anypersonalclaimsupportedbydifferentreasonscanbeknowledge;itisamatterofaperson’sownperspective.

Relyingfirstonsubjectiveoridiosyncraticreasonsthentousingavarietyofevidence

Reflective

Knowledgeisconstructedandsupportedbyreasonsandevidence,withanaimofarrivingatawell-informedandcompleteunderstandingofanissue.

Differentperspectivesandsourcesofevidencearecomparedandevaluatedforconsistency,meaningfulnessandcoherence.

TheMultiple-FacetApproachofEpistemologicalBeliefsEpistemologicalbeliefsarealsoconceptualizedasasetofpersonaltheoriesaboutvariousfacetsofknowledge.BeginningwithSchommer’s(1990)work,fivefacetswereidentified.Theseincludesimpleknowledge(knowledgeasisolatedversusinterrelated),certainknowledge(knowledgeasstaticversusevolving),omniscientauthority(thesourceofknowledgeaspasseddownfromhigheragentversusthesourceofknowledgeaschallengeable),quicklearning(learningtakesplacequicklyornotatallversuslearningtakesplacegradually),andinnateability(intellectualabilityasfixedentityversusabilityasacquired).Thesimplicityandcertaintyfacetspertaintobeliefsaboutstructureandformofknowledge;whereas,omniscientauthorityandquicklearningconcerntheoriginofknowledgeandprocessofknowing.BasedonSchommer’s(1990)work,Hofer(2000)laterproposedarefinedfour-factormodelofepistemologicalbeliefs.Theyincludecertaintyofknowledge,simplicityofknowledge,sourceofknowledgeandjustificationforknowing(justificationbasedonpersonalopinion,authority,ortheuseofmultipleevidences).Inthisrefinedversion,HoferintegratedKingandKitchener’s(1994)emphasisofjustificationintoSchommer’soriginalmodelandputforththefactor

126

justificationforknowing,whichexaminesaperson’sactiveinclinationinusingreasonsandevidenceinknowledgeconstruction. MeasuringEpistemologicalBeliefsExistinginstrumentsprovidemeasurementsforassessingdevelopmentofbeliefs,genericbeliefs,anddiscipline-orissue-specificbeliefsaboutknowledge.

MeasuringDevelopmentofEpistemologicalBeliefs

EpistemologicalDevelopmentKuhn,Cheney,andWeinstock’s(2000)instrumentconsistsof15forced-choiceitemsrepresentingfivejudgmentdomains(seeTable3).Eachitemdescribesapairofcontrastingclaims.Therespondentneedstodecideifoneclaimisright(i.e.,aresponseoftheabsolutistlevel)orwhetherbothcouldberight(i.e.,aresponseofthemultiplistlevel).Ifthelatterischosen,thelearnerhastodecidewhetheroneclaimcouldbemorerightthantheother(i.e.,aresponseoftheevaluativistlevel). Table3InstrumentofEpistemologicalDevelopment(Kuhnetal.,2000)JudgmentDomain SampleItemAesthetic Robinthinksthefirstpaintingtheylookatisbetter.

Christhinksthesecondpaintingtheylookatisbetter.

Value Robinthinkslyingiswrong.Christhinkslyingispermissibleincertainsituations.

PersonalTaste Robinsayswarmsummerdaysarenicest.Chrissayscoolautumndaysarenicest.

SocialWorld Robinhasoneviewofwhycriminalskeepgoingbacktocrime.Chrishasadifferentviewofwhycriminalskeepgoingbacktocrime.

PhysicalWorld Robinbelievesonebook’sexplanationofwhatatomsaremadeupof.Chrisbelievesanotherbook’sexplanationofwhatatomsaremadeupof.

Thescoringessentiallyincludescountingthedominatingresponseconformingattheabsolutist,multiplist,orevaluativistlevelforeachjudgmentdomain.Itwasreportedthatmorepeoplehavedemonstratedmultiplistthoughtsinthedomainsofpersonaltasteandaesthesticdomains,andevaluatistthoughtsinthedomainsofsocialandphysicalworlds(Kuhnetal.,2000).Convergentvaliditywassupportedbystudiescomparingresultsofthisinstrumentwithothersimilaroneswith70%-80%reportedcompatibilityintermsofidentifiedepistemologicallevels(Kuhnetal.,2000).Thisisastraight-forwardinstrumentthatiseasytounderstandandadminister.Itcanbeadministeredinagroupsettingwithanestimatedcompletiontimeof15minutes.Ithasbeenfoundsuitablefor5th–12thgradersaswellasadultrespondents(Kuhn&

127

Park,2005).Themeritoftheinstrumentalsoincludesprofilingthepatternofaperson’sepistemologicalbeliefsacrossdomainsusingamoreeconomicalmethodcomparedtothetraditionalapproachofinterviewingtheparticipant.Thedatadepictedbythismodelareparticularlyusefulinaddressingthetheoreticalquestionofwhetherpersonalepistemologyisdomain-genericorspecific.Thelimitationofthisinstrumentincludestherelativelylimitedstatisticalanalysisthatcanbedonewiththeforced-choicedatagenerated.Ahola(2009)discussedthepotentialofthisinstrumentandhowqualitativecomponentscanbeintegratedtobettercaptureindividuals’reasoningunderlyingtheirchoices,allowingmoresophisticatedanalysis.

ReasoningAboutCurrentIssuesTest(RCI)TheRCI(KingandKitchener,1994;seeWood,Kitchener,&Jensen,2002fordescriptionandformatofthetest)isatestcreatedbasedontheReflectiveJudgmentModel.Thetestaskstest-takers’opinionsaboutcontroversialauthenticissues(e.g.,artificialsweeteners,federaldebt,globalwarming).Test-takersareaskedtorate,onafive-pointcontinuumfromverydissimilartoverysimilar,howalikehisorheropinionisoneachissuecomparedtotenprovidedstatements.ThestatementsareclaimsaboutjustificationsthatmatchdifferentstagesoftheReflectiveJudgmentModel,suchas“Researchersdisagreebecausetheyarereallystudyingdifferentfacetsoftheissueandthebestwaystoaddressonefacetoftheissuearedifferentthanthebestwaystoaddressotherfacets”and“Researchersarriveatdifferentconclusionsbecausetheevidenceitselfiscomplexandtheyexamineitfromseveralperspectives.Theyarriveatadecisionbysynthesizingtheirknowledge,experiences,andexpertopinions.”Inthelastsection,test-takersselectuptothreestatementsthatbestmatchtheirownthinkingabouttheissue.TheRCIscoreisbasedsolelyonthechoicesmadeinthissection,whereastheratingservesonlytoprobetest-takers’personaljudgmentsonparticularissues(King&Kitchener,2004).Inotherwords,thescoringfocusesnotonassessingfactualknowledgeorcognitiveskillsinmakingreflectivejudgments,buttheassertionsthatapersonholdsabouthowjudgmentsaremade.Theresultsthereforerevealmoreaboutepistemicinclinationthanintellectualability. TheRCIcanbeadministeredonacomputer(seehttp://www.reflectivejudgment.org/index.cfm)andinpaper-and-pencilformat.ThereportedCronbach’salphasoftheRCIrangedfrom.50sto.70s(Duell&Schommer-Aikins,2001;King&Kitchener,2004).Across-sectionalanalysisofover9,000studentsofundergraduate,graduate,andprofessionaleducationprogramsreportedthattheRCIisabletodiscriminatebetweenstudentsacrossschoolingaftercontrollingforco-variants(Woodetal.,2002),confirmingaprogressivedevelopmentaltrendaspredictedbytheReflectiveJudgmentModel.

MeasuringMultipleEpistemologicalBeliefsInstrumentsusingvariationsofSchommer’s(1990)modelarelisted.Allthreeofthemareself-reportquestionnairesinpaper-and-pencilformatutilizingfive-pointLikert-scales(exceptforHofer’s,2004,whichusesa7-pointLikert-scale),withthedegreeofagreementmeasuredreflectingsophisticationofanindividual’sepistemologicalbeliefs.

128

EpistemologicalQuestionnaire(EQ)The62-itemEQ(Schommer,1990)isconstructedbasedonSchommer’sfive-facetmodelofepistemologicalbeliefs.Fourvalidationstudiesutilizingdifferentpopulationsampleswithdifferenteducationalandworkexperienceswereconducted;thereportedinternalconsistencyreliabilityrangedfrom.50sto.70s(Schommer,1993;1998).GroupcomparisonsrevealedsensitivityoftheEQtoage,gender,educationallevel,andculturaldifferences.Adultsandfemalestudentsweremorelikelytobelieveinabilityasacquiredasopposedtopredisposed(Schommer,1993;1998).AsianstudentswerefoundtobeextremelyconsistentinagreeingthatauthoritiesarenottobecriticizedasopposedtothoseoftheWest(Chan&Elliot,2000).TheEQhasreportedasomewhatmixedfactorstructuredespiteitspopularity.Theproposedfive-factorstructurehasnotbeenconsistentlyreplicatedinlaterstudies(Braten&Stromso,2005;Clarebout,Elen,Luyten,&Bamps,2001;Duell&Schommer-Aikins2001;Schraw,Bendixen,&Dunkle,2002;Wood&Kardash,2002).Researcherstendtoconcludethatthelowtomodestinternalconsistencymightbecausedbytheratherambiguousnatureoftheconstructofpersonalepistemologyitself.Additionally,theinconsistencyofrespondents’ratingsofsomeitemscouldreflectthatrespondentsholdcontrastingbeliefsthat(e.g.theonlythingthatiscertainisuncertaintyitself)donotlendthemselveseasilytoempiricalresearch(Qian&Alverman,1995).

EpistemologicalBeliefInventory(EBI)TheEBI(Schrawetal.,2002)isarefinedversionoftheEQinanattempttoenhanceitspsychometricproperties.Thisisashorterinstrumentwithamorestablefactorstructuregeneratingupto60%ofvariance(Schrawetal.,2012).Itconsistsof32itemsandyieldsfivefactorswithreliabilitiesthattypicallyrangefrom.50to.65.Theoriginal62-tiemEQandtheEBIareamongstthemostpopularmulti-facetinstrumentsofpersonalepistemology;however,bothofthemarenotfreefrommeasurementproblems.Yetincomparison,theEBIisslightlymorefavorableastheauthorshavemadeaneffortinthisrefined32-itemversiontoremoveredundantitems,andrephraseambiguouslywordeditems.Sucheffortleadtoimprovedclarityinthecomprehensionoftheitems,anenhancedstructuralstability,aswellasreliabilitiessetwithinanacceptablerange.

Discipline-FocusedEpistemologicalBeliefsQuestionnaire(DFEBQ)TheFEDBQ(Hofer,2000)isadomain-specificquestionnairewithitemssimilartothoseofSchommer’s(1990)EQ.Itcontains18items,assessingfourfactorscorrespondingtoHofer’srevisedmodelofepistemologicalbeliefs,usinga7-pointLikertscale.Respondentsareaskedtofocusonaspecificacademicdiscipline(e.g.,ScienceorPsychology)astheyrateeachitem.

129

Similarfactorstructureswereproducedacrossdisciplines.Theresultsthatthescience-focusedversionreportedlessmaturebeliefsthanthoseofthepsychology-focusedversionseemtosupportdomainspecificitytosomeextent. MeasurementIssuesEmpiricalresearchgenerallysupportsthepositiverelationshipbetweensophisticatedepistemologicalbeliefsandanumberofdesiredlearningoutcomes.Forinstance,researchershavefoundcorrelationsbetweensophisticatedepistemologicalbeliefsandtwo-sidedreasoningaboutcontroversialsocialissues(Kardash&Scholes,1996;Mateosetal.,2011;Schommer-Aikins&Hutter,2002),positivechangesinscientificconception(Sinatra,Southerland,McConaughy&Demastes,2003),andacademicachievement(Rodríguez&Cano,2006).Self-reportinstrumentsmeasuringmultipleepistemologicalbeliefsaremorewidelyusedinempiricalstudiesfortheiramenabilitytorelativelymorecomplexstatisticalanalyses(Schraw,2012),ascomparedtothosemeasuringthedevelopmentofepistemologicalunderstanding.Inparticular,thepopularityoftheEBIhasbeengrowinginthepastyearsbecauseofitsdomaingenericnatureanditsadvantageofmeasuringallfivefacetsoftheoriginalSchommer’smodelusingonlyhalfoftheEQ’sitems.Althoughmeasurementproblemshavebeenpersistentinempiricalepistemologicalresearch,multi-facetquestionnairesstillcontributetothefieldbecauseoftheirstrengthforallowinganotherwisefuzzyandcomplexconstructtobebrokendownintoagreeablefacets.Examinationofinterrelationshipsamongthesefacetsandtheiruniqueandcombinedvariancesinpredictingteachingandlearningvariablesisalsomadepossible.Forinstance,Chanandcolleagues(2011)foundthatevenaftercontrollingforcognitiveability,individualswhoholdthebeliefthatknowledgeiscertainandabsoluteperformedmorepoorlyintwo-sidedthinkingtasksthanthosewhorecognizedthatknowledgeischangeable.Thisfindingsupportedtheuniquecontributionofepistemologicalbeliefstoreasoningbeyondaperson’soverallintellectualcapacity.Anotherissuetotakenoteofisthatinstrumentsusingdomaingenericbeliefsmightnotbeadequateincapturingepistemicattitudesthatarespecifictoaparticularcontextordiscipline.Likewise,thosemeasuringdomainspecificbeliefssufferfromlimitedgeneralizationoffindings.Giventhelackofaunifiedconceptualizationofpersonalepistemologyandpsychometricinconsistenciesinfactorstructureacrossstudieswithdifferentinstruments,researchersshouldtakecautionwithresultinterpretation. ConclusionInrecentyears,alternatetheorieshavebeenputforththatareworthtakingnoteofforfuturestudies.Inparticular,thefieldhascalledformoreconsiderationwheninterpretingwhatconstitutessophisticatedandnaïvebelieversusinggenericmeasurementsofbeliefsaboutknowledge.Forinstance,researcherssuchasElbyandHammer(2001)haveproposedtheepistemologicalresourcestheorythatchallengedthesimpleeither-orclassificationofalearner’sbeliefasnaïveorsophisticated.Theyholdthatthecorrectsophisticatedunderstandingofthedisciplinescienceaschangingandtentativemightnotbeproductivewhen

130

itcomestounderstandingsomespecificandfundamentalscientificphenomenon,suchasthatlivingorganismsevolve.ThistheorywasfoundtobesupportedwithsomepreliminaryempiricalevidenceinastudyofKuandcolleagues(2014),inwhichsophisticationofepistemologicalbeliefsweremadesalientinpredictingstudents’argumentationthinkingonlyunderanexperimentalconditionwherethestudentswerepromptedtoconsidertheambiguousinformation.Likewise,Bromme,PieschlandStahl(2010)revealedintheirresearchthataperson’ssophisticatedbeliefswereactivatedin“reflective”taskcontextsandnotinothers.Inotherwords,alearnermightapproachataskina“naïve”mannerifthatenhanceseffectivenessintaskcompletion,despitehisorherabilitytoadoptamoresophisticatedapproach.Inaddition,Hofer(2001)hasaddedmetacognitiveprocessesinthemulti-facetmodelbyemphasizingbeliefsaboutknowledgeasafunctionofmetacognitiveknowledge,andmetacognitivemonitoringandjudgments.Moreempiricaltestingofthesemodelswillenrichgeneralunderstandingofpersonalepistemology.Thechapterdiscussedthedevelopmentalandmulti-facetapproachintheorizingepistemologicalunderstanding.Thescope,strength,andlimitationofself-reportsinstrumentsmeasuringbeliefsaboutknowledge,aswellasobjectiveassessmentsofepistemologicaldevelopmentwerereviewed.Theprevioussectionalsohighlightedwhatisnotadequatelycapturedbyexistingmeasurements.Moreattentionshouldbepaidtotheinterplayofbeliefsaboutknowledgeandothercontextualfactorsinfuturestudies.Lastly,itissuggestedtoconsiderusingmorethanoneformofinstrumenttotriangulateresults,andwhenapplicable,moresophisticatedstatisticalanalysestoenhancereliabilityoffindings.

131

ReferencesReferencesmarkedwithanasteriskindicateascale.Ahola,S.(2009).Measurementissuesinstudyingpersonalepistemology.Psychologyand

Society,2(2),184-191.Bråten,I.,&Strømsø,H.I.(2004).Epistemologicalbeliefsandimplicittheoriesofintelligenceas

predictorsofachievementgoals.ContemporaryEducationalPsychology,29(4),371-388.doi:10.1016/j.cedpsych.2003.10.001

Bromme,R.,Pieschl,S.,&Stahl,E.(2010).Epistemologicalbeliefsarestandardsforadaptivelearning:Afunctionaltheoryaboutepistemologicalbeliefsandmetacognition.MetacognitionandLearning,5(1),7-26.doi:10.1007/s11409-009-9053-5

Chan,K.W.&Elliott,R.G.(2000).ExploratorystudyofepistemologicalbeliefsofHongKongteachereducationstudents:Resolvingconceptualandempiricalissues.Asia-PacificJournalofTeacherEducation,28,225-234.doi:10.1080/713650691

Clarebout,G.,Elen,J.,Luyten,L.,&Bamps,H.(2001).Assessingepistemologicalbeliefs:Schommer’sQuestionnairerevisited.EducationalResearchandEvaluation,7(1),53–77.doi:10.1076/edre.7.1.53.6927

Duell,O.K.&Schommer-Aikins,M.(2001).Measuresofpeople’sbeliefsaboutknowledgeandlearning.EducationalPsychologyReview,13(4),419-449.doi:10.1023/A:1011969931594

Elby,A.&Hammer,D.(2001).Onthesubstanceofasophisticatedepistemology.ScienceEducation,85(5),554-567.doi:10.1002/sce.1023

*Hofer,B.K.(2000).Dimensionalityanddisciplinarydifferencesinpersonalepistemology.ContemporaryEducationalPsychology,25,353-405.doi:10.1006/ceps.1999.1026

Hofer,B.K.(2001).Personalepistemologyresearch:Implicationsforlearningandteaching.EducationalPsychologyReview,13(4),353-383.doi:10.1023/A:1011965830686

Inglis,M.,&Mejia-Ramos,J.P.(2009).Theeffectofauthorityonthepersuasivenessofmathematicalarguments.CognitionandInstruction,27(1),25-50.doi:10.1080/07370000802584513

Kardash,C.M.,&Scholes,R.J.(1996).Effectsofpreexistingbeliefs,epistemologicalbeliefs,andneedforcognitiononinterpretationofcontroversialissues.JournalofEducationalPsychology,88(2),260-271.doi:10.1037/0022-0663.88.2.260

*King,P.M.&Kitchener,K.S.(1994).ThedevelopmentofReflectiveJudgmentinadolescenceandadulthood.SanFrancisco,USA:JosseyBass.

King,P.M.,&Kitchener,K.S.(2004).Reflectivejudgment:Theoryandresearchonthedevelopmentofepistemicassumptionsthroughadulthood.EducationalPsychologist,39(1),5-18.doi:10.1207/s15326985ep3901_2

Ku,K.Y.L.,Lai,C.M.,&Hau,K.T.(2014).Epistemologicalbeliefsandtheeffectofauthorityonargument-counterargumentintegration:Anexperiment.ThinkingSkillsand

Creativity,13,67-79.doi:10.1016/j.tsc.2014.03.004Kuhn,D.(1991).Theskillsofargument.Cambridge,UK:CambridgeUniversityPress.Kuhn,D.,&Park,S.H.(2005).Epistemologicalunderstandingandthedevelopmentofintellectualvalues.InternationalJournalofEducationalResearch,43,111-124.

doi:10.1016/j.ijer.2006.05.003Kuhn,D.,Cheney,R.,&Weinstock,M.(2000).Thedevelopmentofepistemological

understanding.CognitiveDevelopment,15,309-328.doi:10.1016/j.ijer.2006.05.003

132

Mateos,M.,Cuevas,I.,Martı´n,E,Martı´n,A.,Echeita,G.,&Luna,M.(2011).Readingtowriteanargumentation:Theroleofepistemological,readingandwritingbeliefs.JournalofResearchinReading,34(3),2011,281-297.doi:10.1111/j.1467-9817.2010.01437.x

Perry,W.G.,Jr.(1970).Formsofintellectualandethicaldevelopmentinthecollegeyears:Ascheme.NewYork:Holt,Rinehart,andWinston.

Qian,G.,&Alvermann,D.(1995).Roleofepistemologicalbeliefsandlearnedhelplessnessinsecondaryschoolstudents'learningscienceconceptsfromtext.JournalofEducationalPsychology,87(2),282-292.doi:10.1037/0022-0663.87.2.282

Rodríguez,L.&Cano,F.(2006).Theepistemologicalbeliefs,learningapproachesandstudyorchestrationsofuniversitystudents.StudiesinHigherEducation,31,617–636.doi:10.1080/03075070600923442

*Schommer,M.(1990).Effectsofbeliefsaboutthenatureofknowledgeoncomprehension.JournalofEducationalPsychology,82(3),498-504.doi:10.1037/0022-0663.82.3.498

Schommer,M.A.(1993).Epistemologicaldevelopmentandacademicperformanceamongsecondaryschools.JournalofEducationalPsychology,85(3),406-411.doi:10.1037/0022-0663.85.3.406

Schommer,M.A.(1998).Theinfluenceofageandschoolingonepistemologicalbeliefs.BritishJournalofEducationalPsychology,68,551-562.doi:10.1111/j.2044-8279.1998.tb01311.x

Schommer-Aikins,M.(2004).Explainingtheepistemologicalbeliefsystem:Introducingtheembeddedsystemicmodelandcoordinatedresearchapproach.EducationalPsychologist,39(1),19-29.doi:10.1207/s15326985ep3901_3

Schommer-Aikins,M.,&Hutter,R.(2002).Epistemologicalbeliefsandthinkingabouteverydaycontroversialissues.TheJournalofPsychology,136(1),5–20.doi:10.1080/00223980209604134

*Schraw,G.,Bendixen,L.D.,&Dunkle,M.E.(2002).DevelopmentandvalidationoftheEpistemicBeliefInventory(EBI).InB.K.Hofer&P.R.Pintrich(Eds.),Personalepistemology:Thepsychologyofbeliefsaboutknowledgeandknowing(pp.261-275).Mahwah,NJ:LawrenceErlbaum.

Sinatra,G.M.,Southerland,S.A.,McConaughy,F.,&Demastes,J.W.(2003).Intentionsandbeliefsinstudents’understandingandacceptanceofbiologicalevolution.JournalofResearchinScienceTeaching,40(5),510-528.doi:10.1002/tea.10087

*Wood,P.K.,Kitchener,K.S.,&Jensen,L.(2002).Considerationsinthedesignandevaluationofapaper-and-pencilmeasureofreflectivethinking.InB.HoferandP.Pintrich(Eds.),Personalepistemology:Thepsychologyofbeliefsaboutknowledgeandknowing(pp.277-294).Mahwah,NJ:LawrenceErlbaum.

Wood,P.,&Kardash,C.(2002).Criticalelementsinthedesignandanalysisofstudiesofepistemology.InB.K.Hofer,&P.R.Pintrich(Eds.),Personalepistemology:Thepsychologyofbeliefsaboutknowledge(pp.231-261).Mahwah,NJ:LawrenceErlbaum.

133

Chapter11:MeasuringWell-BeingintheScholarshipofTeachingandLearningKristinLayous1,S.KatherineNelson2,andAngelaM.Legg3 1CaliforniaStateUniversity,EastBay,2Sewanee:TheUniversityoftheSouth,3PaceUniversity

Students’achievementandlearningdonotsimplyreflecttheirlatentabilitiesortheirconscientiousness.Asinstructors,wewanttoknowtheX-factorthatcouldenhancestudents’learningexperience,aswellasthenegativefactorsthatcouldhinderit.Weproposethatstudents’psychologicalwell-being—includingpositiveaspectslikehavingaglobalsensethatone’slifeisgoodandnegativeaspectslikecripplingstressandanxiety—isanimportantfactorinunderstandingstudents’experienceintheclassroom,includingtheirlearning,growth,motivation,andultimategrade.Forexample,subjectivewell-being—one’sgloballifesatisfaction,frequencyofpositiveemotions,andinfrequencyofnegativeemotions(Diener,Suh,Lucas,&Smith,1999)—ispredictiveofsuccessinmultiplelifedomains,includingrelationships,health,andwork(Lyubomirsky,King,&Diener,2005).Thischapterwillprovideinstructorswithanoverviewofscalestoassessdifferentaspectsofwell-beingandillustratehowinstructorscanincorporatethesemeasuresintheirscholarshipofteachingandlearning(SoTL).PositiveAspectsofWell-BeingSomeresearcherssubsetpositiveaspectsofwell-beingintohedonicandeudaimoniccomponents,withtheformerbeingthe“subjectivewell-being”constructdescribedearlier(e.g.,lifesatisfaction,positiveandnegativeemotions)orjustplainold“happiness,”andthelatterbeingthedegreetowhichonehasasenseofmeaningorpurposeinlife(e.g.,Ryff,1989).Inpracticality,hedonicandeudaimonicwell-beingarehighlyoverlapping(Kashdan,Biswas-Diener,&King,2008),butscalesexisttomeasuretheconceptuallydistinctconstructsandweincludeavarietyofoptionshere.Notably,althoughvariousaspectsofwell-beinghavebeenlinkedwithacademicallyrelatedoutcomesamongcollegestudents(seebelow),researchonwell-beingandstudentperformance,engagement,motivation,andlearningisnotasprevalentasmightbeexpected,andcouldbearipeareaforfutureresearch.

LifeSatisfactionTheSatisfactionWithLifeScale(SWLS;Diener,Emmons,Larsen,&Griffin,1985),themostwidelyusedlifesatisfactionmeasure,assessesrespondents’currentsatisfactionandhasbeenlinkedwithacademicachievementamongcollegestudents(Borrello,2005;Lepp,Barkley,&Karpinsky,2014).TheSWLSconsistsoffivequestions(e.g.,“Inmostwaysmylifeisclosetomyideal,”“Iamsatisfiedwithmylife”),whichareratedon7-pointLikert-typescales(1=stronglydisagree,7=stronglyagree).ValidationstudieshaveshownthattheSWLScomprisesasinglefactorandpossesseshighinternalreliability(Cronbach’sα=.87)andhightest-retestreliability(r=.82;Dieneretal.,1985).Recentevidencesuggeststhatjustusingoneitem,“Ingeneral,howsatisfiedareyouwithyourlife?,”hadsimilarpatternswithotherrelatedvariablesasusing

134

therestofthescale,anditalonemightbesufficienttocaptureaquickpictureoflifesatisfaction(4-itemscalefrom1=verysatisfiedto4=verydissatisfied,reverse-scored;Cheung,&Lucas,2014;alsoseeHussey&Lehan’s(2015)chapterofthise-bookforadditionalinformationonscalevalidation).Oneofthesimplestwaysforinstructorstoincorporatewell-beingscalesintoclassroomresearchistoadministertraitversionsatthebeginningofacourseandthenusethemtopredictcoursegradesorotheroutcomesthroughoutthecourse(e.g.,papergrades,attendance).Inonesuchstudy,researchersadministeredtheSWLSonthefirstdayofanintroductorypsychologycourseandfoundthatlifesatisfactionwaspositivelyrelatedtostudents’finalgrades(Borrello,2005).Surprisinglylittleresearchhastakenthisapproach,soSoTLresearcherscouldcontributegreatlytotheliteratureonwell-beingandacademicachievement.Inaddition,exposingstudentstoactualscalescouldbeavaluablelearningtoolastheycanseefirst-handwhatitmeanswhenaresearcharticlesaysthatparticipantsreportedtheirlifesatisfaction.

PositiveandNegativeEmotionsInadditiontoknowinghowstudentsfeelabouttheirlivesingeneral,instructorsmightalsowanttoknowhowpositivelyornegativelystudentshavefeltoverthepastweek,pastfewdays,today,oreven“rightnow.”Indeed,inaccordancewiththebroaden-and-buildtheory(Fredrickson,2001),positiveemotionsatonetimepointpredictedschool-relatedpersonalresources(i.e.,acompositeofacademicself-efficacy,andstudy-relatedhopeandoptimism)atasubsequenttimepointamonguniversitystudents(Ouweneel,LeBlanc,&Schaufeli,2011).Toassesspositiveandnegativeemotions,werecommendtheAffect-AdjectiveScale(AAS;Diener&Emmons,1984;Cronbach’sα=.89and.84,respectively),theModifiedDifferentialEmotionsScale(mDES;Fredrickson,Tugade,Waugh,&Larkin,2003;α=.79andα=.69,respectively),orthePositiveandNegativeAffectSchedule;(PANAS;Watson,Clark,&Tellegen,1988,αs>.84).Thequestionstemsofanyofthethreescalescanallbealteredtoassessthetimeinthecourseofinterest(e.g.,“rightnow”vs.“pastweek”).Eachscaleassessespeople’sexperienceofpositiveandnegativeemotionsoveraspecifiedtimeperiod,buteachhasuniqueattributesthatmightmakeitbetterorworseforyourpurposes.Forexample,theAASandPANASarelargelycomposedofhigh-arousalemotions(e.g.,happy,joyful,distressed,irritable),whereasthemDESincludesbothlowandhigharousalemotions(e.g.,interested,alert,curious).BoththeAASandthePANASassesstheextenttowhichpeoplehavefeltacertainway(notatalltoverymuchorextremely),whereasthemDESasksthefrequencywithwhichpeoplehavefeltacertainway(never,hardly,someofthetime,often,andmostofthetime).Theformermayassessintensityofemotionsratherthanfrequencyofoccurrence,whereasthelattermaymisstheintensityoftheemotion,butcapturefrequency.BoththeAASandthePANASassessoneemotionatatime,whereasalimitationofthemDESisthatitliststhreesimilaremotionsatonce(e.g.,amused,fun-loving,andsillyappeartogether),thusintroducingconfusionthroughtriple-barreledquestions(e.g.,Gehlbach,2015).Thenumberofitemsforeachmeasuredifferwidely,withtheAASincluding9items,themDES23

135

items,andthePANAS44items,andthemeasuresalsodifferonnumberofscaleanchors,withthePANASandthemDESbothusing5-pointscalesandtheAffect-Adjectiveusinga7-pointscale.Instructorscouldtakethepulseoftheclassbyaskingstudentshowtheyfeel“rightnow”beforeanimportantclassactivityorexamortheycouldaskhowstudentshavefeltduringthe“pastfewweeks”or“pastweek”whentheyareabouttoturninanimportantandlaboriouspaper.Ineitherinstance,instructorscouldexplorewhetherpositiveornegativeemotionsrelatetoperformance.Inaddition,SoTLresearcherscanusenovelmethodologytoexplorepositiveandnegativeemotionsandtheirrelationshiptocollegeachievement.Forexample,recentinnovationsinpositiveandnegativeemotionresearchincludeexploringthewithin-personvariabilityintheexperienceofemotions.First,researcherscanassessemodiversity,thedegreetowhichapersonexperiencesavarietyofemotionsthroughouttheweek(orpastfewdays,orpastmonth,etc.;seehttp://www.emodiversity.orgforanequation;Quoidbachetal.,2014).Second,researcherscanexplorethestandarddeviationofpositiveornegativeemotionsthroughoutmultiplemeasurementperiodsthroughouttheweektoassessthedegreetowhichpeopleexperiencefluctuationsinextremityofpositiveandnegativeemotions(e.g.,highlevelsofpositiveemotionsononeday,andthenpossiblynoneatallthenextday;Gruber,Kogan,Quoidbach,&Mauss,2013).Theserelativelynewwaysofevaluatingpositiveandnegativeareaground-breakingwayofexploringemotionsandcollegeachievement.

SubjectiveWell-BeingCompositeResearcherscommonlyassessoverallwell-beingbyaveragingparticipants’lifesatisfactionandfrequencyofpositiveand(reverse-scored)negativeemotionstorepresentthetheoreticaltripartitestructureofwell-being(Dieneretal.,1999).IfthethreeconstructsaremeasuredondifferentLikertscales(e.g.,1-5vs.1-7),researcherscantransformscorestoz-scoresforeachscaleandaveragethestandardizedscores.Additionally,althoughlifesatisfactionandpositiveandnegativeemotionsaretypicallyhighlycorrelated(Pavot,Diener,Colvin,&Sandvik,1991),researchersshouldexplorethecorrelationsintheirowndatabeforecombining.Althoughresearchershaveproposedmorecomplicatedwaysofcombiningthethreeconstructs(Busseri&Sadava,2010),averagingthemisthemostcommonapproach.Inadditiontosubjectivewell-beingpredictingcourseoutcomes,SoTLresearcherscouldalsousesubjectivewell-beingandtheothersimilarconstructsasdependentvariables.Researchhasnowdemonstratedahostofsimpleandbriefactivitiesthatboostwell-being(i.e.,positiveactivities;Lyubomirsky&Layous,2013;Sin&Lyubomirsky,2009)andreducethenegativeeffectsofthreatonacademicachievement(i.e.,self-affirmationandbelonginginterventions;Cohen&Sherman,2014;Walton&Cohen,2011).Manyoftheseactivitiestakeabout10-15minutesandcouldeasilybeadministeredwithinaclasssessionoronlineashomework.Furthermore,instructorscouldalsotesttheeffectsofintervention-inducedchangesinwell-beingonsubsequentexamsorclassassignmentsbyadministeringtheintervention,measuringwell-beingasamanipulationcheckandmediator,andthenassessingperformanceasabehavioraloutcome.

136

HappinessRatherthanrepresenthappinesswiththesubjectivewell-beingcomposite,someresearchersrecommendtosimplyaskpeopleiftheyarehappytotapwhateverthatpersonthinksitmeanstobehappy.Onesuchfacevalidmeasureisthe4-itemSubjectiveHappinessScale(SHS;Lyubomirsky&Lepper,1999),whichasksparticipantstoconsiderhowgenerallyhappytheyare,howhappytheyarerelativetotheirpeers,(1=lesshappy,7=morehappy),andtheextenttowhichadescriptionofa“veryhappy”anda“veryunhappy”person,respectively,characterizesthem(1=notatall,7=agreatdeal;Cronbach’sα’s>.79).Althoughresearchersshouldtakecautionbeforedroppingitemsfromavalidatedscale,recentresearchsuggeststhatnegativelywordedandreverse-scoreditemscontributetopoorreliability(Gehlbach,2015)andrecentresearchontheSHSinspecificsuggeststhatdroppingthefourthitemimprovesscalereliability(O’Connor,Crawford,&Holder,2014;alsoseeWilson-Doenges,2015formorediscussionofthisissue).AmonghighschoolstudentsinHongKong,studentscoresontheSHSwererelatedtotheirperceptionsoftheirschool’squalityandtheirowngrades(Kashdan&Yuen,2007).Similarly,liketheSWLS,theSHStakenatthebeginningofanintroductorypsychologycoursewasalsorelatedtofinalgrades(Borrello,2005).Inadditiontoexploringhowwell-beingrelatestofinalgrades,researcherscouldalsoexplorewhetherwell-beingrelatestoexamperformanceorpossiblyjustrelatestofinalgradesduetoperseveranceonhomeworkassignmentsandclassparticipation.Additionally,researcherscouldmeasuresstressoranxiety(seeNegativeAspectsofWell-Beinginthischapter)toexplorewhetherwell-beingsimplybuffersthenegativeeffectsofstressonacademicperformanceorisuniquelyrelated.

EudaimonicWell-BeingTheQuestionnaireforEudaimonicWell-Being(QEWB;Watermanetal.,2011)wasdevelopedtoassesspeople’swell-beingderivedfromthedevelopmentandfulfillmentoftheirindividualpotential(asopposedtojustassessinghowpeoplegenerallyfeelabouttheirlives,regardlessofthesource,likethelifesatisfactionandhappinessscales).Thescaleincludes21items(e.g.,“Mylifeiscenteredaroundasetofcorebeliefsthatgivemeaningtomylife”),whichareratedonascalerangingfrom0(stronglydisagree)to4(stronglyagree).Moreover,thescaledemonstratesstrongreliability,withCronbach’sα=.86.TheQEWBalsoincludesquestionsaddressingpersonalidentityandgrowth.Indeed,onestudyfoundthatwell-being(measuredwiththeQEWB)significantlymediatedtheassociationbetweenidentitydevelopment(e.g.,commitmentmaking,identificationwithcommitment)andinternalizingsymptoms,externalizingsymptoms,andhealth-riskbehaviors(Ritchieetal.,2013).Accordingly,instructorsmaywishtoadministerthisscaleintheirclassroomsinthecontextofdiscussingidentitydevelopmentinemergingadulthood.

137

MeaninginLifeTheMeaninginLifeQuestionnaire(MLQ;Steger,Frazier,Oishi,&Kaler,2006)isoneofthemostwidelyusedscalesassessinglifemeaning.Itincludes10itemstoassessthepresenceof(e.g.,“Mylifehasaclearsenseofpurpose”)andsearchfor(e.g.,“Iamalwayslookingforsomethingthatmakesmylifefeelmeaningful”;1=absolutelyuntrue,7=absolutelytrue)meaninginlife.Bothsubscalesdemonstratedstrongreliability(Cronbach’sαs=.86and.87,respectively),aswellasconvergentanddiscriminantvalidity.Demonstratingtheapplicabilityacrossawiderangeofdemographics,bothsubscaleswerelargelyunrelatedtoage,gender,race,andreligion(Stegeretal.,2006).InstructorscouldcontrasttheMLQandtheQEWBwiththeSWLSandSHStoillustratethevariouswaysinwhichresearchersassesswell-being(seealsothenextfewscales).Ifthefocusofthecourseisonwell-being,instructorscoulddoamoreelaboratestudyofhowthesetypesofcoursesaffectstudentwell-being.Forexample,onestudymeasuredstudents’happiness,lifesatisfaction,self-actualization,hope,andsearchforandpresenceofmeaninginlifeatthebeginningandendofasemester-longcourseonpositivepsychology(Maybury,2013).Throughoutthecourse,studentscompletedcourseactivitiesregardingtheirpersonalvaluesandcharacterstrengths,gratitude,andoptimism.Overthecourseofthesemester,studentsreportedgainsinhope,self-actualization,lifesatisfaction,happiness,andsearchformeaning,butnotpresenceofmeaning.Accordingly,theseclassresultsmayofferinstructorstheopportunitytodiscusswhatmakesforahappylifeandthedifferencebetweensearchingformeaninginlifeandfeelingthatlifeisalreadymeaningful.

PsychologicalWell-BeingIncontrasttothesingularfocusonmeaninginlife,theScalesofPsychologicalWell-Being(PWB;Ryff,1989)conceptualizeswell-beingasincludingmultiplefacets.ThePWBwasoriginallydevelopedasa120-iteminstrumentencompassing6subscalesrepresentingeachfacet(20itemsperscale):self-acceptance(e.g.,“Ilikemostaspectsofmypersonality”),positiverelationswithothers(e.g.,“IknowthatIcantrustmyfriends,andtheyknowtheycantrustme”),autonomy(e.g.,“Iamnotafraidtovoicemyopinions,evenwhentheyareinoppositiontotheopinionsofmostpeople”),environmentalmastery(e.g.,“Ingeneral,IfeelIaminchargeofthesituationinwhichIlive”),purposeinlife(e.g.,“Ihaveasenseofdirectionandpurposeinlife”),andpersonalgrowth(e.g.,“Ithinkitisimportanttohavenewexperiencesthatchallengehowyouthinkaboutyourselfandtheworld”).Thelengthofthisscalemakesitcumbersomeforusewithintheclassroom,butinstructorscouldpresentasubsetofitemstocontrastthePWBwiththeaforementionedscales.Additionally,thisscaleisincludedintheMidlifeintheUnitedStatesstudy(MIDUS),anationallongitudinalstudyofhealthandwell-being,sointerestedstudentscouldrequestuseofthisdataforanindependentresearchprojectonhowwell-beingrelatestoahostofphysicalhealthanddemographicvariables(http://www.midus.wisc.edu/).

138

PsychologicalFlourishingTheMentalHealthContinuum(MHC;Keyes,2002)wasdevelopedasatoolformeasuringmentalhealthasastateofflourishing,ratherthanmerelytheabsenceofdisease.Accordingly,thismeasureincludesthreesubscales:emotionalwell-being(6items;e.g.,“Howmuchofthetimeinthelast30daysdidyoufeelfulloflife?”;1=noneofthetime,5=all;Cronbach’sα=.91),psychologicalwell-being(18itemsadaptedfromRyff,1989;e.g.,“Ilikemostpartsofmypersonality”;1=disagreestrongly,7=agreestrongly;α=.81),andsocialwell-being(15items;e.g.,“Ifeelclosetootherpeopleinmycommunity”;1=disagreestrongly,7=agreestrongly;α=.81).A14-itemshort-formoftheMHC(Lamers,Westerhof,Bohlmeijer,tenKlooster,&Keyes,2011)wasrecentlydeveloped,includingshortenedversionsofeachofthethreesubscales,whichalsodemonstratedstrongreliability(αs>.74forthethreesubscales,α=.89forthetotalMHC-SF).Todiagnosementalhealth,averagesforthethreesubscalesarecalculated,andthosewhoscoreintheuppertertileareconsideredtobeflourishingandthosewhoscoreinthelowertertileareconsideredtobelanguishing.

PsychologicalNeedSatisfactionBasedonself-determinationtheory(Ryan&Deci,2000),theBalancedMeasureofPsychologicalNeeds(BMPN;Sheldon&Hilpert,2012)isan18-itemscalethatmeasurespeople’sfeelingsofautonomy(e.g.,“Iwasfreetodothingsmyownway”),competence(e.g.,“Itookonandmasteredhardchallenges”),andconnectedness(e.g.,“Ifeltcloseandconnectedtootherpeoplewhoareimportanttome”).Eachitemisratedonascalerangingfrom1(noagreement)to5(muchagreement).Eachsubscaledemonstratesstrongreliability,Cronbach’sαs>.78.TheBMPNoffersanumberofadvantagesforclassroomuse.Studentscancompletethisrelativelyshortscalequickly,leavingplentyoftimefordiscussionandotheractivitiesinclass.Inaddition,theitemsreflectgeneralfeelingsratherthandomain-specificsatisfaction,renderingthescaleapplicabletostudentsacrossdiverseexperiences.Finally,thescaledemonstratesstrongreliabilityandvalidityasthreeindependentscalesorasasinglescalerepresentingoverallpsychologicalneedsatisfaction,whichprovidesinstructorswithavarietyofwaystousethisscaleintheclassroom.Forexample,aninstructorcouldadministeronlytheconnectednesssubscalewhendiscussingrelationshipsinacourseonsocialpsychologybutcouldadministertheentirescalewhendiscussingself-determinationtheoryinacourseonmotivation.

OptimismOptimismistheglobalbeliefthatgoodthingswillhappen(i.e.,generalizedoutcomeexpectancies;Scheier&Carver,1985).Therevisedlifeorientationtest(LOT-R;Scheier,Carver,&Bridges,1994)isa6-itemmeasuredesignedtoassesstraitoptimism.Respondentsareaskedtoindicatetheirdegreeofgeneralagreement“overthepastyear”withstatementssuchas“I’malwaysoptimisticaboutmyfuture,”usinga5-pointresponsescale(1=stronglydisagree,5=stronglyagree;Cronbach’sα>.78).Tomeasurestateoptimism(Kluemper,Little,&DeGroot,2009),changethequestionstemto“overthepastweek”andre-wordindividualitemstoindicatecurrentstateofoptimismwithstatementssuchas“Currently,I’moptimisticaboutmy

139

future,”usinga5-pointscale(1=stronglydisagree,5=stronglyagree).Collegestudentshighintraitoptimismwerelesslikelytoseetheireducationasasourceofstressintheirlives(Krypel&Henderson-King,2010),morelikelytoexpectsuccess(Robbins,Spence,&Clark,1991),butwerenomoreorlesslikelytoachievegoodgrades(closetozerocorrelation;Robbinsetal.,1991;seealsoRand,Martin,&Shea,2011).

HopeDistinctfromoptimism,hopetypicallyrelatestopositivefeelingsaboutspecificgoalsandplanningtomeetgoalsratherthangeneralizedexpectancies(Snyderetal.,2002).The12-itemTraitHopeScale(THS;Snyderetal.,1991)includestwosubscalestomeasurehopeviaagency(i.e.,goaldirecteddetermination;e.g.,“Ienergeticallypursuemygoals)andpathway(i.e.,planningofwaystomeetgoals;e.g.,“Icanthinkofmanywaystogetthethingsinlifethatareimportanttome”).Eachsubscaleincludesfouritems,whichareratedonascalefrom1(definitelyfalse)to8(definitelytrue).Thescaledemonstratedgoodreliabilityinacollegestudentsample(agencysubscale:Cronbach’sαs>.71,pathwayssubscale:αs>.63,totalscale:αs>.74).Hopepositivelypredictsacademicsuccessincollege(Snyderetal.,2002),andhope,butnotoptimism,positivelycorrelateswithgradesforfirst-yearlawstudents,controllingforadmissionstestscoresandundergraduategrades(Randetal.,2011).Instructorsmaywanttohighlightthedifferencesbetweenhopeandoptimismandengagestudentsinaclassdiscussionaboutwhyhope,butnotoptimism,isrelatedtoacademicachievement.NegativeAspectsofWell-BeingNegativeaspectsofwell-beingcanrangefromfeelingsofsadness,tension,lowself-efficacy,andlearnedhelplessness.Althoughmanymeasuresofclinicalsymptomologyexisttoassessstudents’depressivesymptomsoranxietydisordercriteria,forthepurposesofthischapter,wechosetofocusonnon-clinicalmeasuresofstressandanxietythatgeneralizewelltostudents.Generalfeelingsofstressandanxietyharmpeople'shealth(DeLongis,Folkman,&Lazarus,1988)and,unsurprisingly,alsonegativelyinfluencestudents'academicwell-beingandperformance(Richardson,Abraham,&Bond,2012).Self-reportmeasuresofstudentstressfallintotwocategories:(a)measuresofgeneralstressandanxiety,and(b)domain-specificstressandanxiety.

GeneralStressandAnxietySOTLresearchersandprofessorsmaywishtoassessstudents'generallevelsofstresstogaugewhethertheypredictpooreracademicperformance,difficultyadjustingtocollegelife,orothernegativeoutcomes.Thefollowingdiscussionprovidesresourcestoassesstheseoverarchinglevelsofstress.

CollegeStudents'GeneralStressTheSocialReadjustmentRatingScale(Holmes&Rahe,1967)isacommonlyusedmeasureoflifestressorsbut,forstudentpopulationswerecommendtheCollegeUndergraduateStress

140

Scale(CUSS;Renner&Mackin,1998).The51itemsontheCUSSlistmajornegativelifestressors,suchasbeingraped(oraccusedofrape),experiencingthedeathofafriendorfamilymember,contractinganSTD,andfinancialdifficulties.Thescalealsoincludeslessseverestressors,suchasbeginninganewacademicsemesterandconcernsoverphysicalappearance.Tocalculateatotalstressscore,studentsaddupthenumericalstressvaluesnexttoeachitemtheyexperiencedwithinthepastyear.Forexample,beingrapedcarriesastressvalueof100andfallingasleepinclasscarriesavalueof40.Thehighestpossiblestressscoreis3,623andthesamplefromRennerandMackin's(1998)initialvalidationstudyreportedanaveragestressscoreof1,247(SD=441).AlthoughthisscalehasbeenunderutilizedintheSoTLliterature,onepaperestablishesthisscaleasausefultoolforteachingaspectsofresearchmethodologyanddatacollection(Thieman,Clary,Olson,Dauner,&Ring,2009).Adiscussionofgeneralstresslevelscancoincidewithcoursematerialfromhealthpsychology,biopsychosocialapproachestohealth,andtheinterplaybetweenmentalstatesandphysicaloutcomes.Thescalealsoprovidesopportunitiestodiscusspositivesourcesofstresssuchasgettingmarried,gettingstraightA's,orattendinganathleticevent.Studentswilllikelyenjoytakingthiseasy-to-scoreassessmentandcanconsiderhowtheiruniquestressorsaffecttheircollegeexperience.ForfutureSoTLresearch,thisscaleprovidesarelativelyshortmeasuretogatherdataonabroadrangeofstressors.SoTLresearchersmayquestionwhetheracademic-relatedstressors(e.g.,finalsweek)posethesamelevelsofharmtotheacademicexperiencethannon-academic-relatedstressors(e.g.,seriousillnessinaclosefriendorfamilymember).

State-TraitMeasuresofStressTheCUSSapproachesstressfromalifeeventsperspective,butsomeinstructorsmayprefertoassesstheaffectiveorcognitivecomponentsofstressandanxiety.OneofthemostcommonmeasuresforanxietyistheState-TraitAnxietyInventory(STAI;Spielberger,1983),a20-itemmeasureofgeneralandtransient,stateanxiety.AlthoughtheSTAIisconsideredoneofthegoldstandardsinmeasuringanxiety,onepotentialdisadvantageisthatitsuseisfee-basedthroughthepsychologicalassessmentcompanyMindGardenInc.SomealternativestotheSTAIfullformarethe6-itemstateanxietyscale(STAI-6;MarteauandBekker,1992)andthePerceivedStressScale(PSS;Cohen,Kamarck,&Mermelstein,1983).TheSTAI-6listssixanxiety-relatedemotionssuchas“tense”and“worried.”Participantscompletethescalebyansweringtheextenttowhichtheycurrentlyfeelthoseemotionsona4-pointscale(0=notatallto4=verymuch).ThePSSconsistsof14itemsassessedona5-pointscale(0=neverto4=veryoften).Theseitemsaskparticipantstoratetheextenttowhichtheyfeltorexperiencedanxiety-provokingstressors(e.g.,"Inthelastmonth,howoftenhaveyoufoundthatyoucouldnotcopewithallthethingsyouhadtodo?").BoththeSTAI-6andPSSareinternallyreliable(STAI-6Cronbach’sα=.82;PSSα=.85).Thesestate-traitassessmentsofstressgiveinstructorsanexcellentwaytoillustratethedifferencebetweenpersonalitytendenciesandmomentary,transientstates.Formoreadvanceddiscussion,state-traitassessmentscanillustratethedifferencebetweenmoderatorswhichtendtobepersonalityvariablesandmediatorswhichcanmanifestinstatevariables.

141

Domain-SpecificStressandAnxietyAnumberofmeasuresaregearedtowardassessingdomain-specificanxiety.Forinstructorsseekingtoenhancethespecificityoftheirresearchquestions,thesescalesofferanexcellentsolution.Wediscussthedomain-specificscalestomeasuretestanxiety,mathanxiety,computeranxiety,andsocialanxiety.

TestAnxietyForinstructorsexaminingtestanxiety,the21true/false-itemTestAnxietyScale(Cronbach'salphasrangebetween.68-.81;Sarason,1984)andthe10-itemWorry-EmotionalityQuestionnaire(Liebert&Morris,1967)areviableoptionsthatbothassesstwofactorspresumedtounderlietestanxiety:cognitivethoughtsofworry,andaffectiveorphysiologicalemotionality.Forthoseresearchersneedingastatemeasureoftestanxiety(bothfactors),werecommendthe8-itemStateTestAnxietyscale(Hong&Karstensson,2002).Arguingthatthecognitivecomponentoftestanxietymoststronglypredictsdeficitsinperformance,CassadyandJohnson(2002)developedareliable(Cronbach’sα=.86)27-itemscaletoassesscognitiveworry.TheCognitiveTestAnxietyscaleincludesitemstoassessintrusive,ruminativethoughtsduringtest-takingandengaginginsocialcomparisonortest-irrelevantthinkingduringtest-taking.Participantsrespondusinga4-pointscaletosampleitemssuchas,"Duringtests,Ifindmyselfthinkingoftheconsequencesoffailing;""WhenItakeatest,mynervousnesscausesmetomakecarelesserrors;"and"Duringtests,thethoughtfrequentlyoccurstomethatImaynotbetoobright."Instructorscanfindopportunitiestoillustratetheadvantagesanddisadvantagesofamoderateamountofanxietybyaskingstudentstocompleteatestanxietyscaleeitherbeforeorafteranexam.Theinstructorcanthendemonstratewhethertestanxietypredictsexamgrades.Thesescalescanalsoopenupdiscussionabouttest-takingstrategiesandhowtoimprovetestperformancebyfocusingonanxiety-mitigation,metacognition,mindfulness,orotherrelaxationtechniques.

ComputerAnxietyAstechnologycontinuestoadvance,itmaybedifficulttoimaginestudents'experiencinganxietytowardusingcomputers.However,computeranxietyisveryreal,caninfluencestudents'attitudestowardtakingcomputer-administeredexams(Schult&McIntosh,2004),andcannegativelyinfluencestudents'performance(Brosnan,1998).TheComputerAnxietyandLearningMeasure(CALM;McInerney,Marsh,&McInerney,1999)isa65-itemmeasureconsistingoffoursubscales:gaininginitialcomputerscale(22items),stateanxiety(20items),senseofcontrol(12items),andcomputingself-concept(11items).TheCALMisreliable(Cronbach’sαs>.78)andthesubscalesallowresearcherstoadministerall,

142

some,oronlyone,andstillgaininsightregardingaspectsofstudents'computeranxiety(seeSchult&McIntosh,2004foraSoTLstudyinwhichonlythestateanxietysubscalewasused).OnepossibledisadvantagetotheCALMmeasureisthatafewitemsmaybeoutdated(e.g.,questionsmeasuringcomfortusingamouseorprintingdocuments)duetotheprevalenceofcomputersinmostpeople’severydaylives.However,oneotherwell-validatedcomputeranxietyscaleexistsandmayserveasanexcellentalternativeiftheCALMdoesnotmeetaninstructor'sneeds.TheComputerAnxietyScale(CAS;Lester,Yang,&James,2005)containssixitemsandparticipantsrespondusinga6-pointagreementanchor(stronglyagreetostronglydisagree).Thesixitemsloadontoasingleunderlyingfactorofcomputeranxiety(Chronbach’sαs>.74acrossmultiplesamples).Itemssuchas,"Ifeelconfidentandrelaxedwhilingworkingonacomputer"and"Icanusuallymanagetosolvecomputerproblemsbymyself"makeupthescale.Forinstructorssolelyseekingtoassesstheaffectivecomponentsofcomputeranxiety,werecommendthestatesubscaleoftheCALM;butforresearchersseekingtoassesscognitivecomponents,theotherfoursubscalesoftheCALMortheCASofferpracticalsolutions.

MathAnxietyPsychologicalstatisticsprofessorsoftenquipthatteachingpsychologicalstatisticsisonepartteachingmathandonepartanxietymitigation.Itcomesasnosurprisethatseveralscalesexisttomeasurestudents'self-reportedanxietiestowardmath.Anearly,yetstillwidelyused,measureistheMathAnxietyScale(Betz,1978;Fennema&Sherman,1976).Betz(1978)rewrote10itemsfromtheMathematicsAttitudesScale(Fennema&Sherman,1976)withthegoalofassessingcollegestudents'mathanxiety.Participantsrespondtothese10itemsona5-pointagreementscale(stronglydisagreetostronglyagree).Theitemsoffergoodreliability(split-halfcoefficientof.92;Betz,1978)andmeasureworryaboutsolvingmathproblems,feelingsoftensionduringmathexamsorwhenthinkingofdifficultmathproblems,andlossofclearthinkingwhenworkingwithmath.AnotherwidelyusedmeasureistheMathAnxietyRatingScale(MARS-R;Plake&Parker,1982).AlthoughtheoriginalMARS-Rcontained24items,Hopko(2003)conductedare-validationstudyandreducedthemeasureto12items.Participantsrespondtoitemsusinga4-pointscalefrom0(noanxiety)to4(highanxiety).Boththeoriginalandrevisedversionsconsistoftwosubscales:LearningMathAnxietyandMathEvaluationAnxiety.Bothsubscaleshavegoodreliability(LearningMathAnxiety,Cronbach’sα=.87;MathEvaluationAnxiety,α=.85;Hopko,2003).Byassessingstudents'traitorstatefeelingsbeforeengaginginanacademictask,instructorscanassesswhethersomeofthepositivewell-beingconstructsbufferagainstthenegativeeffectsoftestorcomputeranxiety.Forinstance,studentswithhighmathanxietytendtoperformmorepoorlyonmathexams,exceptwhentheypossesshighmetacognitiveskills(Legg&Locker,2009).Thehighmetacognitiveskillallowsthesestudentstocompensateforandovercomepotentiallydebilitatingmathanxiety.Instructorscanalsousethesescalesto

143

demonstratepeoples'varyinglevelsofanxiety(e.g.,somepeoplemayscorehighonmathanxietybutlowoncomputeranxiety).ConclusionInsum,avarietyofscalesexisttomeasurethepositiveandnegativeaspectsoftraitandstatewell-being,andtheseassessmentscouldserveaseitherpredictorsordependentvariablesinresearchprojectsforSoTLscholars.Importantly,mostofthescalesareself-reportandbrief,andarethereforehighlyconvenientforusewithintheclassroom.Inaddition,althoughwereportedsomeexamplesofinstructorsexploringwell-beingintheclassroom,wealsopointedoutmanynewwaysinwhichSoTLscorescancontributetothisburgeoningliterature.Lastly,notonlycanthescalesthemselvesbeinformativetostudents’understandingofresearchconstructs,buttheycanalsohelpinstructorsunderstandandpromoteintangibleemotionalcharacteristicsthatmighthelpstudentsthrive.

144

ReferencesReferencesmarkedwithanasteriskindicateastudy.*Betz,N.E.,(1978).Prevalence,distribution,andcorrelatesofmathanxietyincollegestudents.

JournalofCounselingPsychology,25,441-448.doi:10.1037/0022-0167.25.5.441Borrello,A.(2005).Subjectivewell-beingandacademicsuccessamongcollegestudents.

(Unpublisheddoctoraldissertation),CapellaUniversity,Minneapolis,Minnesota.Brosnan,M.J.(1998).Theimpactofcomputeranxietyandself-efficacyuponperformance.

JournalofComputerAssistedLearning,14(3),223-234.doi:10.1046/j.1365-2729.1998.143059.x

Busseri,M.A.,&Sadava,S.W.(2010).Areviewofthetripartitestructureofsubjectivewell-being:Implicationsforconceptualization,operationalization,analysis,andsynthesis.PersonalityandSocialPsychologyReview,15(3),290-314.doi:10.1177/1088868310391271

*Cassady,J.C.,&Johnson,R.E.(2002).Cognitivetestanxietyandacademicperformance.ContemporaryEducationalPsychology,27(2),270-295.doi:10.1006/ceps.2001.1094

*Cheung,F.,&Lucas,R.E.(2014).Assessingthevalidityofsingle-itemlifesatisfactionmeasures:Resultsfromthreelargesamples.QualityofLifeResearch,23(10),2809-2818.doi:10.1007/s11136-014-0726-4

*Cohen,S.,Kamarck,T.,&Mermelstein,R.(1983).Aglobalmeasureofperceivedstress.JournalofHealthandSocialBehavior,24(4),385-396.doi:10.2307/2136404

Cohen,G.L.,&Sherman,D.K.(2014).Thepsychologyofchange:self-affirmationandsocialpsychologicalintervention.AnnualReviewofPsychology,65,333–71.doi:10.1146/annurev-psych-010213-115137

DeLongis,A.,Folkman,S.,&Lazarus,R.S.(1988).Theimpactofdailystressonhealthandmood:Psychologicalsocialresourcesasmediators.JournalofPersonalityandSocialPsychology,54(3),486-495.doi:10.1037/0022-3514.54.3.486

*Diener,E.,&Emmons,R.A.(1984).Theindependenceofpositiveandnegativeaffect.JournalofPersonalityandSocialPsychology,47(5),1105-1117.doi:10.1037/0022-3514.47.5.1105

*Diener,E.,Emmons,R.A.,Larsen,R.J.,&Griffin,S.(1985).Thesatisfactionwithlifescale.JournalofPersonalityAssessment,49(1),71-75.doi:10.1207/s15327752jpa4901_13

Diener,E.,Suh,E.M.,Lucas,R.E.,&Smith,H.L.(1999).Subjectivewell-being:Threedecadesofprogress.PsychologicalBulletin,125(2),276-302.doi:10.1037/0033-2909.125.2.276

*Fennema,E.,&Sherman,J.A.(1976).Fennema-ShermanMathematicsattitudesscales:Instrumentsdesignedtomeasureattitudestowardthelearningmathematicsbymalesandfemales.JournalforResearchinMathematicsEducation,7(5),324-326.

Fredrickson,B.L.(2001).Theroleofpositiveemotionsinpositivepsychology:Thebroaden-and-buildtheoryofpositiveemotions.AmericanPsychologist,56(3),218–226.doi:10.1037/0003-066X.56.3.218

*Fredrickson,B.L.,Tugade,M.M.,Waugh,C.E.,&Larkin,G.R.(2003).Whatgoodarepositiveemotionsincrises?AprospectivestudyofresilienceandemotionsfollowingtheterroristattacksontheUnitedStatesonSeptember11th,2001.JournalofPersonalityandSocialPsychology,84(2),362-376.doi:10.1037/0022-3514.84.2.365

145

Gehlbach,H.(2015).Sevensurveysins.JournalofEarlyAdolescence.doi:10.1177/0272431615578276.

Gruber,J.,Kogan,A.,Quoidbach,J.,&Mauss,I.B.(2013).Happinessisbestkeptstable:Positiveemotionvariabilityisassociatedwithpoorerpsychologicalhealth.Emotion,13(1),1-6.doi:10.1037/a0030262

Holmes,T.H.,&Rahe,R.H.(1967).Thesocialreadjustmentratingscale.JournalofPsychosomaticResearch,11(2),213-218.doi:10.1016/0022-3999(67)90010-4

Hong,E.,&Karstensson,L.(2002).Antecedentsofstatetestanxiety.ContemporaryEducationalPsychology,27(2),348-367.doi:10.1006/ceps.2001.1095

Hopko,D.R.(2003).Confirmatoryfactoranalysisofthemathanxietyratingscale-revised.EducationalandPsychologicalMeasurement,63(2),336-351.doi:10.1177/0013164402251041

Hussey,H.D.,&Lehan,T.(2015).Aprimeronscaledevelopment.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Kashdan,T.B.,Biswas-Diener,R.,&King,L.A.(2008).Reconsideringhappiness:Thecostsofdistinguishingbetweenhedonicsandeudaimonia.TheJournalofPositivePsychology,3(4),219-233.doi:0.1080/17439760802303044

Kashdan,T.B.,&Yuen,M.(2007).Whetherhighlycuriousstudentsthriveacademicallydependsonperceptionsabouttheschoollearningenvironment:AstudyofHongKongadolescents.MotivationandEmotion,31(4),260-270.doi:10.1007/s11031-007-9074-9

*Keyes,C.L.M.(2002).Thementalhealthcontinuum:Fromlanguishingtoflourishinginlife.JournalofHealthandSocialBehavior,43(2),207-222.doi:10.2307/3090197

Krypel,M.N.,&Henderson-king,D.(2010).Stress,copingstyles,andoptimism:Aretheyrelatedtomeaningofeducationinstudents'lives?SocialPsychologyofEducation:AnInternationalJournal,13(3),409-424.doi:10.1007/s11218-010-9132-0

*Kluemper,D.H.,Little,L.M.,&DeGroot,T.(2009).Stateortrait:Effectsofstateoptimismonjob-relatedoutcomes.JournalofOrganizationalBehavior,30(2),209-231.doi:10.1002/job.591

Lamers,S.M.A.,Westerhof,G.J.,Bohlmeijer,E.T.,tenKlooster,P.M.,&Keyes,C.L.M.(2011).EvaluatingthepsychometricpropertiesoftheMentalHealthContinuum-ShortForm(MHC-SF).JournalofClinicalPsychology,67,99-110.doi:10.1002/jclp.20741

Legg,A.M.,&Locker,L.(2009).Mathperformanceanditsrelationshiptomathanxietyandmetacognition.NorthAmericanJournalofPsychology,11(3),471-485.

Lepp,A.,Barkley,J.E.,&Karpinski,A.C.(2014).Therelationshipbetweencellphoneuse,academicperformance,anxiety,andsatisfactionwithlifeincollegestudents.ComputersinHumanBehavior,31,343-350.doi:10.1016/j.chb.2013.10.049

*Lester,D.,Yang,B.,&James,S.(2005).Ashortcomputeranxietyscale.PerceptualandMotorSkills,100(3),964-968.doi:10.2466/pms.100.3c.964-968

*Liebert&Morris(1967).Cognitiveandemotionalcomponentsoftestanxiety:Adistinctionandsomeinitialdata.PhysiologicalReports,20(3),975-978.doi:10.2466/pr0.1967.20.3.975

146

Lyubomirsky,S.,King,L.,Diener,E.(2005).Thebenefitsoffrequentpositiveaffect:Doeshappinessleadtosuccess?PsychologicalBulletin,131(6),803-855.doi:10.1037/0033-2909.131.6.803

Lyubomirsky,S.,&Layous,K.(2013).Howdosimplepositiveactivitiesincreasewell-being?CurrentDirectionsinPsychologicalScience,22(1),57-62.doi:10.1177/0963721412469809

*Lyubomirsky,S.,&Lepper,H.S.(1999).Ameasureofsubjectivehappiness:Preliminaryreliabilityandconstructvalidation.SocialIndicatorsResearch,46(2),137-155.doi:10.1023/A:1006824100041

*Marteau,T.M.,&Bekker,H.(1992).Thedevelopmentofasix-itemshort-formofthestatescaleoftheSpielbergerState-TraitAnxietyInventory(STAI).BritishJournalofClinicalPsychology,31(3),301-306.doi:10.1111/j.2044-8260.1992.tb00997.x

Maybury,K.K.(2013).Theinfluenceofapositivepsychologycourseonstudentwell-being.TeachingofPsychology,40(1),62-65.doi:10.1177/0098628312465868

*McInerney,V.,Marsh,H.W.,&McInerney,D.M.(1999).ThedesigningoftheComputerAnxietyandLearningMeasure(CALM):Validationofscoresonamultidimensionalmeasureofanxietyandcognitionsrelatingtoadultlearningofcomputingskillsusingstructuralequationmodeling.EducationandPsychologicalMeasurement,59(3),451-470.doi:10.1177/00131649921969974

O’Connor,B.P.,Crawford,M.R.,&Holder,M.D.(2014).Anitemresponsetheoryanalysisofthesubjectivehappinessscale.SocialIndicatorsResearch.doi:10.1007/s11205-014-0773-9.

Ouweneel,E.,LeBlanc,P.M.,Schaufeli,W.B.(2011).Flourishingstudents:Alongitudinalstudyonpositiveemotions,personalresources,andstudyengagement.TheJournalofPositivePsychology,6(2),142-143.doi:10.1080/17439760.2011.558847

Pavot,W.,Diener,E.,Colvin,C.R.,&Sandvik,E.(1991).Furthervalidationofthesatisfactionwithlifescale:Evidenceforthecross-methodconvergenceofwell-beingmeasures.JournalofPersonalityAssessment,57(1),149-161.doi:10.1207/s15327752jpa5701_17

*Plake,B.S.,&Parker,C.S.(1982).ThedevelopmentandvalidationofarevisedversionoftheMathematicsAnxietyRatingScale.EducationalandPsychologicalMeasurement,42(2),551-557.doi:10.1177/001316448204200218

Quoidbach,J.,Gruber,J.,Mikolajczak,M.,Kogan,A.,Kotsou,I.,&Norton,M.(2014).Emodiversityandtheemotionalecosystem.JournalofExperimentalPsychology:General,143(6),2057-2066.doi:10.1037/a0038025

Rand,K.L.,Martin,A.D.,Shea,A.M.(2011).Hope,butnotoptimism,predictsacademicperformanceoflawstudentsbeyondpreviousacademicachievement.JournalofResearchinPersonality,45(6),683-686.doi:10.1016/j.jrp.2011.08.004

*Renner,M.J.,&Mackin,R.S.(1998).Alifestressinstrumentforclassroomuse.TeachingofPsychology,25(1),46-48.doi:10.1207/s15328023top2501_15

Richardson,M.,Abraham,C.,&Bond,R.(2012).Psychologicalcorrelatesofuniversitystudents'academicperformance:Asystematicreviewandmeta-analysis.PsychologicalBulletin,138(2),353-387.doi:10.1037/a0026838

147

Ritchie,R.A.,Meca,A.,Madrazo,V.L.,Schwartz,S.J.,Hardy,S.A.,Zamboanga,B.L.,...&Lee,R.M.(2013).Identitydimensionsandrelatedprocessesinemergingadulthood:Helpfulorharmful?JournalofClinicalPsychology,69(4),415-432.doi:10.1002/jclp.21960.

Robbins,A.S.,Spence,J.T.,&Clark,H.(1991).Psychologicaldeterminantsofhealthandperformance:Thetangledwebofdesirableandundesirablecharacteristics.JournalofPersonalityandSocialPsychology,61,755-765.doi:10.1002/jclp.21960

Ryan,R.M.,&Deci,E.L.(2000).Self-determinationtheoryandthefacilitationofintrinsicmotivation,socialdevelopment,andwell-being.AmericanPsychologist,55,68-78.doi:10.1037/0003-066X.55.1.68

*Ryff,C.D.(1989).Happinessiseverything,orisit?Explorationsonthemeaningofpsychologicalwell-being.JournalofPersonalityandSocialPsychology,57(6),1069-1081.doi:10.1037/0022-3514.57.6.1069

*Sarason,I.G.,(1984).Stress,anxiety,andcognitiveinterference:Reactionstotests.JournalofPersonalityandSocialPsychology,46(4),929-938.doi:10.1037/0022-3514.46.4.929

Scheier,M.F.,&Carver,C.S.(1985).Optimism,coping,andhealth:Assessmentandimplicationsofgeneralizedoutcomeexpectancies.HealthPsychology,4(3),219–247.doi:10.1037/0278-6133.4.3.219

*Scheier,M.F.,Carver,C.S.,&Bridges,M.W.(1994).Distinguishingoptimismfromneuroticism(andtraitanxiety,self-mastery,andself-esteem):AreevaluationoftheLifeOrientationTest.JournalofPersonalityandSocialPsychology,67(6),1063-1078.doi:10.1037/0022-3514.67.6.1063

Schult,C.A.,&McIntosh,J.L.(2004).Employingcomputer-administeredexamsingeneralpsychology:Studentanxietyandexpectations.TeachingofPsychology,31(3),209-211.doi:10.1207/s15328023top3103_7

Sheldon,K.M.,&Hilpert,J.C.(2012).Thebalancedmeasureofpsychologicalneeds(BMPN)scale:Analternativedomaingeneralmeasureofneedsatisfaction.MotivationandEmotion,36(4),439-451.doi:10.1007/s11031-012-9279-4

Sin,N.L.,&Lyubomirsky,S.(2009).Enhancingwell-beingandalleviatingdepressivesymptomswithpositivepsychologyinterventions:Apractice-friendlymeta-analysis.JournalofClinicalPsychology:InSession,65(5),467-487.doi:10.1002/jclp.20593

*Snyder,C.R.,Harris,C.,Anderson,J.R.,Holleran,S.A.,Irving,L.M.,Sigmon,S.T.,…&Harney,P.(1991).Thewillandtheways:Developmentandvalidationofanindividual-differencesmeasureofhope.JournalofPersonalityandSocialPsychology,60,570-585.doi:10.1037/0022-3514.60.4.570

Snyder,C.R.,Shorey,H.S.,Cheavens,J.,Pulvers,K.M.,AdamsIII,V.H.,&Wiklund,C.(2002).Hopeandacademicsuccessincollege.JournalofEducationalPsychology,94,820-826.doi:10.1037/0022-0663.94.4.820

*Spielberger,C.D.(1983).ManualfortheState-TraitAnxietyInventorySTAI(FormY).PaloAlto,CA:ConsultingPsychologistsPress.

*Steger,M.F.,Frazier,P.,Oishi,S.,&Kaler,M.(2006).TheMeaninginLifeQuestionnaire:Assessingthepresenceofandsearchformeaninginlife.JournalofCounselingPsychology,53,80-93.doi:10.1037/0022-0167.53.1.80

148

Thieman,T.J.,Clary,E.G.,Olson,A.M.,Dauner,R.C.,&Ring,E.E.(2009).Introducingstudentstopsychologicalresearch:Generalpsychologyasalaboratorycourse.TeachingofPsychology,36(3),160-168.doi:10.1080/00986280902959994

Walton,G.M.,&Cohen,G.L.(2011).Abriefsocial-belonginginterventionimprovesacademicandhealthoutcomesofminoritystudents.Science,331,1447–1451.doi:10.1126/science.1198364

*Waterman,A.S.,Schwartz,S.J.,Zamboanga,B.L.,Ravert,R.D.,Williams,M.K.,Agocha,V.B.,…Donnellan,M.B.(2011).TheQuestionnaireforEudaimonicWell-Being:Psychometricproperties,demographiccomparisons,andevidenceofvalidity.TheJournalofPositivePsychology,5,41-61.doi:10.1080/17439760903435208

*Watson,D.,Clark,L.A.,&Tellegen,A.(1988).Developmentandvalidationofbriefmeasuresofpositiveandnegativeaffect:ThePANASscales.JournalofPersonalityandSocialPsychology,54(6),1063-1070.doi:10.1037/0022-3514.54.6.1063

149

Chapter12:AssessingProfessor-StudentRelationshipsUsingSelf-ReportScalesJennaM.MeyerbergandAngelaM.Legg

PaceUniversity

Professor-studentrelationshipsareanimportant,andsometimesunderestimated,partofcollegestudents'experiences.Apositiverelationshipbetweenprofessorandstudentcanyieldgreaterstudentmotivation,learning,classattendance,effort,andoverallsatisfaction(Benson,Cohen,&Buskist,2005;Wilson&Ryan,2013).Professorscanengagestudentsthroughtheirbehaviorsandattitudestowardstudents.Positivebehaviorscanmakestudentsfeelcaredforandfosteraninterestintheclass,motivatingthemtodobetter,resultinginthepositiveresultsmentionedabove(Murray,1997).Thebenefitsofpositiveprofessor-studentrelationshipscancontinueevenafterastudentgraduates.Collegegraduateswhoreporthavinghadatleastoneprofessorwhocaredaboutthemduringcollegeare1.9timesmorelikelytofeelengagedatwork(Gallup,Inc.,2014).Unsurprisinglythen,examiningprofessor-studentrelationshipsthroughtheperspectiveofscholarshiponteachingandlearning(SoTL)isafruitfulendeavor(Wilson,Wilson,&Legg,2012).Manyfactorsinfluencethedevelopmentofrapportandpositiverelationshipsbetweenprofessorsandstudents.Evenpositivefirstimpressionsdevelopedpriortoorbeforethefirstdayofclasscanhavefar-reachingeffectsintermsofincreasedstudentmotivation,retention,andgrades(Legg&Wilson,2009;Wilson&Wilson,2007).Alongwithexamininghowpositiveprofessor-studentrelationshipsdevelop,scholarscanalsoexaminewhattypesofoutcomesariseduetothesepositiveinteractions.A.Richmondandcolleagues,forexample,measuredstudents'evaluationsoftheirprofessors'humor(seetheTeacherHumorScalebelow),perceivedrapport(seetheProfessor-StudentRapportScalebelow),studentengagement,andastandardstudentratingofinstructionforteachereffectiveness(A.Richmond,Berglund,Epelbaum,&Klein,2015).Theseresearchersdemonstratedthatprofessor-studentrelationshipscontributeagreatdealtotheperceptionsofandexperiencesofcollegestudents.Infact,professor-studentrapportaloneaccountedfor54%ofthevariancealoneandthesetofvariablesaccountedfor59%.Scholarstypicallyassessprofessor-studentrelationshipsthroughtheadministrationofself-report,surveymeasuresgiventostudents.Measuresmayfocusontherelationshipitself(Wilson,Ryan,&Pugh,2010)ormayassessotheraspectsoftherelationshipsuchasimmediacybehaviorsoruseofhumor(Frymier,Wanzer,&Wojtaszczyk,2008;Keeley,Smith,&Buskist,2006).Basedontheextantliteraturewithintheprofessor-studentrelationshiparea,wecreatedanillustration(seeFigure1)todemonstratetheconnectionsbetweenthethreemainrelationship-relatedconstructsdiscussedinthischapter:1)immediacybehaviors,2)rapport,and3)thelearningalliance(i.e.,qualitiesoftheprofessor-studentrelationshipandstudentinvestment;Rogers,2012).Usingthisliterature,wetheorizethateachconstructcontributestothenextbroaderconstruct(Rogers,2012;2015;Wilson,Ryan,&Pugh,2010).Thus,verbalandnonverbalimmediacybehaviorsaretwoaspectsthatbuildprofessor-studentrapportand

150

professor-studentrapportformsafoundationforthelearningalliance.Researchalsopointstotherelationshipbetweenthesevariablesanddesiredstudentoutcomessuchasthepositiverelationshipbetweeninstructors’expressionofnonverbalimmediacybehaviorsandstudents’coursegrades(LeFebvre&Allen,2014).Atbroaderlevels,rapportandthedevelopmentofalearningallianceleadtopositiveoutcomessuchasstudentmotivation,attitudestowardthecourseandprofessor,andaffectiveandcognitivelearning(Rogers,2015;Wilsonetal.2011).

Figure1.Constructconceptualizationofimmediacybehaviors,professor-studentrapport,learningalliance,andpositivestudentoutcomes.Basedonthisframework,webeginourdiscussionwithfourscalesthatassessimmediacybehaviors:theImmediacyScale(Gorham,1988),theNonverbalImmediacyScale(V.Richmond,McCroskey,&Johnson,2003),theTeacherHumorScale(Frymieretal.,2008),andtheTeacherBehaviorsChecklist(Keeleyetal.,2006).Wethenconsiderthenextlargestconstruct,rapport,bydiscussingtheProfessor-StudentRapportScale(Wilsonetal.,2010).Weendourscaleanalysiswiththecurrentconceptionofthelargestconstruct,thebond,bydescribingtheLearningAllianceInventory(Rogers,2012).Followingreviewofeachmeasure,exemplarsareprovidedfromtheSoTLliteraturetodemonstratepracticalorempiricalusage.Thediscussion

PositiveStudentOutcomesBeneficialStudentBehaviors(e.g.,ClassAttendance)

AffectiveLearningCognitiveLearning

MotivationStudentRatingsofInstruction

LearningAlliance

Professor-StudentRapport

ImmediacyBehaviorsVerbalimmediacycues(humor,usesstudentstteames,asksquestions)Nonverbalimmediacycues(smiles,leansforward,handgestures,touch)

151

endswithsomeunansweredquestionsregardingtheassessmentofprofessor-studentrelationshipsandcallforfutureresearchinthisarea.ImmediacyScaleImmediacyisthenonverbalandverbalbehaviorsthatcommunicateliking,caring,andapproachability(Mehrabian,1967;1969).Withregardtotheclassroom,immediacyreferstostudents’impressionsoftheirinstructors’availabilityorpsychologicaldistance.Earlydiscussionsofmeasuringimmediacycameaboutinthelate1960s,withafocusoncapturingverbalandnonverbalcommunicationsofcloseness(Mehrabian,1969).Mehrabianidentifiedfiveprimaryimmediacybehaviors:touch,physicaldistancebetweencommunicatorandaudience,leaningforward,makingandmaintainingeyecontact,andphysicalorientationtowardaudience.Whileimmediacydoesnotinandofitselfrepresentrapport(immediacyisbehavior-basedwhereasrapportrelatesmoretocognitiveperceptions),itisconsideredacatalystandpredictorofrapportandrelationship-building(seeabovediscussionofFigure1).Immediacybehaviorsintheclassroomreceivedfurtherexplorationinthelate1980s(Gorham,1988).ExpandingonthefivenonverbalbehaviorsoutlinedbyMehrabian(1969),Gorhamcreatedoneofthefirstvalidatedimmediacyscalesthatincludedbothnonverbalandverbalconstructs.The34-itemscalecontains20verbalitems(e.g.,“addressesstudentsbyname,”“useshumorinclass,”and“asksquestionsthatsolicitviewpointsoropinions)and14nonverbalitems(e.g.,“gestureswhiletalkingtoclass,”and“smilesattheclassasawhole,notjustindividualstudents”).Studentsrespondbyratingthefrequencywithwhicheachbehavioroccursonascaleof0(neveroccurs)to4(occursveryoften).Thisvalidationstudydemonstratedsplit-halfreliabilitiesof.94fortheverbalitemsand.84forthenonverbalitems.Theoriginalscaleunderwentsomecriticismwhenscholarsarguedthatverbalbehaviorsrepresentedteachereffectiveness,notimmediacy(Robinson&Richmond,1995).Inspiredbythiscriticism,WilsonandLocker(2008)setouttoempiricallyaddressthisargument.Theirvalidationprovidedevidencefordiscriminantvaliditybetweenimmediacyandteachereffectiveness.Further,theiranalysisofGorham’s(1988)originalscaleproducedfourdistinctfactors:1)individualfriendliness(8items,α=.88),2)flexibilityduringlecture(5items,α=.80),3)nonverbalimmediacy(7items,α=.76),andasingleitemassessingwhetherprofessorsinvitestudentstoaddressthembytheirfirstname.Further,basedontheiranalysis,WilsonandLocker(2008)recommendedexcludingthefollowingthreeitems:1)“asksquestionsorencouragesstudentstotalk”,2)“referstoclassas‘our’classorwhat‘we’aredoing”,and3)“invitesstudentstotelephoneormeetoutsideofclassiftheyhaveaquestionorwanttodiscusssomething”fromGorham’s(1988)originalscaleastheseitemsdidnotaddanypredictivevaluetothescale.ThemostfinalversionofthisscaleisWilsonandLocker(2008).NonverbalImmediacyScaleWhereastheupdatedimmediacyscale(Gorham,1980;Wilson&Locker,2008)includesmeasuresofbothverbalandnonverbalpsychologicalavailability,theNonverbalImmediacyScale(NIS)focusesonnonverbalcuesspecifically(V.Richmondetal.,2003).Thisscale,whichis

152

notlimitedtoprofessor-studentrelationships,containsatotalof26itemsdrawnfromthepreviousliterature(13positivelywordeditemsand13negativelywordsitems).AnadvantagetotheNISistheabilitytodistributeanother-reportingversion(NIS-O)aswellasaself-reporting(NIS-S)version.AlthoughtheitemsontheNIS-OandNIS-Smeasurethesameconstructs,theirwordingvariesslightlyinthattheNIS-Oframesitemsintermsofwhat"I"doandtheNIS-Sframesitemsintermsofwhats/he(theprofessor)does.Participantsratethefrequencyofthebehaviorusinga5-pointLikert-typescalefrom0(never)to5(veryoften).Sampleitemsfromthescaleinclude,"IsmilewhenItalktopeople,""Iuseamonotoneordullvoicewhiletalkingtopeople,"and"Iavoideyecontactwhiletalkingtopeople."Negativebehaviorssuchasusingamonotonevoiceandavoidingeyecontactarereversescored.ThisscaleyieldedhighinternalreliabilityestimatesfortheNIS-SandNIS-Oof.90-.93acrossseveraltargets(self,teacher,supervisor,andaromanticdate).LeFebvreandAllen(2014)providedevidenceforcriterionvalidityoftheNonverbalImmediacyScale.Intheirstudy,studentsenrolledinlecture/laboratoryorself-containedcoursescompletedtheNIS,ameasureofaffectivelearning,aninstructionevaluationmeasure,andallowedtheirgradestobetracked.Studentswhoperceivedgreaternonverbalimmediacyfromtheirteachingassistantsalsoreceivedhighergradesinthecourses,reportedmoreaffectivelearning(e.g.,likingoftheteachingassistant,wantingtoenrollinanothercoursewiththatteachingassistant),andprovidedmorepositiveinstructionevaluations.Asidefromassessingstudents’perceptionsofprofessors’nonverbalimmediacybehaviors,studentsmayalsobenefitfromusingthisscalewhendiscussingnonverbalcommunication,person-perceptionaccuracy,anduniversalversusculturalunderstandingsofemotionalexpression.BecausetheNIS(V.Richmondetal.,2003)hasaself-assessmentversionandanother-assessmentversion,studentscanratetheirownnonverbalimmediacybehaviorsandaskaclosefriendandanewacquaintancetoalsoratethem.Throughthisactivity,studentscanlearnaboutmeasurementerror,self-vs.other-ratingmethods,andconvergentvalidity.TeacherHumorScaleHumorisonecomponentofverbalimmediacythatmayleadtoincreasesinrapportwhenusedinacontext-appropriateandrespectfulmanner.Researchshowsthattheuseofhumorbytheprofessorcorrespondstoteachers’immediacybehaviorsintheclassroom,andhasanimpactonlearningoutcomes(Gorham&Christophel,1990).Highratingsinprofessorhumororientation,orthepredispositionforaprofessortoengageinhumorouscommunication,areassociatedwithgreaterpositiveaffecttowardtheprofessoraswellasincreasedperceivedlearning.Thereisalsoapositivecorrelationbetweenperceivedprofessorhumorandnonverbalimmediacybehaviorsandresponsivenesstostudents(Wanzer&Frymier,1999).Animportantpartofsuccessfullyusinghumor,especiallywithinaclassroomsetting,istheneedtounderstandhowtheaudience(e.g.,thestudents)interpretshumor.Torok,McMorris,andLin(2004)conductedastudyexploringstudents’perceptionsofprofessorhumor,specificallyhopingtofindthereasoningbehindthepositiverelationshipbetweenprofessorhumorandstudentengagement.Seventythreepercentofstudentsquestionedstatedthattheyfeltvery

153

positivelyabouttheirprofessors’useofhumor,with59%stronglybelievingthatuseofhumorencouragesasenseofcommunitywithintheclassroom.Regardingimpactonabilitytolearn,80%ofstudentsfeltthattheuseofhumorhelpedthemmasteraconcept.AsGladding(1995)noted,though,notallhumorispositive.Thetypesofhumormostoftenendorsedbystudentsasbeingwelcomeare,“funnystories,funnycomments,jokes,andprofessionalhumor,”(Toporeketal.,2004,pp.16).Humorthatstudentsreportedastheleastenjoyablewereaggressiveorhostilehumor,andhumorofanethnicorsexualnature.Althoughtheimmediacyscalesdiscussedpreviously(Gorham,1988;Wilson&Locker,2008)bothincludeitemsaboutprofessors'useofhumor,theTeacherHumorScale(THS;Frymieretal.,2008)isanimportanttoolforresearchersseekingtoassessthisimmediacybehaviormorespecifically.Further,becausehumorcomesinmanyforms,theTHSprovidesresearcherswithatooltoassesswhichtypesofhumorpositivelyinfluenceprofessor-studentrelationshipsandwhichtypesshouldbeavoided.TheTeacherHumorScale(Frymieretal.,2008)canbeusedtomeasuredifferenttypesofhumor,forexample,course-relatedandcourse-unrelatedhumor,self-disparaginghumor,andunplannedhumor.TheTHSisa41-itemscalethatmeasuresstudents’perceptionsoftheappropriatenessorinappropriatenessofaprofessor’suseofhumor.Studentsrespondona5-pointLikert-typescalerangingfrom1(veryinappropriate)to5(veryappropriate).Afactoranalysisofthisscaleyieldedafivefactorsolutionreflectingfivedistinctformsofhumorfor25oftheoriginalitems:1)OtherDisparagingHumor(9items,α=.93),2)RelatedHumor(7items,α=.85),3)UnrelatedHumor(3items,α=.85),4)OffensiveHumor(3items,α=.84),and5)Self-DisparagingHumor(3items,α=.80).OtherDisparagingHumorincludesitemssuchas,“Makeshumorouscommentsaboutastudent’spersonallifeorpersonalinterests.”AnexampleoftheRelatedHumorconstructis,“Usesfunnypropstoillustrateaconceptorasanexample.”UnrelatedHumorincludesitemssuchas,“Usescritical,cynical,orsarcastichumoraboutgeneraltopics(notrelatedtothecourse).”OffensiveHumorexamplesinclude,“Usesvulgarlanguageornonverbalbehaviorsinahumorousway.”Finally,Self-DisparagingHumorincludes,“Makesfunofthemself[sic]whentheymakemistakesinclass."TeacherBehaviorsChecklistOneofthecriticismsofearlyimmediacyscalesisthattheymerelymeasuredteachereffectiveness,notimmediacy(Robinson&Richmond,1995).TheTeacherBehaviorsChecklist(TBC;Buskist,Sikorski,Buckley,&Saville,2002)wascreatedinordertoassessqualitiesofprofessorswhoarehighlyesteemedintheeyesoftheirstudentsandincludesbehaviorsthatalsocancommunicateimmediacy(seeKirk,Busler,Keeley&Buskist,2015)foradditionaldiscussionofthismeasure).Tocreateascalethatcouldserveasanevaluativeinstrument,theoriginal28-itemlistofbehaviorswascreatedsuchthatstudentscouldratetheextenttowhichtheirprofessorsengageinthechecklistbehaviorsona5-pointLikert-typescalefromA(frequentlyexhibitsthesebehaviors)toE(neverexhibitsthesebehaviors)(Keeleyetal.,2006).Sampleitemsinclude,“EnthusiasticAboutTeachingandAboutTopic(Smilesduringclass,preparesinterestingclassactivities,usesgesturesandexpressionsofemotiontoemphasizeimportantpoints,andarrivesontimeforclass),”“Rapport(Makesclasslaughthroughjokes

154

andfunnystories,initiatesandmaintainsclassdiscussions,knowsstudentnames,andinteractswithstudentsbeforeandafterclass),”and“Respectful(Doesnothumiliateorembarrassstudentsinclass,ispolitetostudents,doesnotinterruptstudentswhiletheyaretalking,anddoesnottalkdowntostudents)."Factoranalysisrevealedonefactorencompassingprofessionalcompetencyandcommunicationskills(11items,α=.90)andanotherfactormoreindicativeofrapportwhichtheauthorsrefertoasthecaringandsupportivesubscale(13items,α=.93).Althoughonly24itemsloadedontothetwofactors,theauthorsrecommendadministeringthefull28-itemscaleandcalculatingthreescores,atotalscore,acaringandsupportivescore,andaprofessionalcompetencyandcommunicationskillsscore.AlthoughmostSoTLresearchexaminesstudents'perceptionsoftheiractualprofessors,theTBCandmanyoftheotherscaleswedescribeinthischaptercanbeusedforhypotheticalorimaginedprofessorsaswell.Forexample,studentscompletedtheTBCandseveralotherquestionnaireswhilethinkingabouttheiridealprofessor,onewhommaynotevenexist(Komarraju,2013).Herstudyprovidedevidencethatdifferentstudentsviewtheidealprofessorindifferentways.Studentswithlowself-efficacyandextrinsicmotivationsplacedmoreimportanceonhavingcaringprofessorsthanstudentswhoreportedhighself-efficacyandintrinsicmotivations.ThisstudyfurtherhighlightstheflexibilityoftheTBCasascalethatcanfacilitateresearchonboththepredictorsandoutcomesofprofessor-studentrelationships.Professor-StudentRapportScaleAsdescribedpreviouslyandillustratedinFigure1,immediacybehaviorscontributetoperceptionsofprofessor-studentrapport.Upuntilthispoint,ourcoveragepredominantlyhasfocusedonscalesthatassessthesesmallerconstructsthatpredictrapportandpositiveprofessor-studentrelationships.However,rapportrepresentsmuchmorethanjustbehaviorsandpsychologicalavailability.TheProfessor-StudentRapportScale(PSRS;Wilsonetal.,2010)wasdevelopedtoaddressthepotentiallylimitingfactorofaddressingonlyimmediacy(seeFigure1forourconceptualizationoftheseconstructs).“Rapport”isbestunderstoodastheagreementorharmonybetweentwopeople(Wilsonetal.,2010).Rapporthashistoricallybeenmeasuredwithinthetherapist-clientrelationship,withnomeasuresspecificallyaddressingrapportinanacademicsetting,makingthePSRSunique.Further,thePSRSisdistinguishablefrommeasuresofimmediacybyprovidingalargerscopeofbehaviors;ifrapportisthepositiverelationshipbetweenprofessorandstudent,thenimmediacybehaviorsareonewayofachievingthisbond(Wilsonetal.,2010).The34-itemscaleassessestheprofessor-studentrelationshiponascaleof1(stronglydisagree)to5(stronglyagree).Thescalehasexcellentinternalreliability(α=.96).Exampleitemsinclude,“Myprofessorisunderstanding,”“Myprofessor’sbodylanguagesays,‘Don’tbotherme,’”and“MyprofessorisawareoftheamountofeffortIamputtingintothisclass."Further,thescalepositivelycorrelatedwithprofessorfriendliness,flexibility,andnonverbalbehaviorsThePSRSalsopredictedstudents'attitudestowardtheircourseandprofessoraswellastheirmotivation,withhigherscoresonthePSRSpredictingpositivestudentattitudesandself-identifiedlevelsofmotivation(Wilsonetal.,2010).

155

ThePSRSunderwentfurthervalidationastheresearcherssoughttoreplicatetheinternalconsistencydemonstratedintheirinitialvalidationpaper,establishtest-retestreliability,andprovidefurtherconvergentvalidation(Ryan,Wilson,&Pugh,2011).Asexpected,thescalemaintainedahighlevelofinternalreliabilitywithaCronbach'salphaof.89andadequatetest-retestreliability(r=.72).ConvergentvalidityforthisscalewasassessedusingTheWorkingAllianceInventory(Horvath&Greenberg,1989),perceivedsocialsupport,andverbalaggressiveness.Asexpected,thePSRSpositivelycorrelatedwithperceivedprofessor-studentrelationshipsandperceivedsocialsupportbutnegativelycorrelatedwithverbalaggressiveness.Althoughtheoriginal34-itemmeasure(Wilsonetal.,2010)willworkformanyresearchers,somemayrequireashortermeasureofrapport.Furtherinvestigationledtoashorter15-itemmeasurewithnineitemsassessing"perceptionsofteacher"(α=.92)andsixitemsassessing"studentengagement"(α=.84;Wilson&Ryan,2013).Mostnotably,thesix-item"studentengagement"subscaleemergedasthestrongestpredictorofperceivedteachereffectiveness,attitudetowardtheteacher,studentmotivation,attitudetowardthecourse,andperceivedlearning.Asafinalcriterionvalidityindicator,studentengagementsignificantlypredictedstudents'coursegrades.ThePSRS,especiallythesix-itemversion,isashort,easytoadministerscale.Werecommendincorporatingthisscaleintoclassroomdiscussionsaboutimpressionformation,likingandrelationshipformation,andthehaloeffect.ThePSRSisagoodtooltoillustratehowpeoplecanuseheuristicsandstereotypeswhenformingimpressionsofothers.Forexample,participantswhoviewedolderandyoungerphotographsofamaleorfemaleprofessorgavelower(i.e.,morenegative)ratingstotheolderprofessors(Wilson,Beyer,&Monteiro,2014).Studentsalsotendedtogivelowerratingstotheolderfemaleprofessor.Usingthisstudy,professorscandiscussthePSRS,impressionformation,andthewayinwhichpeopleuseuncontrollableattributes(e.g.,age)toassessothers.Thisexamplecanopenupadiscussionofthevariouswayspeoplecreaterapport(e.g.,byusingimmediatebehaviorsorhumororbyimprovingtheirphysicalattractiveness).LearningAllianceInventoryJustasthePSRSderivedfrommeasuresintheclinicaldomain,theLearningAllianceInventory(LAI;Rogers,2012)alsoreceivedinspirationfromtheclinicalconceptoftheworkingalliance.This18-iteminventoryusesa7-pointLikert-typeresponsescaleinwhichparticipantsratetheextenttowhichthestudentendorseseachstatementfrom1(notatall)to7(verymuchso).Exampleitemsinclude,“MyteacherandIworkwelltogether,”“Myteacherwelcomesallstudentinputandfeedback,”and“Ienjoydoingtherequiredtasksforthiscourse.”TheLAIcontainsthreesubscales,1)collaborativebond(6items,α=.91),2)teachercompetency(6items,α=.93),and3)studentinvestment(6items,α=.95).Thescale’sinitialvalidationdemonstratedadequatetest-retestreliabilitywithreliabilitiesforeachofthesubscalesthatareinlinewithothersimilarscales(rs=.63-.73)(e.g.,Ryanetal.,2011).Additionally,allthreesubscalesshareasmalleffectinpredictingcoursegrades(rs=.19-.25).Rogers(2015)providedfurthervalidationofhisscaleandspecificallycomparedtheLAItothePSRS(Wilsonetal.,2010)andtheNIS(V.Richmondetal.,2003).TheLAIdidpositivelycorrelatewithbothscales,as

156

expected,thoughtheLAIsharedastrongerrelationshipwiththePSRSthanwiththeNIS.Further,thecollaborativebondsubscalesignificantlycontributedtothestudents'self-reportedlearningandactualcoursegradeevenaftercontrollingfortheNISandPSRS.Figure1providesaconstructconceptualizationoftherelationshipsbetweenimmediacybehaviors,rapport,perceptionsofalearningalliance,andpositivestudentoutcomes.Ofnote,severalmediationalrelationshipsappearinthefigure.Forexample,immediacybehaviorshavedirecteffectsonlearningbutalsohaveindirecteffectsthroughtheirconnectionstorapportandthelearningalliance(Rogers,2015).AlthoughRogersprovidedanexcellentvalidationoftheLearningAllianceInventory,itsuseintheSoTLliteratureisnascent.Werecommenditsincorporationinfutureresearchthatexaminesrapport,professor-studentrelationships,andtheeffectsofpositiverelationshipsonstudents'andprofessors'outcomes(e.g.,gradesforstudents,burnoutforprofessors).FutureDirectionsThestudyofprofessor-studentrelationshipshascomealongwaysinceMehrabianbeganoperationalizingimmediacyinthe60s.Hundredsofstudiesexistontherelationshipsbetweenimmediacybehaviors,instructors'useofhumor,professor-studentrapport,andperceptionsofalearningalliance.Further,eachoneoftheseconstructspredictpositivestudentoutcomes.Despitethevastattentiongiventothevalidandreliablemeasurementofprofessor-studentrelationships,severalimportantfuturedirectionsexist.Giventheincreaseinonlineclassesemail,learningmanagementsystems,andsocialmedia,studentsnowfrequentlyinteractwithprofessorsoverelectronicmodes.Onedisadvantageofthescaleswedescribedisthatsomeoftheitemsmaynottranslatewellinanonlinecourse(e.g.,nonverbalimmediacyitemsmeasuringeyecontactandphysicaldistance).Atthepresenttime,acomputer-mediated-communicationscalethatassessesprofessor-studentrelationshipsdoesnotexist,despiteevidencethatprofessorbehaviorscanhinderorhelprapportbuildingthroughonlineclasses(Arbaugh,2001)orthroughtheuseofelectroniccommunication(Legg&Wilson,2009).Futurescaleswouldbenefitfromincludingitemsthataddressonlineclassesandelectroniccommunication,forexample:“Instructorrespondswithinareasonableamountoftimetomyelectronicrequests,”“Myinstructorshowss/hecaresaboutmeintheways/herespondstomyemails,”or“Icantellmyinstructorcaresaboutstudentsbytheways/hedesignedourLearningManagementSystem(e.g.,Blackboard).”Thevastmajorityofresearchonprofessor-studentrelationshipsexaminestherelationshipformedbetweenundergraduatestudentsandtheirprofessors.However,thesefindingsmaynotgeneralizetotherelationshipformedandexperiencedbygraduatestudentsandtheiradvisors.Forexample,thesameamountofwarmthandcaringthatmaypromotelearningforundergraduatestudentswholackintrinsicmotivation(Komarraju,2013)maybeseenascoddlingorlackinginrigorwithinthecontextofgraduateeducation.Comparedtoundergraduates'relationshipswiththeirprofessors,graduatestudentsandtheiradvisorsmayspendmoretimetogether,bothinacademicandcasualsettings.Thisaddedtimemayfacilitatebuildinglearningalliancesandrapportbutmayalsoposesomeadditionalchallengesthatcould

157

helporhinderthegraduatestudents'success.Inonerarestudyfocusingonagraduatepopulation,graduatestudentslistedinterest/supportandcharacteristicssuchassenseofhumorandempatheticassomeoftheirtopqualitiesinagoodgraduatementor(Cronan-Hillix,Gensheimer,Cronan-Hillix,&Davidson,1986).Thisstudy,however,didnotcomparegraduatestudents'perceptionswiththoseofundergraduates.Werecommendthatfutureresearchassessthesimilaritiesanddifferencesbetweengradstudentsandundergrads.Inaddition,futureresearchcanapplytheexistingvalidatedmeasurestothegraduatestudent-professordyadordevelopnewmeasurestoassessthisuniquerelationship.ConclusionAlargeamountofresearchnowpointstothecriticalinfluenceofpositiveprofessor-studentrelationships(A.Richmondetal.,2015;Rogers,2015;Wilson&Ryan,2013).Studentslearnbestwhentheyhavecaringandcompetentmentorstofacilitatetheirlearning,sparktheirmotivation,andprovideemotionalandcognitivesupport.ItisourhopethatthischapterprovidesaresourceandfoundationfornewSoTLresearcherswhowishtomeasurethepredictorsandoutcomesofpositiveprofessor-studentrelationships.

158

ReferencesReferencesmarkedwithanasteriskindicateascale.Arbaugh,J.B.(2001).Howinstructorimmediacybehaviorsaffectstudentsatisfactionand

learninginweb-basedcourses.BusinessCommunicationQuarterly,64(4),42-54.doi:10.1177/108056990106400405

Benson,A.T.,CohenL.A.,&Buskist,W.(2005).Rapport:Itsrelationtostudentattitudesandbehaviorstowardteachersandclasses.TeachingofPsychology,32(4),237–239.doi:10.1080/00986283.2010.510976

*Buskist,W.,Sikorski,J.,Buckley,T.,&Saville,B.K.(2002).Elementsofmasterteaching.InS.F.Davis&W.Buskist(Eds.),Theteachingofpsychology:EssaysinhonorofWilbertJ.McKeachieandCharlesL.Brewer(pp.27–39).Mahwah,NJ:LawrenceErlbaumAssociates,Inc.

Christophel,D.M.(1990).Therelationshipsamongteacherimmediacybehaviors,studentmotivation,andlearning.CommunicationEducation,39(4),323–340.doi:10.1080/03634529009378813

Christophel,D.M.,&Gorhamn,J.(1995).Atest-retestanalysisofstudentmotivation,teacherimmediacy,andperceivedsourcesofmotivationanddemotivationincollegeclasses.CommunicationEducation,44(4),292–306.doi:10.1080/03634529509379020

Cronan-Hillix,T.,Gensheimer,L.K.,Cronan-Hillix,W.A.,&Davidson,W.S.(1986).Students'viewsofmentorsinpsychologygraduatetraining.TeachingofPsychology,13(3),123-127.doi:10.1207/s15328023top1303_5

*Frymier,A.B.,Wanzer,M.B.,&Wojtaszcyk,A.M.(2008).Assessingstudents’perceptionsofinappropriateandappropriateteacherhumor.CommunicationEducation,57(2),266–288.doi:10.1080/03634520701687183

Gallup,Inc.(June1,2014).Greatjobs,greatlives:The2014Gallup-PurdueIndexInauguralnationalreport.Retrievedfrom:http://www.gallup.com/services/176768/2014-gallup-purdue-index-report.aspx

Gladding,S.T.(1995).Humorincounseling:Usinganaturalresource.JournalofHumanisticEducation&Development,34(1),3–13.doi:10.1002/j.2164-4683.1995.tb00106.x

*Gorham,J.(1988).Therelationshipbetweenverbalteacherimmediacybehaviorsandstudentlearning.CommunicationEducation,37(1),40–53.doi:10.1080/03634529009378813

Gorham,J.,&Christophel,D.M.(1990).Therelationshipofteachers’useofhumorintheclassroomtoimmediacyandstudentlearning.CommunicationEducation,39(1),46–62.doi:10.1080/0363452900937878

Horvath,A.O.,&Greenberg,L.S.(1989).DevelopmentandvalidationoftheWorkingAllianceInventory.JournalofCounselingPsychology,36(2),223-233.doi:10.1037/0022-0167.36.2.223

*Keeley,J.,Smith,D.,&Buskist,W.(2006).Theteacherbehaviorschecklist:Factoranalysisofitsutilityforevaluatingteaching.TeachingofPsychology,33(2),84–91.doi:10.1207/s15328023top33021

Kirk,C.,Busler,J.,Keeley,J.&Buskist,(2015).Effectivetoolsforassessingcharacteristicsofexcellentteaching:Theteacherbehaviorschecklist.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

159

Komarraju,M.(2013).Idealteacherbehaviors:Studentmotivationandself-efficacypredictpreferences.TeachingofPsychology,40(2),104-110.doi:10.1177/0098628312475029

LeFebvre,L.,&Allen,M.(2014).Teachingimmediacyandstudentlearning:Anexaminationoflecture/laboratoryandself-containedcoursesections.JournalofScholarshipofTeachingandLearning,14(2),29-45.doi:10.14434/josotl.v14i2.4002

Legg,A.M.,&Wilson,J.H.(2009).E-mailfromprofessorenhancesstudentmotivationandattitudes.TeachingofPsychology,36(3),205-211.doi:10.1080/00986280902960034

Mehrabian,A.(1967).Attitudesinferredfromnonimmediacyofverbalcommunication.JournalofVerbalLearningandVerbalBehavior,6(2),294-295.doi:10.1016/S0022-5371(67)80113-0

Mehrabian,A.(1969).Somereferentsandmeasuresofnonverbalbehavior.BehaviorResearchMethodsandInstrumentation,1(6),203–207.doi:10.3758/BF03208096

Murray,H.(1997).Effectiveteachingbehaviorsinthecollegeclassroom.InR.Perry&J.Smart(Eds.),Effectiveteachinginhighereducation:Researchandpractice(171–204).NewYork:Agathon.

Richmond,A.S.,Berglund,M.B.,Epelbaum,V.B.,&Klein,E.M.(2015).A+(b1)professor–studentrapport+(b2)humor+(b3)studentengagement=(Y)studentratingsofinstructors.TeachingofPsychology,42(1),119–125.doi:10.1177/0098628315569924

*Richmond,V.P.,McCroskey,J.C.,&Johnson,A.D.(2003).Developmentofthenonverbalimmediacyscale(NIS):Measuresofself-andother-perceivednonverbalimmediacy.CommunicationQuarterly,51(4),504-517.doi:10.1080/01463370309370170

Robinson,R.Y.,&Richmond,V.P.(1995).Validityoftheverbalimmediacyscale.CommunicationResearchReports,12(1),80-84.doi:10.1080/01463370309370170

*Rogers,D.T.(2012).Thelearningallianceinventory:Instrumentdevelopmentandinitialvalidation.InternationalJournalfortheScholarshipofTeachingandLearning,6(1),1-16.

Rogers,D.T.(2015).Furthervalidationofthelearningallianceinventory:Therolesofworkingalliance,rapport,andimmediacyinstudentlearning.TeachingofPsychology,42(1),19–25.doi:10.1177/0098628314562673

Ryan,R.G.,Wilson,J.H.,&Pugh,J.L.(2011).Psychometriccharacteristicsoftheprofessor-studentrapportscale.TeachingofPsychology,38(3),135–141.doi:10.1177/0098628311411894

Torok,S.E.,McMorris,R.F.,&Lin,W.(2004).Ishumoranappreciatedteachingtool?Perceptionsofprofessors'teachingstylesanduseofhumor.CollegeTeaching,52(1),14–20.doi:10.3200/CTCH.52.1.14-20

Wanzer,M.B.,&Frymier,A.B.(1999).Therelationshipbetweenstudentperceptionsofinstructorhumorandstudents’reportsoflearning.CommunicationEducation,48(1),48-62.doi:10.1080/03634529909379152

Wilson,J.H.,Beyer,D.,&Monteiro,H.(2014).Professorageaffectsstudentratings:Haloeffectforyoungerprofessors.CollegeTeaching,6(1),20-24.doi:10.1080/87567555.2013.825574

*Wilson,J.H.,&LockerJr.,L.(2008).Immediacyscalerepresentsfourfactors:Nonverbalandverbalcomponentspredictstudentoutcomes.JournalofClassroomInteractions,42(2),4-10.

160

*Wilson,J.H.,Ryan,R.G.,&Pugh,J.L.(2010).Professor-studentrapportscalepredictsstudentoutcomes.TeachingofPsychology,37(4),246-251.doi:10.1080/00986283.2010.510976

Wilson,J.H.,&Ryan,R.G.(2013).Professor-studentrapportscale:Sixitemspredictstudentoutcomes.TeachingofPsychology,40(2),130–133.doi:10.1177/0098628312475033

Wilson,J.H.,&Wilson,S.B.(2007).Thefirstdayofclassaffectsstudentmotivation:Anexperimentalstudy.TeachingofPsychology,34(4),226–230.doi:10.1080/00986280701700151

Wilson,J.H.,Wilson,S.B.,&Legg,A.M.(2012).Buildingrapportintheclassroomandstudentoutcomes.InB.M.Schwartz&R.A.R.Gurung(Eds.),Evidence-basedteachingforhighereducation(pp.23-38).Washington,DC:AmericanPsychologicalAssociation.

161

Chapter13:EffectiveToolsforAssessingCharacteristicsofExcellentTeaching:TheTeacherBehaviorsChecklistasExemplarClaireKirk1,JessicaBusler1,JaredKeeley2,andWilliamBuskist1

1AuburnUniversity,2MississippiStateUniversity

Learningtheintricatecraftofteachingrequiresthatteachersundergofrequentandmultiple-sourcedassessmenttoimprovetheireffectiveness.Fortunately,toolsandtechniquesforevaluatinginstructionarenumerous.Beneficialandcomprehensivefeedbackisbestderivedfromimplementingseveraldifferentassessmentmethodsacrossmultipleinformants,whichmayincludestudents,colleagues,andself-reflection.Gatheringinformationfromthesesourcesregardingone’steachingisimportantbecauseeachcontributesauniqueperspectivetounderstandingone’steachingabilities(Keeley,2012).Thischapterwillbrieflyreviewsomegeneralstrategiesregardingtheassessmentofteachingcharacteristicssuchasfeedbackfromstudentsandpeers,aswellasactualteachingmaterials,followedbyanin-depthexaminationofonemeasure:theTeacherBehaviorsChecklist(Buskist,Sikorski,Buckley,&Saville,2002;Keeley,Smith,&Buskist,2006). MeasuringTeachingQuality—MultimodalAssessmentThemostcommonlyusedteachingassessorsarenaturallytheindividualsbeingtaught:students.Theyexperienceandobserveanyparticularteacherquitefrequently,andcanbeanexcellentsourceofinformation.However,gatheringstudents’feedbackonone’steachingiscontroversialduetotheirlackofobjectivity.Additionally,studentsareoftenunabletoaccuratelyassesstheamountofinformationtheyarelearning,orhowwellthatknowledgewillberetainedovertime.Relyingsolelyonstudentstoprovideinsightaboutone’steachingisalsocontroversialduetothelackofobjectivityamongstudents.Forexample,thewayinwhichstudentsperceivetheirprogressinacoursemayvaryasafunctionoftheiruniqueindividualexperienceswiththecoursematerialandtheinstructor.Factorssuchasgradingleniencyanddifficultyofthematerialalsomaypotentiallyaffectstudentratings(Ellis,Burke,Lomire,&McCormack2004;Marsh&Roche,2000).Regardless,studentperspectivesofteachingandlearningarevaluableinthattheymaybecomparedtoinstitutionalandteacher-developedmeasuresoflearning(Keeley,2012).Toincreasethevalidityofstudentevaluationsofteaching,severalmethodsmaybeusedinconjunction,includingmini-assessments,ratingscales,gradedassignmentsandtests,andstudentfocusgroups.

Mini-AssessmentsThereareavarietyofsimpleandquickassessmentsthatstudentscancompleteinclasstohelpteachersgatherbasicinformationonstudentlearning(Angelo&Cross,1993).Oneexampleisthe“minutepaper,”whichisaquicklywrittenresponsetoatopicorquestionoftheteacher’schoice.Theminutepapermayfocusonatopiccoveredinclass,ormaybeusedasastudentevaluationofteaching.Studentsmaywritedirectlyaboutwhattheteacherisdoingwellandwhatmightbeimproved.Oncethetimelimit(frequently1-2minutes)hasexpired,theteacher

162

asksstudentstoturnintheirpapersanonymously,andlaterreadsthroughtheresponseslookingforthemesregarding,forexample,whatstudentsareorarenotlearningwellorwhatimprovementsmightbemadeinthecoursetoimprovestudentlearning.

GradedAssignmentsandTestsStudents’gradedassignmentsandtestsarereadilyavailableyetoftenoverlookedasevaluationtoolsofinstruction.Gradesprovideanobjectivealbeitanindirectmeasureofteachingqualityaswellasstudentlearning.Gradesmaybeagoodindicatorofhowwellstudentsunderstandcertainmaterial;teacherscanusethisinformationtoimprovethewayinwhichsubsequentinstructionisstructuredandpresented.Anygradedassignment(e.g.,tests,quizzes,papers,activities)couldpotentiallyservethispurpose,assumingtheteacherisabletodirectlytiethematerialinthegradedassignmenttoalearningobjectiveforthecourse(e.g.,atestquestiontoidentifythestatisticalconceptsinanadvertisementasameasureofstatisticalliteracy).However,keepinmindthatgradedassignmentsandtestsmaynotbeaseasilyinterpretableasothermeasuresofteachingqualitybecausemanyfactorscontributetoanindividual’sperformance(e.g.,motivation,ability,studyskills,andeducationalhistory).

StudentFocusGroupsThepurposeofstudentfocusgroupsistoobtainmoredetailedinformationthanmightbepossibleiftheentireclasswassurveyed.Toavoidbiasinstudentfeedback,itisadvantageousforanoutsideconsultant(e.g.,stafffromtheinstitution’steachingandlearningcenter,instructorfromanotherdepartment)togatherinformationfromasubsetofstudentsinacourse.Instructorsmayworkwiththeconsultantbeforehandtodeterminethetypeofinformationtobegatheredfromthestudentgroups.

StudentEvaluationsofTeachingThemostcommonformofteachingevaluationisthestudentevaluationofteaching(SET),whichistypicallyasetofwrittenfixed-answerquestionsthatismostbeneficialwhenwelldevelopedandempiricallysupported.Numerousratingscaleswithunknownpsychometricpropertiesareaccessible;however,therearealsoseveralempirically-supportedinstrumentsavailable.Theseinstrumentsarereliable,valid,andconsistentovertime,andincludetheStudentEvaluationofEducationalQuality(SEEQ;Marsh,1982),Barnesetal.’s(2008)measure,andtheTeacherBehaviorsChecklist(TBC;Buskistetal.,2002;Keeleyetal.,2006).Inadditiontotheseratingscales,theOfficeofEducationalAssessmentattheUniversityofWashington(2005)hasdevelopedasystemconsistingofseveralevaluationformstomeasureteachingeffectiveness.TheOfficeofEducationalAssessmentattheUniversityofWashingtondevelopedthisdetailed,empirically-supportedevaluationsystemthroughaseriesofmethodicalinterviewswithfaculty,administrators,andthroughstudentfocusgroups. Despitethevaluablefeedbackstudentsmightprovide,theydonothavetheexperienceandexpertisethatprofessionalteachingcolleaguesmaycontributetotheevaluationprocess.Peersmayalsohavetheabilitytoassistintroubleshootingclassroomandteachingissuesaswellas

163

offerguidancebasedontheirexperiences.Fellowteachersmayprovideamoreobjectiveteachingevaluationduetotheremovalofvariousfactorssuchasgradeleniencyanddifficultyofmaterial(Keeley,2012).Similartostudentevaluationsofinstruction,peerevaluationsareavailableinseveralformsincludinginvivo,peerreview,andteachingportfolios.

PeerEvaluationofTeachingPeerevaluationofteaching,sometimescalledpeerreviewofteachingorpeerconsultation,providesteacherswithhighlyspecificinformationbasedonasampleofactualteachingbehavior.Thisformofteachingevaluationinvolvesaqualifiedpeer(i.e.,apersonwhoisknowledgeableinpedagogyorwhohasotherwisebeentrainedintheintricaciesofpeerreviewofteaching)observinganinstructorteachaclasssessioninordertogatherobservationalinformationthatisthenlaterusedasthebasisforofferingconstructivecommentsregardingteachingcontent,style,andinteractionwithstudents.Thesefacetsofteaching,amongothers,aredifficult,ifnotimpossible,tocapturebyotherevaluativemethods.Inordertoobtainthemostusefulinformationfrompeerreviewofteaching,Ismail,Buskist,andGroccia(2012;seealsoBuskist,Ismail,&Groccia,2013)recommendafive-step,thoroughpracticethatincludesapre-observationmeetingwiththeteacher,classroomobservation,studentfocusgroups,awrittenreportpreparedbytheobserver,andpost-observationmeetingwiththeteacher. Peerreviewallowsforamoredetailedandcomprehensiveanalysisofteachingbehaviorthanothersources,andisheldbysomepedagogicalresearchersasthehighestqualityevaluativemeasurefortheanalysisofteachingeffectiveness(Ismailetal.,2012).Benefitsofthepeerreviewprocessincludeempiricalsupportfortheeffectiveimprovementofteaching,andtheopportunityforteacherstoimprovetheirteachingifconductedmid-semester(asopposedtothemoretypicalend-of-the-semesterSET).Peerreviewoffersbothobserversandobserveestheopportunitytolearnnewteachingtechniquesandparticipateincollegialdiscussionsregardingeffectiveteachingpractices(Ismailetal.,2012;Buskistetal.,2013).

TeachingPortfoliosTeachingportfoliosmaytakemanyformsandsometimescanprovideamorein-depthsampleofaninstructor’steachingthanpeerreview.Portfoliosareadvantageousinthattheyareparticularlyeffectiveforself-reflection(Seldin,2004).Aportfoliocanalsobedisseminatedwidely,therebybenefitingnumerousteachersinadditiontotheinstructorwhocreatedit.Instructorsmaychoosewhethertheirportfoliocoverstheirentireteachingcareerorsimplyasinglecourse.Edgerton,Hutchings,andQuinlan(1991)suggestedthattherichestandmostbeneficialteachingportfolioscombineprimaryteachingdocumentssuchassyllabi,tests,andpresentationswithcorrespondingpersonalreflectiveessays.Asanexemplarofthedevelopmentofteachingportfolios,XavierUniversityofLouisianahaslongimplementedwhatitcallstheCoursePortfolioWorkingGroup(CPWG),whichencouragesteachingimprovementacrosscolleges,departments,andcontentareas.Instructorswhoparticipateinthegroupfocusonimportanttopicssuchasstudentlearningoutcomes,teachingmethodsandpractices,andassessment(Schafer,Hammer,&Berntsen,2012).Attheendof

164

eachschoolyear,CPWGparticipantssubmitacompletedcourseportfolio.ThelastsessionofXavier’sCPWGisdevotedtoreviewingeachother’sworkandprovidingsupportivefeedbackandsuggestionsonhowtoimprovetheteachingprocessinordertobestbenefitstudents.Courseportfolioscanbeatoolwithwhichtobreathenewlifeintoacoursebyconsultingdifferentcolleaguesandthinkingcriticallyaboutpositiveandfeasiblechanges(Schaferetal.,2012).

SyllabusandTeachingPhilosophyReviewUnfortunately,peersarenotalwaysreadilyavailabletoreviewone’steachingpractices,andworkgroupsakintoCPWGatXavierUniversitydonotexistateveryinstitution.Similarly,sometimesstudentsdonotrespondtoevaluationrequestsorthereisnotenoughclasstimetoallotforteachingevaluations.However,thepracticeofself-reflectionispossibleatanytimethroughouttheyear,asitonlyrequiresthecourseinstructor’sparticipation.Teachers,likestudentsandpeers,haveauniqueperspectiveontheirteachingpracticesandhowimprovementsmaybemade.Therefore,itisworthwhiletofrequentlyreflectoverone’steachingthroughacoursesyllabusandmaterialsreviewaswellasateachingphilosophyreview.Otheraspectsofacourse,besidessimplytimespentteaching,maycontributetotheoveralleffectivenessofone’steaching.Asystematicreviewofthecoursesyllabusmayprovideawealthofinformationregardingclassgoalsanddirection,whichoftenchangesthroughoutthesemester.Inadditiontoreviewingthecoursesyllabus,areviewofone’steachingphilosophycanbebeneficialinensuringcongruencebetweenone’spersonalphilosophyofteachingandcurrentcourserelatedactivities.Ifcourseactivitiesandgoalsdonotalignwithone’steachingphilosophy,changingspecificelementswithinacoursemaybewarranted.

DevelopmentoftheTeacherBehaviorsChecklistWenowturntothespecificcaseoftheTBCasanillustrativeexampleofhowtodevelop,investigate,andutilizeaneffectivemeasureofteachingquality.TheTBCisaSETthatcanbeusedinseveralwaystobenefitone’steaching.UnlikemostinvestigatorswhodevelopSETs,wedidnotsetouttodevelopanevaluativeinstrumentofteaching.Instead,ouroriginalaimwastoconductanexploratoryinvestigationofthekeybehavioralattributesofexcellentteachers.Ourideawasthatifwecouldidentifysuchattributes,thenperhapswecouldteachthesebehaviorstoothers,particularlynewfacultyandgraduatestudentswhoaspiretotheprofessoriate.Ourreviewofthevastcollegeanduniversityteachingliteratureatthistime(circa1998)revealedlistafterlistofglobalteachertraitsthatresearcherslinkedtooutstandingteaching,forexample,beingapproachable,caring,enthusiastic,interesting,andknowledgeable(e.g.,Baiocco&DeWaters,1998;Eble,1984;Feldman,1976;Lowman,1995).Unfortunately,suchlistsdonotlendthemselveswelltoteachingotherstoteach—afterall,whatdoesitmeantobeapproachableorenthusiasticorknowledgeable?Howdoteachersactuallydemonstratesuchtraits?Thusbeganoursearchforconcreteanddemonstrablebehaviorsthatcomprisemasterteaching.

165

TheOriginalTBCStudyOurapproachtoaccomplishingthistaskinvolvedexploringarangeofpersonalityqualitiesandtheircorrespondingbehaviors(Buskistetal.,2002).Additionally,wecomparedfacultyandstudentperspectivesonwhichofthesequalities/behaviorsaremostimportanttoexcellentteaching.Phase1ofourresearchaskedundergraduatestolistatleastthreequalitiestheyjudgedtobereflectiveofmasterteachingatthecollegeanduniversitylevel.Thissampleproducedalistof47characteristics.Wethenpresentedthislisttoadifferentgroupofundergraduateswhomweinstructedto“listorotherwiseindicateuptothreespecificbehaviorsthatreflectthesequalitiesandcharacteristics.”Wenextanalyzedstudents’behavioraldescriptorsforcommonalities.Inmanyinstancestheyfoundthedescriptorsstudentsusedtocharacterizethe47qualitiesshowedsubstantialoverlap,whichresultedincollapsingthenumberofthosecategoriesto28.InPhase2oftheoriginalTBCstudy,anothersetofundergraduatesandasampleofAuburnUniversityfacultymembersselectedthetop10qualities/behaviorstheyjudgedtobekeytomasterteachingatthecollegeanduniversitylevel.Studentsandfacultyagreedon6ofthetop10qualities/behaviors(althoughindifferentorder):(a)realisticexpectations,(b)knowledgeable,(c)approachable/personable,(d)respectful,(e)creative/interesting,and(f)enthusiasm.Withrespecttothefourremainingitemsonwhichstudentsandfacultydidnotagree,therewasaninteresting,andasitturnsout,generalizabledifferencebetweenfacultyandstudentrankings.Facultytendedtoemphasizespecificelementsrelatedtoteachingtechnique(i.e.,effectivecommunication,prepared,current,andpromotingcriticalthinking),whereasstudentsemphasizedaspectsofthestudentandteacherrelationship(i.e.,understanding,happy/positive/humorous,encouraging,flexible).Indeed,recentworkhasfoundthatperceivedteacher-studentrapportisoneofthemostimportantpredictorsofstudentSETs(Richmond,Berglund,Epelbaum,&Klein,2015).Thus,teachersandstudentsappeartoshareseveralsimilarviewsonbehaviorsreflectiveofmasterteachingbutatthesametimeshowimportantdifferencesintheirperspectivesonkeyelementsofexcellentteaching.

FactorAnalysisoftheTBCNowthatwehadascale,weneededtodeterminewhetheritwasavalidandreliableinstrument.Weconductedafactoranalysistoexaminethebasicfactorstructureoftheinstrumentaswellasmeasureitsconstructvalidityandinternalreliability(Keeleyetal.,2006).

ConversiontoSETTocollectpsychometricdataontheTBC,weconvertedtheinstrumenttoanevaluativeinventorybyaddingasetofinstructionsanda5-pointLikert-typeratingofthefrequencyofexhibitingeachqualityrangingfrom1(never)to5(frequent).Theinstructionsaskedstudentstoratetheirteacherontheextenttowhichtheybelievedthattheirprofessorpossessedeachthe28teacherqualitiesandtheirattendantbehaviors.OursampleofstudentscompletedtheTBCaswellasthestandardAuburnUniversityend-of-the-semestereight-itemteaching

166

evaluation.Itemsonthestandardevaluationaddressedteacherqualitiesincludinghelpfulness,organizationandpreparednessforthecourse,abilitytomotivatestudentsandstimulatetheirthinking,clarityofteaching,andwhethertheprofessorspokeaudibly.HavingstudentscompletebothevaluationsallowedustohaveastandardofcomparisonforhowtheTBCrelatedtotheAuburnUniversityevaluation.

FactorAnalysisResultsWesubmittedstudents’ratingstoafactoranalysis,whichproducedtwosubscales:(a)professionalcompetency(11items:approachable/personable,authoritative,confident,effectivecommunicator,goodlistener,happy/positive/humorous,knowledgeable,prepared,punctuality/managestime,respectful,andtechnologicallycompetent)and(b)caringandsupportivebehaviors(13items:accessible,encouragesandcaresforstudents,enthusiastic,flexible/open-minded,humble,promotesclassdiscussion,intellectstimulating,providesconstructivefeedback,rapport,realisticexpectationsandgrading,sensitive/persistent,strivestobeabetterteacher,andunderstandingloadontothecaringandsupportivefactor).OurdataderivedfromTBCstudentevaluationsoffourdifferentinstructors.Weusedtwoone-wayANOVAstocomparetheseteachersinordertoassesswhetherthesesubscalesdiscriminatedamongprofessors.Foreachsubscale,wefoundsignificantdifferencesamongprofessorsthatcorrelatedwellwithstudents’evaluationsoftheseprofessorsonthestandardAuburnUniversityevaluation.Wefoundinternalconsistencytobe.95forthetotalofallitems.Theprofessionalcompetencysubscalehadareliabilitycoefficientof.90andthecaringandsupportivesubscale.93.Wealsoexaminedthetest-retestreliabilityofthescaleusinganewsetofdatafromanothergroupofdifferentinstructorsandfoundthatthetotalscalehadacoefficientof.70usingmidtermandend-of-termcomparisons.Thetwosubscaleswerealsostronglyreliablewith.68forthecaringandsupportivesubscaleand.72fortheprofessionalcompetencysubscale.Thus,theTBCisapsychometricallysoundandeffectiveinstrumentforevaluatingteachingqualityandinparticular,forassessingteachingexcellence.ThestrongpsychometricpropertiesoftheTBCalongwithitsclearbehavioralanchorsallowteachersandotherstodiagnoseandremediatespecificproblemsthatmaycharacterizeone’steaching.

TheTBCasaResearchToolSoonafterwepublishedourinitialarticleontheTBC(Buskistetal.,2002),weandothersbeganexaminingthescale’sapplicabilityforthestudyofexcellenceinteachingacrossdifferentinstitutionalandculturalenvironments.Ataliberalartscollegeandacommunitycollege,facultyandstudentstendedtoagreeonthetopteachingqualities(Schaeffer,Epting,Zinn,&Buskist,2003;Wann,2001).Wefoundthatstudentsandfacultyatbothinstitutionsratedsixqualitiessimilarly(realisticexpectations,knowledgeable,approachable,respectful,creative/interesting,andenthusiastic).Whencomparingonlythefacultyfromeachinstitution,wefoundagreementonsevenofthetop10qualities(thesamesixqualitiesasthecombined

167

student/facultyratingwithcriticalthinkingastheseventhquality).Inlookingatstudentratingsonly,studentsfrombothinstitutionsrankedallofthesamequalitiesasbeingintheirtop10.Thetop10ratedTBCqualitiesforstudentsconsistedofthesamesixasthecombinedstudent/facultyratingsalongwithhappy/positive/humorous,encouraging,flexible,andunderstanding.Thisfindingfurthersupportsthenotionthatstudentsdoindeedcaresignificantlyaboutthestudent-teacherrelationship.Inamorerecentstudy,wecomparedthesesetsoffacultywithnationalaward-winningfacultyonthetop10TBCqualities(Keeley,Ismail,&Buskist,inpress).Interestingly,eightofthetop10qualitiesselectedbynationalaward-winningfacultyfellwithinthetop15ofbothresearchinstitutionfacultyandcommunitycollegefaculty.Thosequalitieswere:enthusiasticaboutteachingandtopic,strivestobeabetterteacher,creativeandinteresting,knowledgeableaboutsubjectmatter,approachable/personable,effectivecommunicator,respectful,andencourages/caresforstudents.Notably,however,thetworemainingqualitiesinthetop10fornational-awardwinningfaculty,preparednessandrapport,wererankedasbeinginthe20thpositionorworseforbothresearchinstitutionandcommunitycollegefaculty.Thisfindingsuggeststhatexcellentteachers(asoperationalizedbyindividualswhohavewonanationalteachingaward)placemoreemphasisonbeingthoroughlypreparedforclassaswellasmakinganefforttocreateacaringandsupportiveclassroomatmospherefortheirstudents.Thusfar,ourfindingscomparingdataatdifferenttypesofinstitutionsprovideevidenceforthegeneralizabilityoftheTBCasameasureofteachingexcellence.Severalcross-culturalstudiesextendthegeneralnatureoftheTBCevenfurther(e.g.,Jõemaa,2013;Keeley,Christopher,&Buskist,2012;Liu,Keeley,&Buskist,2015;Vulcano,2007).Forexample,Vulcano(2007)surveyedtwosamplesofCanadianundergraduatesontheirviewofa“perfectinstructor.”Studentsidentifiedasmanydescriptorsastheywished,whichVulcanothencategorizedinto26setsofqualitiesandbehaviors.Thetop10categorieswere(a)knowledgeable;(b)interestingandcreativelectures;(c)approachable;(d)enthusiasticaboutteaching;(e)fairandrealisticexpectations;(f)humorous;(g)effectivecommunicator;(h)flexibleandopen-minded;(i)encouragesstudentparticipation;and(j)encouragesandcaresforstudents.Ofthe26categoriesdevised,24ofthemwerethesameorsimilartoTBCitems,whichofferssomeininternationalsupportforgeneralcategoriesofexcellentteaching,atleastintermsofNorthAmerica.Keeleyetal.(2012)recruitedstudentsatasmallliberalartsschoolinJapan—MiyazakiInternationalCollege—andfromasmallliberalartsschoolintheU.S.—AlbionCollege—andhadallparticipantscompletethe28-itemTBCbyratingtheextenttowhicha“masterteacher”displayseachqualityanditsattendantbehaviors.AmericanandJapanesestudentsagreedon7ofthetop10teacherqualities:knowledgeable,confident,approachable/personal,enthusiastic,effectivecommunicator,prepared,andgoodlistener.ThethreediscordantqualitiesforAmericanstudentswereaccessible,respectful,andintellectuallystimulating.Thesequalitieswereranked21st,20thand25th,respectively,fortheJapanesestudents.FortheJapanesestudents,thequalitiesofbeingcreativeandinteresting,strivestobeabetterteacher,and

168

humbleroundedouttheirtop10teacherqualities.Americanstudentsratedthesequalitiesas18th,22nd,and24th,respectively. Inasimilarstudy,participantsatalargeuniversityinEasternChinaratedthreeTBCqualitiesthesameasJapaneseandU.S.students:prepared,sensitiveandpersistent,andunderstanding(Liu,Keeley,&Buskist,2015).Wealsoobservedsomeinterestingdifferencesamongthethreesetsofstudents.Onetheonehand,Chinesestudentsplacedlessemphasisontheirteachersbeingapproachable,confident,enthusiastic,knowledgeable,aneffectivecommunicator,andagoodlistenerthaneithertheJapaneseorU.S.students.Ontheotherhand,Chinesestudentsplacedmoreemphasisononlyonequality,technologicallycompetent,thanboththeJapaneseandU.S.students.ChinesestudentsseemedlessinterestedintheinterpersonalfactorsofteachingthanJapanesestudents.Chinesestudentsalsorankedqualitiessuchasaccessible,flexible,punctual,respectful,establishesdailyandacademictermgoals,presentscurrentinformation,promotescriticalthinking,andprovidesconstructivefeedbacklowerthantheU.S.students.Incomparingourstudentrankingsofthetop10TBCqualitiesofmasterteachingresultsfromthethreeU.S.samples,Canada,Japan,andChina,onlyonequality,knowledge,madeitswayintothetop10ineachofthosecountriesineverysample.However,intermsofrankorder,sixqualities,knowledgeable,approachable/personable,realisticexpectations,creativeandinteresting,enthusiastic,andeffectivecommunicator,wererankedinthetop10inatleastfiveofthesixsamples.Theimplicationofthesefindingsisthatstudentsfromdifferentworldregionsmayvalueteacherqualitiesdifferently.Thus,teachersusingtheTBCmustrecognizethathigherorlowervaluesonaparticularitemmustbeinterpretedinthenormativecontextofstudentpreferencesforthatregion.Examiningdifferencesacrossacademicdisciplines,Liu,Keeley,andBuskist(inpress)foundthatChinesestudentsmajoringinpsychology,education,andchemicalengineeringregardedfiveitemsastopqualitiesofamasterteachers:respectful,knowledgeable,confident,strivestobeabetterteacher,andrealisticexpectations.ChemicalengineeringstudentsplacedgreateremphasisontheTBCitemsofprepared,punctuality/managesclasstime,anddailyandacademicgoalsthantheirpsychologyoreducationcounterparts.Educationstudentsratedtheapproachable/personablequalityasbeingmoreimportantthanpsychologystudents.ThesefindingssuggestthatChinesestudentshaveagenerallyrecognizedsetofqualitiestheyassociatewithmasterteachersregardlessofdiscipline,butnevertheless,studentsacrossacademicdisciplinesmaydiffermodestlyinwhichqualitiestheylinktoexcellenceinteaching.Thecross-culturalworkusingtheTBCdiscussedthusfarhasfocusedonstudents.Ismail(2014)comparedtwogroupsoffaculty—U.S.-educatedfacultyandforeign-educatedfaculty(atthebaccalaureatelevel)—teachingatU.S.institutions.Foreigneducated-facultyandU.S.-educatedfacultywereinagreementon9oftheirtop10TBCqualities:knowledgeable,enthusiastic,creativeandinteresting,promotescriticalthinking,effectivecommunicator,approachable,encouraging,managestimewell,andaccessible.Comparingthesedatawithourresearchinstitution(Buskistetal.,2002)andcommunitycollegefacultydata(Schaefferetal.,2003)

169

revealsthatfivequalities(knowledgeable,enthusiastic,creative/interesting,promotescriticalthinking,andapproachable)wererankedinthetop10acrossthefoursamples.Insummary,severalTBCqualitiesemergedasbeinginthetop10asratedbyfacultyinatleastthreeofourfacultysamples:knowledgeable,enthusiastic,creative/interesting,effectivecommunicator,approachable,promotescriticalthinking,encouraging.Weseeasimilarpatternforstudentsinfiveofsixstudentsampleswehavestudiedresultinginthefollowingtopqualities:knowledgeable,enthusiastic,creative/interesting,effectivecommunicator,approachable,andrealisticexpectations.Thestudentandfacultylistsoftopqualitieshavefivequalitiesincommon:knowledgeable,enthusiastic,creative/interesting,effectivecommunicator,andapproachable.Suchconsistentagreementsuggeststhattheremayexistauniversalsetofqualitiesthatcompriseexcellentteaching.

TheTBCasaSETandProfessionalDevelopmentToolAlthoughresearchintoexcellentteachingqualitiesusingtheTBCisvaluableinitsownright,theTBCalsohassubstantialpracticalutilityasastudentevaluationofteaching(SET).Asnotedearlier,Keeleyetal.(2006)adaptedtheTBCbyincorporatinga5-pointratingscaleforuseasaSET.ThemeasurementstructureoftheTBCissuchthatitprovidesaglobalestimateofqualityteaching(theTotalscale),whichcanbesplitintosubscalesof(a)CaringandSupportivebehaviorsand(b)ProfessionalCompetencyandCommunicationSkills.StudentratingsusingtheTBCarereliable(Keeleyetal.,2006;Landrum&Stowell,2013),andhavebeenshowntomeaningfullydifferentiatethequalityofinstruction(Keeley,English,Irons,&Henslee,2013;Keeley,Furr,&Buskist,2009).TheTBCisabehaviorally-basedscale:eachcharacteristicofteachingincludedinthemeasureencompassesseveralspecificbehaviors.ThesebehavioralanchorsundergirdtheutilityoftheTBCasatoolforteachingimprovement.Forexample,ifateacherscorespoorlyonaparticularcharacteristic,heorshemayrefertothebehaviorscomprisedbythatcharacteristicforsuggestionsashowtoimplementpositivechangestohisorherteaching.Otherteachingevaluationinstrumentstypicallyonlyincludequalitativedescriptorsofteaching,andcanbedifficulttotranslateintosuggestionsformakingbehavioralchangeinteaching(Keeleyetal.,2006).Feeley(2002)foundthatifstudentshaveaparticularlystrongpositiveornegativeviewofateacher,theyarelikelytoratethatteachermorepositivelyornegativelyoverall,basedontheiropinionofonlyasingleaspectoftheteacher.Ratingteachersonanoverall“feeling”insteadofprovidingobjective,behaviorally-anchoredfeedback,makesitdifficultforteacherstoknowpreciselywhichspecificaspect(s)oftheirteachingtoimprove.SeveralfactorsinadditiontoratingtoobroadlyhavebeenfoundtodecreasetheaccuracyofTBCscores(Keeleyetal.,2013).Studentstendtorateteachersmorehighlyinsmallercourses,andnon-instructionalvariablessuchasprofessors’personalitystylecanskewratingsineitherdirection(Clayson&Sheffet,2006).Similarly,oneareaoflowscorescandecreasealloftheothers,creatinganinaccurateratingprofile(i.e.,anegativehaloeffect).Itisimportantforteacherstobeawareoftheseratingbiasesandinterpretscoresappropriately.

170

OnewaytoincreasetheaccuracyofstudentTBCratingsistoexplaintostudentstheimportanceofaccurateandconscientiousrating.Alltoooften,teachersrushthroughSETsattheendofthesemester,withoutmuchexplanationorguidance.Bernardin(1978)foundthatstudentsprovidedmoreaccurateratingsafterbeingtrainedandeducatedaboutratingerrors.Teacherscanalsocomparetheirscorestoanormativebasisofsimilarcoursesatsimilarinstitutionstoobtainmoreinformationregardingthemeaningoftheirscores(Keeleyetal.,2013).TheTBCisprimarilyintendedforuseinformativeassessment—itsfocusisonimprovingone’steaching.However,itisequallyusefulforsummativeassessment,orthekindofevaluativeassessmentdoneforpromotion,tenure,qualitycontrol,andmeritraises.Asanexample,theTBChasbeenusedintheteachingtrainingprogramforgraduatestudentsintheAuburnUniversityPsychologyDepartmentformanyyears.Aspartoftheirdoctoraltraining,graduatestudentsenrollinateachingpracticumcoursewhilesimultaneouslyservingasateachingassistant.Aspartofthatexperience,studentsusetheTBCtoevaluatetheirperformanceatthemiddleandendofthesemester.Theyalsoprovidemocklecturesinthedidacticportionoftheexperience,whichareevaluatedbyanexperiencedfacultymemberusingtheTBCasaratingform.Thebehavioralanchorsprovideboththestudentandthefacultyraterwithdirectsuggestionsforhowtoimproveareasofweakness.

FinalThoughtsTheoriginaldevelopmentoftheTBCtookplacemorethanadecadeago.AlthoughithasgeneratedmuchresearchandhasbeenwidelyusedasaSET,itispossiblethatchanges,especiallyintechnologyandcorrespondingclassroompractices,haveoccurredinthepast10yearsthatwouldchangethebehavioralanchorsoftheTBC.Assuch,wearecurrentlyundergoingarevitalizationoftheTBC(tentativelyentitledTBC2.0)thatwillupdatethelistofqualitiesandtheirattendantbehaviors.Insummary,itiscrucialforallteachers—youngandold,newandadvanced,toutilizeavarietyofavailablemethodsofevaluationinordertoimprovethequalityoftheirteaching.ThedevelopmentanduseoftheTBCprovidesanexemplarofhowtocreatemeasuresofqualityteachingandevaluatetheirutility.Unfortunately,suchaprocesshasonlybeenrarelyundertaken,andmostmeasuresofteachingqualitylackempiricalsupport.However,wehope,andindeedchallenge,allteacherstoadoptarigorousandsystematicapproachtothemeasurementofthequalityoftheirteaching.

171

ReferencesReferencesmarkedwithanasteriskindicateascale.*Angelo,T.A.,&Cross,K.P.(1993).Classroomassessmenttechniques:Ahandbookforcollege

teachers(2nded.).SanFrancisco,CA:Jossey-Bass.Baiocco.S.H.,&DeWaters,J.N.(1998).Successfulcollegeteaching:Problem-solvingstrategiesof

distinguishedprofessors.Boston,MA:Allyn&Bacon.*Barnes,D.,Engelland,B.,Matherine,C.,Martin,W.,Orgeron,C.,Ring,J,Smith,G.,&Williams,

Z.(2008).Developingapsychometricallysoundmeasureofcollegiateteachingproficiency.CollegeStudentJournal,42,199-213.

*Bernardin,H.(1978).Effectsofratertrainingonleniencyandhaloerrorsinstudentratingsofinstructors.JournalofAppliedPsychology,63,301-308.doi:10.1037/0021-9010.63.3.301

Buskist,W.,Ismail,E.,&Groccia,J.E.(2013).Apracticalmodelforconductinghelpfulpeerreviewofteaching.InJ.Sachs&M.Parsell(Eds.),Peerreviewoflearningandteachinginhigher

education:Internationalperspectives(pp.33-52).Berlin,Germany:Springer.*Buskist,W.,Sikorski,J.,Buckley,T.,&Saville,B.K.(2002).Elementsofmasterteaching.InS.F.Davis

&W.Buskist(Eds.),Theteachingofpsychology:EssaysinhonorofWilbertJ.McKeachieandCharlesL.Brewer.(pp.27-39).Mahwah,NJ:Erlbaum.

Clayson,D.E.,&Sheffet,M.(2006).Personalityandthestudentevaluationofteaching.JournalofMarketingEducation,28,149-160.doi:10.1177/0273475306288402

Eble,K.E.(1984).Thecraftofteaching.SanFrancisco,CA:Jossey-Bass.Edgerton,R.,Hutchings,P.,&Quinlan,K.(1991).Theteachingportfolio:Capturingscholarship

inteaching.Washington,DC:AmericanAssociationforHigherEducation.Ellis,L.,Burke,D.M.,Lomire,P.,&McCormack,D.R.(2004).Studentgradesand

averageratingsofinstructionalquality:Theneedforadjustment.JournalofEducationalResearch,9,35-41.doi:10.1080/00220670309596626

Feeley,T.(2002).Evidenceofhaloeffectsinstudentevaluationsofcommunicationinstruction.CommunicationEducation,51(3),225-236.doi:10.1080/03634520216519

Feldman,K.A.(1976).Thesuperiorcollegeteacherfromthestudent’sview.ResearchinHigherEducation,5,243-288.

*Ismail,E.A.(2014).ForeignandUS-educatedfacultymembers’viewsonwhatconstitutesexcellentteaching.UnpublishedDoctoralDissertation,AuburnUniversity,AuburnAL.

Ismail,E.A.,Buskist,W.,&Groccia,J.E.(2012).Peerreviewofteaching.InM.E.Kite(Ed.),Effectiveevaluationofteaching:Aguideforfacultyandadministrators(pp.79-91).RetrievedfromtheSocietyfortheTeachingofPsychologywebsite:http://teachpsych.org/ebooks/evals2012/index.php

*Jõemaa,K.(2013).StudentperceptionsofmasterteachersinEstonianuniversities.UnpublishedMaster’sThesis,UniversityofTartu.

*Keeley,J.,Christopher,A.N.,&Buskist,W.(2012).Emergingevidenceforexcellentteachingacrossborders.InJ.E.Groccia,M.Al-Sudairy,&W.Buskist(Eds.),Handbookofcollegeanduniversityteaching:Globalperspectives(pp.374-390).ThousandOaks,CA:Sage.

*Keeley,J.(2012).Courseandinstructorevaluation.InW.Buskist&V.A.Benassi(Eds.).Effectivecollegeanduniversityteaching:Strategiesandtacticsforthenewprofessoriate(pp.173-180).ThousandOaks,CA:Sage.

172

*Keeley,J.W.,English,T.,Irons,J.,&Henslee,A.M.(2013).Investigatinghaloandceilingeffectsinstudentevaluationsofinstruction.EducationalandPsychologicalMeasurement,73,440-457.doi:10.1177/0013164412475300

*Keeley,J.,Furr,R.M.,&Buskist,W.(2009).Differentiatingpsychologystudents'perceptionsofteachersusingtheteacherbehaviorschecklist.TeachingofPsychology,37,16-20.doi:10.1080/00986280903426282

*Keeley,J.,Ismail,E.A.,&Buskist,W.(inpress).Excellentteachers’perspectivesonexcellentteaching.TeachingofPsychology.

*Keeley,J.,Smith,D.,&Buskist,W.(2006).TheTeacherBehaviorsChecklist:Factoranalysisofitsutilityforevaluatingteaching.TeachingofPsychology,33,84-91.doi:10.1207/s15328023top3302_1

*Landrum,R.E.,&Stowell,J.R.(2013).Thereliabilityofstudentratingsofmasterteacherbehaviors.TeachingofPsychology,40,300-303.doi:10.1177/0098628313501043

*Lowman,J.(1995).Masteringthetechniquesofteaching(2nded.).SanFrancisco,CA:Jossey-Bass.

*Liu,S.,Keeley,J.,&Buskist,W.(2015).Chinesecollegestudents’perceptionsofcharacteristicsofexcellentteachers.TeachingofPsychology,42,83-86.doi:10.1177/0098628314562684

*Liu,S.,Keeley,J.,&Buskist,W.(inpress).Chinesecollegestudents’perceptionsofexcellentteachersacrossthreedisciplines:Psychology,chemicalengineering,andeducation.TeachingofPsychology.

*Marsh,H.W.(1982).SEEQ:Areliable,validandusefulinstrumentforcollectingstudents’evaluationsofuniversityteaching.BritishJournalofEducationalPsychology,52,77–95.doi:10.1111/j.2044-8279.1982.tb0205.x

*Marsh,H.W.,&Roche,L.A.(2000).Effectsofgradingleniencyandlowworkloadofstudents’evaluationsofteaching:Popularmyth,bias,validity,orinnocentbystanders?JournalofEducationalPsychology,92,202-228.doi:10.1037/0022-0663.92.1.202

Richmond,A.S.,Berglund,M.B.,Epelbaum,V.B.,&Klein,E.M.(2015).a+(b1)professor-studentrapport+(b2)humor+(b3)studentengagement=(Ŷ)studentratingsofinstructors.TeachingofPsychology,42,119-125.doi:10.1177/0098628315569924

*Schaeffer,G.,Epting,K.,Zinn,T.,&Buskist,W.(2003).Studentandfacultyperceptionsofeffectiveteaching.Asuccessfulreplication.TeachingofPsychology,30,133-136.

Schafer,P.,Hammer,E.Y.,&Berntsen,J.(2012).Usingcourseportfoliostoassessandimproveteaching.Effectiveevaluationofteaching:InM.E.Kite(Ed.),Effectiveevaluationofteaching:Aguideforfacultyandadministrators(pp.71-78).RetrievedfromtheSocietyfortheTeachingofPsychologywebsite:http://teachpsych.org/ebooks/evals2012/index.php

Seldin,P.(2004).Theteachingportfolio:Apracticalguidetoimprovedperformanceandpromotion/tenuredecisions(3rded.).SanFrancisco,CA:Jossey-Bass.

Vulcano,B.A.(2007).Extendingthegeneralityofthequalitiesandbehaviorsconstitutingeffectiveteaching.TeachingofPsychology,34,114-117.doi:10.1080/00986280701293198

173

*Wann,P.D.(2001,January).Facultyandstudyperceptionsofbehaviorsofeffectivecollegeteachers.PosterpresentedattheNationalInstitutefortheTeachingofPsychology,St.PetersburgBeach,FL.

174

Chapter14:SoTLScales:TheCaseofMissingLinksAaronS.Richmond

MetropolitanStateUniversityofDenver

SoTLisNOTCryptozoology:ABriefSoTLHistoryThroughoutthiswonderfulcompendiumofScholarshipofTeachingandLearning(SoTL)scales,manyhavedescribedspecificSoTLscales,howtousethem,andhowtocreatethem.Forinstance,scalestomeasurestudentlearningand/orself-efficacy(seeMarek,Williamson,&Taglialatela,2015),criticalthinkingskills(seeLandrum&McCarthy,2015),studentinterestsandperceptions(seeZabelandHeger,2015),servicelearning(seeSimons,2015),metacognitiveawarenessandskills(seeKu,2015),studenthappiness,stress,andanxiety(seeLayous,Nelson,&Legg,2015),professor-studentrelationships(seeMeyerberg&Legg,2015),andprofessorefficacy(seeKirk,Busler,Keeley,&Buskist,2015)haveallbeendescribed.However,thisisnotwhereSoTLbegan.Morelikely,SoTLhasbeenaroundforaslongasteachershavebeeninexistence—atleastinformally.Thatis,ifyouhaveeverobservedgreatteachers,theyoftenreflectandassesstheirteachingandmakeneededmodificationsandadjustmentsbasedontheirconclusions.Informally,wealsoseeSoTLasearlyasthe1920s.RobertMaynard,theincomingpresidentoftheUniversityofChicago,famouslystatedinhisinauguraladdress,

[APh.D.candidatewhoplanstobeateacher]…mustbeintouchwiththemostrecentandmostsuccessfulmovementsinundergraduateeducation,ofwhichhenowlearnsofficiallylittleornothing.Howshouldhelearnaboutthem?Notinmyopinionbydoingpracticeteachinguponthehelplessundergraduate.Ratherheshouldlearnaboutthemthroughseeingexperimentscarriedoninundergraduateworkbythemembersofthedepartmentinwhichheisstudyingforthedegree….(ScholarshipofTeachingandlearning:History,n.d.)

Yetformally,SoTLbeganwiththeonsetofBoyer’s(1990)ScholarshipReconsidered:PrioritiesoftheProfessoriateandcontinuedbyShulman’s(1993)workonTeachingasCommunityProperty:PuttinganEndtoPedagogicalSolitude.Betweenthesetwoscholars,SoTLbegantotakerootasaformofresearchintentonimprovingteachinginhighereducation.Sincethistime,scholarshaveconducted1000sofSoTLstudiesin100sofacademicfields.Asanartifactofthisproductivity—naturally—SoTLscaleshavebeendeveloped,tested,andimplemented.ConsideringtheexplosionofSoTLandsubsequentSoTLscales(i.e.,thelongoverdueneedforthise-book),whatthenisneededormissing?Thatis,whatareweSoTLscholarsnotmeasuringthatweshouldbemeasuring?WhataresomeissueswithpreexistingSoTLscales?Orinotherwords,howcanweimproveuponexistingSoTLscales?Itismyhope,throughoutthischaptertoanswerthesequestionsbyidentifyingandilluminatingthesemissinglinks(i.e.,debunkcryptozoology!).Specifically,Iwilldiscusstheneedfornewandimprovedmetacognitiveandlearningstrategymeasures,theneedforSoTLscalesthatassesssyllabi,commentontheissuethatmanySoTLself-reportscaleslackmatchingbehavioralmeasures,andtheneedforaSoTLscaleassessingmodelteachingcharacteristics.

175

NessietheMythicalLochNessMonsterRivalstheMythicalSoTLSyllabusScaleAsmythicalandmysteriousasNessietheLochNessMonster,researchonsyllabusconstructionandbestpracticeshasshownequalmysteryandelusiveness.However,thesyllabushasreceivedsomeSoTLattention,asoflate,asakeyelementtobestpracticesinhighereducationinstruction(e.g.,Boysen,Richmond,&Gurung,2015;Richmondetal.,2014;Slattery&Carlson,2005).Forexample,thesyllabuscanhaveanimmense(positiveornegative)impactonhowstudentsperceiveteachingeffectiveness(Richmond,Becknell,Slattery,Morgan,&Mitchell,2015;Saville,Zinn,Brown,&Marchuk,2010).Asthisbodyofresearchgrows,theneedtoassesssyllabi,bothreliablyandvalidly,becomesmoreandmoreimportant.Todate,theonlySoTLscaletoevaluatesyllabiwasdevelopedbyCullenandHarris(2009).CullenandHarriscreatedarubrictoassessthedegreetowhichasyllabusisconsideredtobelearning-centeredasopposedtoteacher-centered.Intherubric,theydescribethreemainfactors(e.g.,community,powerandcontrol,andevaluation/assessment).Withineachmainfactorthereareseveralsubfactors.SeeTable1foracompletelistanddescription.Thescaleismeasuredonacategoricallevelfrom1(moreteacher-centered)to4(morelearner-centered)ratedbytheinstructor,notbystudents.Forexample,ifasyllabuswaslearner-centereditwouldhavealearningrationalethathada“rationalprovidedforassignments,activities,methods,policies,andprocedures;tiedtolearningoutcomes”(p.123). Table1CullenandHarris(2009)RubricAssessingLearner-CenteredSyllabi

FactorsandSubFactorsCommunity PowerandControl Evaluation/AssessmentAccessibilityofTeacher Teacher’sRole GradesLearningRationale Student’sRole FeedbackMechanismsCollaboration OutsideResources Evaluation SyllabusTone LearningOutcomes

SyllabusFocus Revision/Redoing Whereas,ifthesyllabuswereteacher-centereditwouldhavealearningrationalethathad“norationaleprovidedforassignmentsoractivities”(p.123).Unfortunately,thereareanumberofissueswiththisSoTLmeasure.First,thelevelofmeasurementiscategorical.Thatis,therubricisona1-4rubric/scalethatonlydescribescategoriesordegreesofleveloflearner-centeredness.Becauseofthislevelofmeasurementitmakesitalmostimpossibletounderstandafactor-structuretothescale,andassessreliabilityandvalidity.Second,althoughafewstudieshaveusedtherubric(e.g.,Slatteryetal.,2014),thereisvirtuallynofurtherevidencethatmaysuggesthowreliableorvalidthescaleis.Third,thisrubriconlytakesthepedagogicalperspectiveofstudent-centeredinstruction.Thereareseveralothereffectiveformsofpedagogiesthatshouldalsobeassessedasvaluabletoolsforsyllabusconstruction(e.g.,Inter-teaching,Just-in-timeteaching,etc.).BasedonmyreviewoftheCullenandHarris(2009)SoTLrubricandthelackofanyotherSoTLscalesthatassesssyllabi,IsuggesttwomajordirectionsinthisareaofSoTL.First,SoTL

176

researchersshouldmodifyCullenandHarris’rubrictoatleasthaveanintervalorbetterscale.Thatis,convertthe1-4categoriesintoLikert-typequestions.Forinstance,thefactorofEvaluation/Assessmentandthesubfactoroffeedbackmechanisms,scalequestionwouldread,"Thesyllabusdescribessummativeandformativeevaluationsincludingwrittenandoralpresentations,groupwork,self-evaluationandpeerevaluation"anchoredin1(stronglyagree)to5(stronglydisagree)orcouldhaveanchorsoffrequency,suchas1(always)to6(never).ThiswouldallowSoTLresearcherstoconductfactoranalyses,test-retestreliability,split-halfreliability,andconvergent,construct,content,andpredictivevaliditystudiesofthisSoTLscale.Second,otherformsofSoTLscalesthatassesstheefficacyofsyllabiareneeded.Ithasbeenarguedthatexemplarsyllabishouldserveasacontracttostudents,apermanentrecord,acognitivemap,alearningtool,andacommunicationdevice(Matejka&Kurke,1994;Parkes&Harris,2002).Assuch,SoTLresearchersshoulddevisescalesthatassessthedegreetowhichsyllabicontaintheseelements.Forexample,researcherscouldusethesescalestoassessdifferencesbetweencollegeanduniversityteachers’pedagogy.Or,researcherscouldusethesescalestoassesstheefficacyofthesyllabusdesignandhowitmayaffectstudentlearning.Intheend,let’sdemystifythesyllabus(akaNessetheLochnessMonster)becausenotonlyissyllabusresearchunder-studiedinSoTL,thereisgreatroomandneedtodevelopandvalidateSoTLscaleswhichattempttoassesstheefficacyofsyllabi. ProvethatBigfootisReal!Self-ReportSoTLScalesNeedMatchingBehavioralMeasuresDoyouknowsomeonewhobelievesinBigfoot?Howdotheyknow?Didtheyseetheelusivecreature?Ordidtheysay,“MycousinoncesawBigfoot”or“MyneighborBillylostthreegoatstoBigfoot”?UnliketheTVshowMonsterQuestbythebelovedHistoryChannel,itislikelythatmostpeoplewhobelieveinBigfootdosobecauseofself-reportorthereportofothers.Hereinliestherub.Somewhatlikecryptozoology,manySoTLscalesrelyheavilyonself-reportandnotenoughonactualbehavior.Aspsychologists,weknowalltowellthepitfallsofself-reportscales.Thatis,issuesofhonesty,introspection,socialdesirability,understandingandcomprehension,responsebias,responseset,andonandon.AsBaumeister,Vohs,andFunder(2007)sopoignantlysaid,

theeclipseofbehavior…inwhichdirectobservationofbehaviorhasbeenincreasinglysupplantedbyintrospectiveselfreports,hypotheticalscenarios,andquestionnaireratings.Weadvocatearenewedcommitmenttoincludingdirectobservationofbehaviorwheneverpossibleandinatleastahealthyminorityofresearchprojects.(p.396)

However,thisdoesnotmeanthatself-reportSoTLscalesareworthless.Rather,I’mheretosuggestthattherearethreeprimarymethodologicalsolutions.First,whenusingself-reportSoTLscales,itisimportanttohavematchingorcomplementingbehavioralmeasuresthatsupportandhaveconsistentresultswiththeself-reportscales.Let’sillustratethisissuethroughacommonmeasureofmetacognitiveawareness,theMetacognitiveAwarenessInventory(MAI;Schraw&Dennison,1994).TheMAIhasbeenusedandcitedinover1000studiesandisa52-iteminventorythatattemptstomeasurethemetacognitive

177

componentsofknowledgeofcognitionandregulationofcognition(Schraw&Dennison,1994).Forexample,oneoftheitemsusedtoassesstheknowledgeofcognitionsubscalestates,“IknowwheneachstrategyIusewillbemosteffective”(p.473).TheMAIisscoredona5-pointLikertscalewiththeanchorsof1(trueofme)to5(nottrueofme).TheproblemhereistheMAIreportswhattheparticipantbelievestobetrue,notwhatheorsheactuallydoes(i.e.,behavior).Instead,IsuggestthatwhenusingsuchSoTLscalesordevelopingnewSoTLscales,thatresearchersshouldalsousecomplementingbehavioralmeasurestocomplementandvalidatesaidself-reportmeasures.Inthisexample,thecomplementtotheMAIwouldbetocollectthebehavioralmeasureofactualmetacognitivebehavior(e.g.,calibration).Calibrationisameasureofmetacognitionthatmeasuresofthedifferencebetweenone’sjudgmentofperformanceandone’sactualperformance(seeSchraw,Kuch,&Guitierrez,2013;Schraw,Kuch,Guitierrez,&Richmond,2014).Tomeasurecalibrationyouaskindividualstoansweraquestiononaassessment,youthenaskindividualstoanswerwhethertheybelievedtheygottheanswercorrectornot.Next,yourecordwhethertheygotthequestioncorrectorincorrect.Thenyoumathematicallycalculatethedifferencebetweenwhattheysaidtheygotcorrectorincorrectandwhattheyactuallydid.Theresultingproduct-1.0(alwaysunderconfident)to+1.0(alwaysoverconfident)isthedegreetowhichtheywereaccurateatknowingwhattheyknoworknowingwhattheydonotknow.Ascorecloserto0indicateshighaccuracy.(FormoreinformationonscoringseeSchrawetal.,2013;2014).BycombiningboththeMAIandcalibration,SoTLresearcherswillnotonlyhaveahighlyreliablemeasureofmetacognitionandtheeffectsthatatreatmentmayhaveonit,buttheywillalsohaveaveryinternallyandexternallyvalidmeasureofmetacognition.LandrumandStowell(2013)provideagreatexampleofhowtomatchself-reportmeasureswithactualbehavior.ThepurposeoftheirstudywastovalidatetheTeacherBehaviorsChecklist(TBC;Keeley,Smith&Buskist,2006)bymatchingcorrespondingself-reportedbehaviorstoobservedteachingbehaviors.Specifically,LandrumandStowellhadover700studentswatchseveral5-minutevideovignettesthatweredesignedtodemonstratemasterteacherbehaviorspurportedbytheTBC(e.g.,respectful,enthusiastic,approachable,etc.).TheythenaskedthestudentstorateeachprofessorontheTBC.LandrumandStowellfoundthatwhenstudentsviewedthesamevignette(e.g.,demonstratingrespecttostudents),thatthestudentswereoftenveryconsistentintheirratingoftheTPC(i.e.,agreementrangedfrom68%-91%).Therefore,asdemonstratedbyLandrumandStowell,whenconductingfutureSoTLstudiesandscales,researchersshouldincludebothself-reportandcomplementarybehavioralmeasures.Asecondsolutiontotheproblemswithself-reportscalesistousescalesthatareeitherbehavioralchecklistsorscenario-basedscales.Asdiscussedinmultiplechaptersofthise-book,oneoftheprimeexamplesofaSoTLscale,whichmeasuresbehavior,istheTeacherBehaviorsChecklistbyKeeley,Smith,andBuskist(2006).TheTeacherBehaviorsChecklist(TBC)isa28-itemLikertTypescalewherestudentsassessthebehaviors(e.g.,effectivecommunicator,preparedness,knowledgeable,enthusiastic,flexible/open-minded,etc.)ofteachers.TheTBCis

178

ratedona5-pointLikertscalefrom1(never)to5(frequent)onhowoftentheprofessorexhibitsthegiventeachingbehavior.Forinstance,forthebehaviorof'providesconstructivefeedback,'thequestionwouldstate,“Writescommentsonreturnedwork,answersstudents’questions,andgivesadviceontesttaking”(Keeleyetal.,2006,p.85).InthecaseofSoTLscalessuchastheTBC(orcheckoutTheTeacherImmediacyScalebyGorham,1988)thesemeasuresare,albeitindirectrecordsofspecificbehaviors,butallowresearcherstoapproximatespecificbehavioraloutcomes.Third,tomitigatesomeofthepitfallswithself-reportscalesistouseSoTLscalesthatarescenariobased.Forinstance,Berry,West,andDenehey(1989)developedaself-reportSoTLscaletomeasureself-efficacyformemorytaskscalledtheMemoryforSelf-EfficacyQuestionnaire(MSEQ).TheMSEQhasbeenwidelyusedandretainsacceptablevalidityandreliability(Berryetal.,1989;West,Thorn&Bagwell,2003).Themeasureiscomprisedof40confidenceratingson10memoryscenarios.Withineachscenariothefirstquestiondescribesthesimplestmemorytaskforthescenariothenthe4thquestiondescribesthemostdifficultmemorytask.Individualsareaskedtogivetheirconfidencerating(0-100%)ontheirabilitytosuccessfullyaccomplishthememorytask.Forexample,intheMSEQindividualsaregivenascenarioabouttheirabilitytorememberagrocerylist.Thefirstquestion(whichistheeasiestmemorytask)wouldstate,“IfIheardittwice,Icouldremember2itemsfromafriend’sgrocerylistofthe12items,withouttakinganylistwithmetothestore”(Berryetal.,1989,p.713).Whereasthemostdifficultmemorytaskwouldstate,“IfIheardittwice,Icouldremember12itemsfromafriend’sgrocerylistof12items,withouttakinganylistwithmetothestore”(p.713).Confidenceratingsarethentotaledforeachmemoryscenario(e.g.,grocerylist,phonenumbers,pictures,location,words,etc.).Asyoucansee,SoTLscalessuchastheMSEQareself-report,howevertheyarerootedinpastperformanceandactualreal-lifeexamplesofwhatstudentsorteachersmayexperience.Thebenefitofthesetypesofmeasuresisthattheyattempttoobtainexternalvaliditybyputtingtherespondentsinreal-lifescenarios.InSoTLresearch,self-reportsareinevitableanddoserveapurpose.However,wearenotcryptozoologists,weareSoTLscholars.SoinordertodebunkthemythicalcreaturessuchasBigfoot,itisimportantthatwecomplementself-reportmeasureswithbehavioralmeasuresand/orselectandcreateSoTLscalesthatarerootedinactualbehavior. MeasuringYeti’sMetacognitionandUseofLearningStrategiesDoyouthinkYeti,akatheabdominalsnowman,thinksaboutthinkingoremployseffectivelearningstrategies?Likely—not!Butifhedid,howwouldweknow?Forthatmatter,howdoweknowourstudents’metacognitionanduseoflearningstrategies?WecouldusethepreviouslydescribedMAIbySchrawandDennison(1994),butitis52-itemslongandalittletheoreticallydated(e.g.,metacognitiveawarenessismorethanknowledgeandregulationofcognition).Ifyouarecombiningthiswithothermeasuresforastudy,youruntheriskofgettingparticipantfatigueandconsequentialresponsesets(e.g.,answering5toallquestionsorplayingconnectthedots).IfyouwantsomethingshorteryoucouldusetheNeedforCognitionScale(NCS)byCacioppo,Petty,Feinstein,andJarvis(1996).TheNCSisapersonalitymeasureofmetacognitionintendedtomeasuretheextenttowhichanindividualmightneedtoengagein

179

acognitiveactivity(Cacioppoetal.,1996).TheNCSconsistsof18questionsratedona5-pointLikertscaleandhasdemonstratedtobequitereliable.However,manymetacognitiveresearcherswouldsuggestthattheNCSdoesnotmeasuremetacognition,ratheritmeasuresapersonalitytrait.Oryoucanusebehavioralmeasuressuchascalibration(althoughalittletrickytocalculate),orjudgmentsoflearningorfeelingsofknowingthataremeasuredonconfidenceratings(e.g.,0-100%confident).Butthisdoesn’tmeasureholisticbeliefsaboutmetacognition(i.e.,allaspectsofmetacognition).YoumayalsousetheMSEQ,aspreviouslydescribed,butittakesalongtimetocompleteandishyper-specifictomemoryandself-efficacy—onlyonecomponentofmetacognition.Whattrulyisneeded,isabriefcurrentSoTLmeasureofmetacognition.Forexample,aresearchercouldpossiblycondensetheMAI(Schraw&Dennison,1994)andcombineitwithbehavioralmeasuressuchastheMSEQ(Berryetal.,1989)and/orpersonalitymeasuressuchastheNCS(Cacioppoetal.,1996).FutureSoTLresearchersshouldtakeupthisendeavortoallowSoTLaquick,reliable,andvalidmetacognitionmeasurethatcanbeeasilyadministered.WhatiftheYetiwasateacherandyouwantedtomeasurewhetherornothewasusingmetacognitioninhisteaching.Howwouldyoumeasurethis?ThereisanewSoTLareathatreferstothisprocessasmeta-teaching(Chen,2013).Chendescribesmeta-teachingas,

Likemeta-cognitionandmeta-learning,meta-teaching,as‘teachingaboutteaching’,canservetodesign,examineandreflectonteaching.Frompractice-orientation,itdefineswhatteachingactivityisandwhatitisfor,underwhichtheoreticalframeworkitisbeingcarriedout,andwhatexperienceandrulescanbeappliedtoit.Meanwhile,meta-teachingcanassistteachersindiscoveringdrawbacksintheteachingsystemandsolvingproblems.Thisdemonstratesthatmeta-teachingcontainssuchfunctionssuchasunderstandingteaching,changingteachingandreflectingonteaching.(p.S64)

Furthermore,Spring(1985)arguedthateffectivecollegeanduniversityteachersshouldusethemeta-teachingstrategiesofproperlessonplanningandgoalsetting,reflectingcriticallyonappropriateuseofeffectiveinstructionalstrategiestoachievetheinstructionalgoals,bothformallyandinformallymonitorstudentlearning,andconstantlyevaluatetheefficacyofchoseninstructionalstrategies.Todate,therearenoSoTLscalesthatattempttomeasuremeta-teaching.Assuch,SoTLresearchersandpsychometriciansshouldfocusonbuildingameta-teachinginventory(bothself-reportandbehavioralmeasure),whichassesshowoftenteachersengageinthesepractices.WhataboutmeasuringhowtheYeti(akastudents)useslearningstrategies?ThehallmarkandstalwartmeasureforlearningstrategiesistheMotivatedStrategiesforLearningQuestionnaire(MSLQ)byPintrich,Smith,Garcia,andMcKeachie(1991).TheMSLQisawell-documentedmeasureofmotivationandmetacognition,withconsistentlyhighreliabilityandvalidity.SeeTable2foralistofscalesandsubscales.TheMSLQiscomprisedof15separatemeasureswith81questionsona7-pointLikertscale.TheMSLQhasbeenusedinmanySoTLstudiesandcanbebrokenintosubscales.OryoucoulduseaSoTLscaledevelopedbyGurung,Weidert,and

180

Jeske(2010)calledtheStudyBehaviorsChecklist(SBC).TheSBCisa35-itemassessmentwithonegeneralfactor(i.e.,studybehaviors)thatmeasuresthingslikenotetaking,highlighting,cramming,andpracticetesting.GurungandcolleaguesreportacceptablereliabilitystatisticsfortheSBC,however,ithasnotbeenusedinmanystudiesandisundeterminedhowvalidtheinstrumentis.Table2Pintrichetal.(1991)MSLQScalesandSubscalesScale SubscaleMotivationScales

ValueComponents 1. IntrinsicGoalOrientation2. ExtrinsicGoalOrientation3. TaskValue

ExpectancyComponents

4. ControlBeliefs5. Self-EfficacyforLearningandPerformance

AffectiveComponents 6. TestAnxietyLearningStrategies

CognitiveandMetacognitive

Strategies

7. Rehearsal8. Elaboration9. Organization10. CriticalThinking11. MetacognitiveSelf0regulation

ResourceManagementStrategies

12. TimeandStudyEnvironment13. EffortRegulation14. PeerLearning15. HelpSeeking

BoththeSBCandMSLQarefinemeasures,buttheyaremissingkeyelementsofcurrentlearningstrategyresearch.Thatis,inalaudedreviewbyDunlosky,Rawson,Marsh,Nathan,andWillingham(2013),theydescribesomeofthemorecurrentpopular,effectiveandineffectivelearningstrategiesandtechniquesstudiedtoday.Theseinclude(inorderfromhigheffectivenesstoloweffectiveness)practicetesting,distributedpractice,interleavedpractice,elaborativeinterrogation,self-explanation,keywordmnemonic,visualimagery,rereading,summarization,andhighlighting.IfyoulookattheMSLQandSBC,manyofthesestrategiesaremissing.Additionally,consistentwithmysuggestionabove,anewSoTLscalethatassesseslearningstrategiesshouldincludebehavioralmeasuresaswell.Forinstance,whenaskingabouttheuseofself-explanationstrategy,SoTLresearchersshouldalsoask,“Howmanyminutesinaweekdoyouusethisstrategy?”Insum,insteadoftheMSLQandSBC,whatisneededisanupdatedSoTLscalethatlooksattheuseofcurrentlearningstrategiesandtechniquesstudentsuseordon’tuseinourclassroomswithcomplementingbehavioralmeasures. SasquatchandtheImportanceofModelTeachingCharacteristicsImagineSasquatchdoesnotroamtheforestaimlesslyassomesurmiseandinsteadisagreatteacherwhoexhibitsmodelteachingcharacteristics.Howwouldweknowsheisagreat

181

teacher?Inotherwords,howdowemeasuremodelteachingcharacteristics?InarecentSocietyofTeachingofPsychologypresidentialtaskforce,Richmondandcolleagues(2014)setouttodefineanddelineatemodelteachingcharacteristics.Intheirresearchtheydeterminedthatmodelteachersexhibitsixmodelcharacteristicswith19separatecriteria.SeeTable3foralistofthesecharacteristics.Table3Richmondetal.’s(2014)ModelTeachingCharacteristicsModelTeachingCharacteristics ModelTeachingCriteriaTraining 1. SubjectKnowledge

2. PedagogicalKnowledge3. ContinuingEducationinPedagogical

KnowledgeInstructionalMethods 4. Pedagogy

5. TeachingSkillsAssessmentProcesses 6. StudentLearningGoalsandObjectives

7. AssessmentofStudentLearningOutcomes8. ReflectiononAssessment9. ScholarshipofTeachingandLearning10. EvaluationDirectness11. EvaluationUtility

Syllabi 12. CourseTransparency13. CoursePlanning

Content 14. ScientificLiteracy15. PsychologyKnowledgeBaseandApplication16. LiberalArtsSkills17. ValuesinPsychology

StudentsEvaluationsofTeaching

18. StudentFeedback19. ReflectiononStudentFeedback

Tofurtherthislineofresearch,Boysen,Richmond,andGurung(2015)empiricallyinvestigatedthesemodelcharacteristicsandtheirrespectivecriteria.Insodoing,theydevelopeda52-itemSoTLscaledesignedtomeasuremodelteachingcompetencies.The52-itemsweremeasuredonadichotomous(e.g.,yes/no)scale.Forexample,tomeasurepedagogy,theyasked,“Classobservationrecordssupporteffectiveuseofeffectiveinstructionalmethods”(p.51).Over200psychologyteachersfromacrossthecountryatalltypesofinstitutionsofhighereducation(e.g.,communitycolleges,research-focuseduniversities,privatecolleges,etc.)participatedinthestudy.Boysenandcolleaguesfoundthatbaselinedataforthisself-reportscalecorrelatedstronglywiththeTBCandtheBig-5Inventoryofpersonality(Gosling,Rentfrow,&Swann,2003).Additionally,theyfoundthatthescalehadstrongintercorrelationsamongthemodelteachingcharacteristicsandcriteria.However,thereweresomeissueswiththisscale.First,the52-iteminventoryislongandcumbersome.Second,thenatureofadichotomousscalemakesitdifficulttoassessthefactor

182

structureofthescaleandconductinternalreliabilityanalyses.Assuch,Boysenandcolleaguesdidnotconductafactoranalyses.Accordingly,SoTLresearchersshouldconsiderchangingthescaletoaLikert-typescale,totrulyinvestigatetheefficacyofthismeasure.Third,althoughthereisinitialevidenceofreliabilityandsomevalidity,furtherresearchisneeded.Thatis,SoTLresearchersshouldinvestigatethisSoTLscalewithothermeasuresofteachingknownmeasuresofteachingeffectiveness(e.g.,ApproachestoTeachingInventorybyTrigwell&Prosser,2004;ortheTeachingGoalsInventorybyAngelo&Cross,1993).Fourth,asmentionedpreviouslyinthischapter,thereneedstobebehavioralmeasureswhichcomplementthisscale(e.g.,classroomobservationrecords).Intheend,modelteachingcharacteristicsareimportanttounderstandandasthisisanewareaofSoTLresearch,thereismuchroomforimprovementandexploration.ACalltoAction!DebunkingCryptozoologyandFocusingontheMissingLinksofSoTLMeasuresWhetheritistheYeti,Sasquatch,Bigfoot,orlaChupacabra(Ijustlikesayingthat),theaimofSoTLresearchandSoTLscalesistodemystifysuchurbanmythsandfocusonthescienceoflearningandteaching.Accordingly,howwemeasureSoTLisofutmostimportanceanddrivesattheheartofcredibilityofourfield.Therefore,Isummarilyandhumblysuggestthat:

1. ThereneedstobemoreSoTLscaledevelopmentthatintentionallytargetstheefficacyofsyllabi.

2. WhenusingpreexistingSoTLscalesalwaysincludecomplementarybehavioralmeasures.Moreso,whendevelopingSoTLscales,considerincludingbehavioralmeasureseitherintheformofchecklistsorscenario-based.

3. ThereneedstobeaSoTLscaledevelopedtomeasuremetacognitionandcurrentuseofeffectivelearningstrategies.Don’tforgettoincludethebehavioralcomponenttothesenewmeasures.

4. ThereneedstobeaSoTLscalethatassessesmeta-teachingskills.ThisisanuntappedSoTLarearipeforthepicking.

5. Thereneedstobeanaccurate,reliable,andvalidmeasureofmodelteachingcompetencies.

Please,please,considerthesesuggestionsasacalltoaction.Ifyouchoosetoanswermycalltoaction,Ihighlyencourageyoutorefertotheoutstandingchaptersinthise-booktoassistyouinthisprocess.Forinstances,refertoReganGurung’schapter(2015)onthebestpracticesinSoTLscaleuse.GurungsuggeststhatinSoTLweshouldbe“measuringtheusualsuspects”suchasself-efficacy,metacognition(seethereissafetyinnumbers),motivation,studybehaviors(YES!),etc.Or,whenwritingitemsforaSoTLscale,useclear,concise,unambiguous,andgenderneutralorculturallysensitivelanguage.Also,takeheedtoGeorgeannaWilson-Doenges’(2015)chapterinwhichsheexplicatesthestateofSoTLscalevalidation.Here,Wilson-DoengesexplainshowtoproperlyassessreliabilityandvaliditywithinthecontextandparticularlynuancednatureofSoTLresearchandsheexplainshowtodevelopSoTLscalesinlightoftheseissues.PleasealsoreadandconsiderAndrewChristopher’s(2015)chapteronhowtoselecttherightSoTLscale.ThischapterwillprovidesomewaysinwhichyoucanavoidcommontrapsorpitfallsthatSoTLscaleshave(e.g.,growingdividebetweenSoTLandthelearningsciences).

183

Christopher(2015)alsosuggeststhatwhencreatingSoTLscales,theyshouldbespecifictoasetofbehaviors,yetsimple. Intheend,ifyouconsiderthegreatadviceprovidedbythesescholars,andtakeupmycalltoaction,IknowthatyouwillcreatesomeoutstandingSoTLscalesthatwilladvanceourbelovedscholarshipofteachingandlearning.

184

ReferencesReferencesmarkedwithanasteriskindicateascale.Angelo,T.A.,&Cross,P.K.(1993).Classroomassessmenttechniques:Ahandbookforcollege

teachers.SanFrancisco,CA:Jossey-Bass.Baumeister,R.F.,Vohs,K.D.,&Funder,D.C.(2007).Psychologyasthescienceofself-reports

andfingermovements:Whateverhappenedtoactualbehavior?.PerspectivesonPsychologicalScience,2(4),396-403.doi:10.1111/j.1745-6916.2007.00051.x

*Berry,J.M.,West,R.L.&Dennehey,D.M.(1989).ReliabilityandvalidityoftheMemorySelf-EfficacyQuestionnaire.DevelopmentalPsychology,25(5),701-713.doi:10.1037/0012-1649.25.5.701

Boyer,E.L.(1990).Scholarshipreconsidered:Prioritiesoftheprofessoriate.CarnegieFoundationfortheAdvancementofTeaching.Princeton,N.J.:PrincetonUniversityPress.

Boysen,G.A.,Richmond,A.S.,&Gurung,R.A.R.(2015).Modelteachingcriteriaforpsychology:Initialdocumentationofteachers’self-reportedcompetency.ScholarshipofTeachingandLearninginPsychology,1,48-59.doi:http://dx.doi.org/10.1037/stl0000023

*Cacioppo,J.T.,Petty,R.E.,Feinstein,J.A.,&Jarvis,W.BG.(1996).Dispositionaldifferencesincognitivemotivation:Thelifeandtimesofindividualsvaryinginfeedforcognition.PsychologicalBulletin,119(2),197-253.

Chen,X.(2013).Meta-teaching:Meaningandstrategy.AfricaEducationReview,10(1),S63-S74.doi:10.1080/18146627.2013.855431

Christopher,A.N.(2015).Selectingtherightscale:Aneditor’sperspective.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

*Cullen,R.,&Harris,M.(2009).Assessinglearner-centrednessthroughcoursesyllabi.Assessment&EvaluationinHigherEducation,34(1),115-125.

Dunlosky,J.,Rawson,K.A.,Marsh,E.J.,Nathan,M.J.,&Willingham,D.T.(2013).Improvingstudents’learningwitheffectivelearningtechniquespromisingdirectionsfromcognitiveandeducationalpsychology.PsychologicalScienceinthePublicInterest,14(1),4-58.

*Gorham,J.(1988).Therelationshipbetweenverbalteacherimmediacybehaviorsandstudentlearning.CommunicationEducation,37(1),40-53.doi:10.1080/03634529009378786

*Gosling,S.D.,Rentfrow,P.J.,&SwannJr.,W.B.(2003).AverybriefmeasureoftheBig-Fivepersonalitydomains.JournalofResearchinPersonality,37,504-528.doi:10.1016/S0092-6566(03)00046-1

Gurung,R.A.R.(2015).BestpracticesinscaleuseinSoTL.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

*Gurung,R.A.,Weidert,J.,&Jeske,A.(2010).Focusingonhowstudentsstudy.JournaloftheScholarshipofTeachingandLearning,10(1),28-35.

*Keeley,J.,Smith,D.,&Buskist,W.(2006).TheTeacherBehaviorsChecklist:Factoranalysisofitsutilityforevaluatingteaching.TeachingofPsychology,33,84-91.doi:10.1207/s15328023top3302_1

185

Kirk,C.,Busler,J.,Keeley,J.,&Buskist,B.(2015).Effectivetoolsforassessingcharacteristicsofexcellentteaching:Theteacherbehaviorschecklistasexemplar.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Ku,K.L.Y.(2015).Measuringindividualdifferencesinepistemologicalbeliefs.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Landrum,R.E.,&McCarthy,M.A.(2015).Measuringcriticalthinkingskills.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Landrum,R.E.,&Stowell,J.R.(2013).Thereliabilityofstudentratingsofmasterteacherbehaviors.TeachingofPsychology,40,300-303.doi:10.1177/0098628313501043

Layous,K.,Nelson,S.K.,&Legg,A.M.(2015).Measuringwell-beinginthescholarshipofteachingandlearning.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Marek,P.,Williamson,A.,&Taglialatela,L.(2015).Measuringlearningandself-efficacy.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Matejka,K.,&Kurke,L.B.(1994).Designingagreatsyllabus.CollegeTeaching,42(3),115-117.doi:10.1080/87567555.1994.9926838

Meyerberg,J.M.,&Legg,A.M.(2015).Assessingprofessor-studentrelationshipsusingself-reportscales.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Parkes,J.,&Harris,M.B.(2002).Thepurposesofasyllabus.CollegeTeaching,50(2),55-61.doi:10.1080/87567550209595875

*Pintrich,P.R,Smith,D.A.,Garcia,T.,&McKeachie,W.J.(1991).AmanualfortheuseoftheMotivatedStrategiesforLearningQuestionnaire(MSLQ).AnnArbor,Ml:NationalCenterforResearchtoImprovePostsecondaryTeachingandLearning.

Richmond,A.S.,Becknell,J.,Slattery,J.,Morgan,R.,&Mitchell,N.(2015,August).Students’perceptionsofastudent-centeredsyllabus:Anexperimentalanalysis.PosterpresentedtheannualmeetingoftheAmericanPsychologicalAssociation,Toronto,Canada.

Richmond,A.S.,Boysen,G.A.,Gurung,R.A.R.,Tazeau,Y.N.,Meyers,S.A.,&Sciutto,M.J.(2014).Aspirationalmodelteachingcriteriaforpsychology.TeachingofPsychology,41,281-295,doi:10.1177/0098628314549699

Saville,B.K.,Zinn,T.E.,Brown,A.R.,&Marchuk,K.A.(2010).Syllabusdetailandstudents'perceptionsofteachereffectiveness.TeachingofPsychology,37,186-189.doi:10.1080/00986283.2010.488523

*Schraw,G.,&Dennison,R.S.(1994).Assessingmetacognitiveawareness.ContemporaryEducationalPsychology,19,460-475.

*Schraw,G.,Kuch,F.,&Gutierrez,A.P.(2013).Measureformeasure:Calibratingtencommonlyusedcalibrationscores.LearningandInstruction,24,48-57.doi:http://dx.doi.org/10.1016/j.learninstruc.2012.08.007

186

Schraw,G.,Kuch,F.,Gutierrez,A.,&Richmond,A.S.(2014).Exploringathree-levelmodelofcalibrationaccuracy.JournalofEducationalPsychology,106(4),1192-1202.doi:10.1037/a0036653

ScholarshipofTeachingandLearning:History(n.d.).Retrievedfromhttp://academics.georgiasouthern.edu/sotlgsu/history/

Shulman,L.S.(1993).Teachingascommunityproperty:Puttinganendtopedagogicalsolitude.Change:TheMagazineofHigherLearning,25(6),6-7.

Simons,L.(2015).Measuringservice-learningandcivicengagement.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Slattery,J.M.,&Carlson,J.F.(2005).Preparinganeffectivesyllabus:Currentbestpractices.CollegeTeaching,53,159-164.doi:10.3200/CTCH.53.4.159-164

Slattery,J.M.,Haney,M.,Richmond,A.S.,Venzke,B.Morgan,R.K.,&Mitchell,N.(2014,August).Projectsyllabus:Studentresponsestosyllabi.AsymposiumpresentedattheannualmeetingoftheAmericanPsychologicalAssociation,Washington,D.C.

Spring,H.T.(1985).Teacherdecisionmaking:Ametacognitiveapproach.TheReadingTeacher,290-295.

*Trigwell,K.,&Prosser,M.(2004).Developmentanduseoftheapproachestoteachinginventory.EducationalPsychologyReview,16(4),409-424.

West,R.L.,Thorn,R.M.,&Bagwell,D.K.(2003).Memoryperformanceandbeliefsasafunctionofgoalsettingandaging.Psychologyandaging,18(1),111.

Wilson-Doenges,G.(2015).ThestateofscalevalidationinSoTLresearchinpsychology.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.

Zabel,K.L.,&Heger,A(2015).Studentengagementtowardcoursework:Measures,considerations,andfuturedirections.InR.S.Jhangiani,J.D.Troisi,B.Fleck,A.M.Legg,&H.D.Hussey(Eds.),Acompendiumofscalesforuseinthescholarshipofteachingandlearning.