Evaluating Creative Teaching Practices and Collaborative Learning Environments

C R E S S T / U C L A

UCLA Graduate School of Education & Information StudiesCenter for the Study of Evaluation

National Center for Research on Evaluation, Standards, and Student Testing

Evaluating Creative Teaching Practices and Collaborative Learning Environments

Ann Mastergeorge and Ingrid Roberson

Annual AERA ConventionApril 1, 2002 New Orleans, LA

C R E S S T / U C L A

Overview

The Construct of Engagement in School Reform

Methodological Issues in Measuring Engagement in School Reform

Discussion and Implications of Engagement for Classroom Practice

C R E S S T / U C L A

Introductionto Creative Learning

Communities

Supports school reform in schools across the country serving at-risk student populations

Promotes change centered on creative teaching practices and collaborative learning environments

Emphasizes the enhancement in students’ joy of learning in engaged learning environments

C R E S S T / U C L A

Procedural Engagement

Focuses on student on-task behavior in learning activities

Emphasizes student concentration and completion of learning activities

Substantive Engagement

Focuses on the emotional quality of student’s involvement in learning activities

Emphasizes student’s enjoyment of learning activities since it is self-motivating and self-reinforcing

The Construct of Engagementin School Reform

C R E S S T / U C L A

Developing A Dynamic Model of Engagement

Substantive Engagement depends on both student and teacher engagement in classroom learning activities

Includes the social dimension of learning

Requires a high degree of reciprocity

Learning Environments and Teaching Practices construct opportunities for students to experience engagement

Communicates support and care

Allows for student choice and control

Models enthusiasm for learning

C R E S S T / U C L A

Student Engagement

Teacher Engagement

 Creative Learning

Community

Learning links

Teaching links

Shifts in attitude

Shifts in attitude

Classroom Collaboration

Learning Communities

Teaching Communities

C R E S S T / U C L A

Methodological Issues in Measuring Engagement in School Reform

Multiple Methods of Evaluation

Utilizes quantitative and qualitative approaches

Implementation Process of School Reform

Impact of School Reform on Engagement

Employs a theory of action

Provides focal points for the evaluation

Suggests a sequence in which various effects may be expected

C R E S S T / U C L A

Anchor incurriculum, instruction

& assessment

Attend to cultural and structural change

Providematerials

Creative Teaching Practice

Professional Development &Accountability

School Environment

Leadership & Goals

Facilitate change

Mobilize change

Build a shared vision

Connect to student learning

Provide Financial Support

ProvideTechnical Support Embed in

school culture

Build a community

Attend to changingattitudes, beliefs, & values

Promote and provide time for on-going professional inquiry

Embed in classroom practice

Provide opportunities for teacher collaboration

Align withproject goals

Provide the type and timely information for

project adjustmentsDevelop measures

aligned with creative teaching practices

Align withstandards

Link to curriculum

Align with assessments (standardized and performance)

Shift teacher attitudes, beliefs, and

expectationsIdentify

innovative practices Embed in school cultureEngaged Student

LearningPromote the “joy of learning” through exploration, collaboration, and interpretation

Improve student attitudes towards learning

C R E S S T / U C L A

Measuring the Implementation Process

Features Five Dimensions of School Reform Framework

Goals

Leadership

School Environment (Culture and Structures)

Professional Development

Accountability

Focuses on the implementation of creative teaching practices

Teacher Survey

Teacher Interview

Classroom Practice Observation Protocol

C R E S S T / U C L A

Teacher Survey

C R E S S T / U C L A

Teacher Interview

C R E S S T / U C L A

Measuring the Impact on Engagement

Features both Procedural and Substantive Engagement Theory

Focuses on Student Engagement

Teacher Survey

Teacher Interview

Student Survey

Classroom Practice Observation Protocol

C R E S S T / U C L A

Teacher Survey

C R E S S T / U C L A

Teacher Interview

C R E S S T / U C L A

Student Survey

C R E S S T / U C L A

Classroom Practice Observation Protocol

Sensitive to initial changes in teaching practices

Appropriate for innovative and creative teaching practices

Captures students’ attention to and enjoyment of learning activities

C R E S S T / U C L A

Dimensions of Classroom Practice

Challenge of Lesson Activity

Implementation of Lesson Activity

Procedural Engagement

Substantive Engagement

Quality of Instructional Discussion

Quality of Instructional Feedback

C R E S S T / U C L A

Implementation of the Learning Activities

Notappli-cableto thisactivity

Challenge of the Lesson Activities

1 2 3 4 Learning activitiesinvolve students intasks that do notrequire any degreeof complex thinkingand do not engagestudents withsubstantive contentmaterial.

(Little or nocomplex thinking)

Example:Using encyclopedias,students record thename of medicinalplants, its scientificname, where it ismost commonlygrown, how tall itcan grow, and how itcan be usedmedicinally onto aworksheet.

Learning activitiesinvolve students intasks that requiresome moderatelycomplex thinking.Student’sengagement withthe material doesnot promotelearning in arigorous way.

(Some complexthinking)

Example:Students recordinformation fromencyclopedias to aworksheet and at theend, answer thefollowing question,"What was the mostsurprising thing youlearned aboutmedicinal plants?”

At least some of thelearning activities requirestrongly complex thinkingas a major focus of thelesson. This level requiresthe use of higher ordercognitive functions, takingstudents beyond recall,recognition, andreproduction ofinformation to evaluation;analysis; synthesis; andproduction of arguments,ideas, and performances.Students may be asked tosynthesize ideas; analyzecause and effect; identifya problem and posereasonable solutions;hypothesize, speculategiving details orjustification; defendopinions or argue aposition with evidence;evaluate; analyze; ordetermine bias, values,intent.

(Mostly high orderthinking)

Example:Students research amedicinal plant, choosingon their own what theythink is importantinformation about theplant. They then createan informationalpamphlet about the plantusing that information.

Much or all of the learningactivities require stronglycomplex thinking as a majorfocus of the lesson. Studentsalso engage in substantivecontent material. Studentsmay be asked to analyzecause and effect, identify aproblem and posereasonable solutions,speculate giving details orjustification, defendopinions or argue a positionwith evidence to a greatextent.

(Almost all complexthinking)

Example:Students research severalmedicinal plants and writea report selecting a fewplants that may be useful totheir families, using specificdetails about the plant tosupport thier choices. Thereport is used to make apresentation to the classabout why the plants theychose are useful.

NA

C R E S S T / U C L A

C R E S S T / U C L A

Challenge of Lesson Activity Exemplar

Teacher: Let’s think about all of the different conditions.

Student: My experiment needs more water.

Teacher: I heard someone else say too cold… so cold or warm.

Student: Ours is darker and theirs is lighter [amount of light] .

Teacher: Can you think of anything else?

Student: I think we should put less water and less worms.. with theworms they will just dry, will shrivel and do nothing and the water will just get everything damp.

Teacher: What do you think is the most important thing to make theleaves decompose?

Student: Dry, the leaves crumble up. It’s like they turn into dirt.

Teacher: Maybe one thing we can do this week, half put water and the other half put no water and keep everything else the same. (CO.10.08.YR2)

C R E S S T / U C L A

C R E S S T / U C L A

C R E S S T / U C L A

Implementation of Lesson Activity Exemplar

After reviewing both the proof reading process and scoring rubric (both clearly written on chart paper), teacher runs through following example.

Dear Curator:I really love the ring of the bell. I also thought the water wheel was cool.Finally, the machines were interesting to see.Thank You, Ms. X

Teacher: Alright, so this letter… Based on that, where would you putthem if you were the teacher?

Student: 2, though it does have an introduction.

Student: 2, it doesn’t give them a sense and is supposed to be in paragraphs.

Teacher: Anybody agree or disagree and why?

Student: It looks like a letter but it doesn’t sound like a letter (CO.01.03.YR2)

C R E S S T / U C L A

1 2 3 4 10-29% of thestudents appearto be procedurally engaged.

Example:Few studentsconduct their lightbulb experimentswhile the rest of thestudents wanderaround the room,chatting with otherstudents aboutunrelated topics orplay with eachother.

NA

Notappli-cableto thisactivity

30-59% of thestudents appearto be procedurally engaged.

Example:Some studentsconduct their scienceexperiment but abouthalf of the classwanders around theroom or sits at theirdesks chatting withother students aboutunrelated topics orstares out into spaceday dreaming,

60-79% of thestudents appearto be procedurally engaged.

Example:Most students areon-task, conductingtheir experiment.They write theirscience notes,detailing their stepsand results. Moststudents complete theexperiment and mostcomplete or nearlycomplete theirscience notes.

80-100%of thestudents appearto be procedurally engaged.

Example:Almost all students areon-task, trying to get thelight bulb to work, usingdifferent metal wires.They note theirobservations in theirscience journal. Moststudents complete both theexperiment and theirwriting-up of the science

experiment in theirjournal.

Approximately what percent of procedurally engaged?

Procedural Engagement

C R E S S T / U C L A

Procedural Engagement Exemplar

All students are on-task, attentive to the teacher. Some students raise their hands to answer teacher questions.

Teacher: Is this a sentence?

Student: Yes.

Teacher: So what does it have to have at the beginning?

Student: A capital letter.

Teacher: What does it have to have at the end?

Student: A question mark.

Teacher: What do I have to do to show these words came out of my mouth? [Three students raise their hand]...come up to the board X.

Student goes to board and puts quotation marks around the sentence.(CO.02.01.YR2)

C R E S S T / U C L A

NA

Notappli-cableto thisactivity

10-29% of thestudents appearto be substantively engaged.

Example:Most students areeither not doing theexperiment, playingor talking with otherstudents or ifconducting theexperiment, they arerushing through it,being careless,paying more attentionto unrelated talk.

30-59% of thestudents appearto be substantively engaged.

Example:About half of thestudent are notactively engaged,even if they areconducting theexperiment, theirattention is focusedelsewhere, listening ortalking to otherstudents, playing withtheir pencils, lookingoutside the window.

60-79% of thestudents appearto be substantivelyengaged.

Example:Most student are activelyengaged. Some studentstalk excitedly with theirneighbors saying, "Didyou see that?, That ispretty neat; Did yours dothat?" Other students,smile to themselves astheir light bulbs lights up,while others still tryingscrunch up their faces indeep concentration.

80-100% of thestudents appearto be substantively engaged.

Example:Almost all student are activelyengaged. Many students shout,"It works!, I did it! or Its alive!"when their light bulb works. Somestudents smile, their eyeswidening, as their light bulb goeson. Other students still tryingreview their notes or ask anotherstudent, “Hey, can you help me?”Near the end, students can beheard talking with each otherabout which metal worked better

as a conducter.

Approximately what percent of students are substantively engaged?

Substantive Engagement

1 2 3 4

C R E S S T / U C L A

Substantive Engagement Exemplar

The students are all listening. They are riveted, looking at the teacher and the book intently. The teacher engages them in a discussion with 90% of the students participating. Students’ hands shoot up in the air waving back and forth frantically with wiggling fingers, going "oh, oh”.

Teacher: What woke him up?

Student: The sun.

Teacher: X, what wakes you up?

Student: My mom.

Student: Sometimes my dog comes in and licks me.

Student: Sometimes my brother.

Student: My mom’s boyfriend hits me with a pillow.

Student: My dad gets me up.

Teacher: So our caterpillar comes out of the egg and how did hefeel? He was what?

Students: HUNGRY!!! (CO.02.03.YR2)

C R E S S T / U C L A

NA

Notappli-cableto thisactivity

1 2 3 4 A discussion doesnot take place at allor interaction betweenteacher and students ispredominantlyrecitation style, withteacher mediating allquestions and answers.The teacher’s questionsare close-ended,known-answerquestions.

(No discussion/Teacher guided, Q&A.)

Example:Teacher asks, “16divided by 4 is?” “4,”answers studentscollectively.

Teacher makes someattempt to engagestudents in truediscussion with unevenresults. Some of theteacher’s questions areopen-ended. Theremay be some attemptto have studentsrespond to otherstudents or invitationsto comment about abook. Teacher’sattempt at engagingstudents may fail.

(Teacher guided Q&Awith few open-endedquestions.)

Example:Teacher asks, " I have16 apples I want toshare with my fourfriends. Do I haveenough?" "Yes", thestudents respondcollectively. “Howmany would eachfriend get?” “4,”answer the studentstogether.

Most of teacher’squestions are of highquality. Adequate timeis available for studentsto respond and teacheractivity solicits studentinput (e.g., "Tell mewhy you think that.""Can you say a littlemore about that?").Teacher builds onstudent contributions.There is some student tostudent discussionregarding thesubject/classroomactivity.

(High qualityopen-ended questionswith few Q & As.)

Example:Teacher says, "I have 16apples that I want toshare with four friends,how many would eachfriend get?" Teacherwaits a few minutes andthen says, "Who can tellme the answer and howyou figured it out?Student responds, "Idrew four circles andthen passed out anapple to each circleuntil they were allpassed out."

Quality of Instructional Discussion

Classroom interactionrepresents true discussion.Students initiate topics andmake unsolicited, on-topiccontributions. Studentsformulate many questions.Teacher’s questions areuniformly high quality withadequate time for students torespond. Teacher builds onstudents’ contributions, andstudents build on each other’scontributions. Student tostudent discussion about thesubject/classroom activity issignificant.

(Discussion betweenstudents and teachersand/or among students.)

Example:The student called uponresponds, "I drew four circlesand then passed out an appleto each circle until they wereall passed out." Anotherstudent calls out, "I did itanother way". Teacher buildson this comment by saying,“Let’s discuss differentstrategies used."Another student says, “I knew4 plus 4 is 8 and 8 plus 8 is16, so I counted by fours.There were 4 fours.“

C R E S S T / U C L A

C R E S S T / U C L A

Quality of Instructional Discussion Exemplar

At one table, there are three rather lively boys and as each of them works on drawing two parallel lines the following discussion ensues:

Student 2: “That’s parallel,” referring to the littlest one’s two parallel

lines.

Student 4: “Can I make it parallel?”

Student 2: “That is parallel. It doesn’t matter the size of the other

line, if they don’t intersect, then they’re parallel.”

Student 1: “If they don’t intersect, they’re parallel.”

The little one is still unconvinced by the two boys, looking at his board with a confused expression. [Student 2] continues to teach the other student the concept of parallel lines, emphasizing the word intersect and then upon seeing the continued state of confusion lets out a loud sigh and starts again. He explains that the length of the line segment or how close they are doesn’t matter as long as they don’t intersect.

(CO.05.03.YR2)

C R E S S T / U C L A

Feedback is uniformly highquality. Provision is madefor students to usefeedback in their learning.Feedback fully supports theattainment of theinstructional goals.Feedback is provided foralmost all studentsdemonstratingneed.

(Uniformly high quality)

Example:While the teacher rovesaround the room helpingalmost all students, shesays, "John, you used agreat adjective becauseyou described how a pillbug looks. Now can youuse an adjective thatdescribes how it feels?"

Feedback is mostlyhigh quality.Feedback mostlysupports theinstructional goals.It is provided eitherconsistentlythroughout theobservation periodor in a focused wayduring a portion ofthe period.Feedback isprovided for moststudentsdemonstratingneed.

(Mostly highquality)

Example:Teacher says, “I likeyour sentence.Bumpy is a gooddescribing word.”Helping moststudent with theirwriting, heoccasionally says,"good job or needsmore work.”

1 2 3 4 Feedback is eithernot provided or is ofuniformly poorquality. Feedbackmay beinappropriate.Feedback does notsupport instructionalgoals. Feedback isprovided only for afew studentsdemonstrating need.

(Not provided ormostly poor quality)

Example:First grade studentsare working on awriting assignmentwith the sentencestarter "A pill bugis…" Teacher sits athis desk whilestudents attempt todo the assignment.

Feedback isinconsistent inquality. Elements ofhigh quality may bepresent during asmall portion of theobservation orminimallyinformative feedbackthat only somewhatsupports theinstructional goalsmay be giventhroughout theobservation.Feedback is providedfor some studentsdemonstrating need.

(Inconsistent inquality)

Example:Teacher rovesaround the roomsaying to somestudents "good job,nice work, or youneed to start writing,this is not anadjective".

NANotappli-cableto thisactivity

Quality of Instructional Feedback

C R E S S T / U C L A

C R E S S T / U C L A

Quality of Instructional Feedback Exemplar

Teacher: “Did you measure first? If you’re going to do me, put my name in, and then you’ll have to do an estimate.”

Both students quickly go to their papers to write down their estimates on a small piece of paper.

Teacher: “Now this is in meters,” upon seeing one student write down 5.

The student nods knowingly but continues to write down the number 5.

Teacher: “Do you think I’m 5 meters high?”

He looks at her questioningly, not understanding. So she takes the yardstick and starts to raise it slowly.

Teacher: “One meter.” Raises it a meter. “Two meters.” Raises it again. “Three meters.”

Student: “Oh, two!” He vigorously erases the number five and writes down the number two. (CO.06.01.YR2)

C R E S S T / U C L A

Comparison of All and Matched Mean Ratings

Domain 2000All Scores

2001All Scores

AllMeanChange

2000MatchedScores

2001MatchedScores

MatchedMeanChange

Challenge ofLesson Activity

2.42 2.46 .02 2.21 2.43 .22

Implementationof LessonActivity

2.88 2.86 -.02 2.54 2.91 .37

ProceduralEngagement

3.20 3.45 .25 2.96 3.52 .56

SubstantiveEngagement

2.32 2.86 .54 2.13 2.91 .78

Quality ofInstructionalDiscussion

2.20 2.49 .29 2.17 2.22 .05

Quality ofInstructionalFeedback

2.46 2.44 -.02 2.33 2.43 .10

C R E S S T / U C L A

Comparison of Year 1 and Year 2 mean ratings by domain across all case study schools

C R E S S T / U C L A

Comparison of Year 1 and Year 2 matched mean ratings by domain across all case study schools

C R E S S T / U C L A

Discussion and Implications for Classroom Practice

Documenting Shifts in Classroom Practice from Year 1 to Year 2

Issues in Accountability and Sustainability of School Reform

Re-focusing, Re-thinking, and Re-tooling Measures of Classroom Practice

C R E S S T / U C L A

Documenting Shifts in Classroom Practice

Overall, small effect on Teacher Practice (Effect Size of .33) and Student Engagement (Effect Size of .201)

Between Schools, difference in degree of implementation of creative practices

Professional development linked to promoted teaching practices

Assessments to monitor progress

C R E S S T / U C L A

Issues in Accountability and Sustainability in School Reform

Combining and creating assessments that are accessible for progress monitoring

Improvements

Inquiry

Creating and sustaining supportive infrastructures to sustain school reform

Leadership

Community Culture

C R E S S T / U C L A

Re-focusing, Re-thinking, and Re-tooling

Measures of Classroom Practice

Re-focus efforts to include realistic expectations for reform implementation

Re-think the product and outcome model of classroom measures: measures should reflect process in action

Re-tool measures to reflect conceptual and procedural realities of classroom practice

C R E S S T / U C L A

Paper will be posted on the UCLA CSE/CRESST website:

www.cse.ucla.edu 

Ann Mastergeorge

Phone 530-754-9629

E-mail ammastergeorge@ucdavis.edu

Ingrid Roberson

Phone 310-794-4404

E-mail roberson@cse.ucla.edu