CrICET: Learning Studios

Elizabeth FosterVice [email protected]

Don GlassDirector of Evaluation and Field [email protected]

Learn more at nctaf.org

Learning Studios:

October 23, 2014 | CEM 2014

National Commission on Teaching & America’s Future (NCTAF)

Founded 20 years ago by Linda Darling-Hammond;

Chaired by Secretary Richard Riley and former ECS

President Ted Sanders;

20 years of leadership in promoting innovative practices in

teaching and learning;

Deep understanding about the teaching conditions that

facilitate successful student learning.

The History of Learning Studios

Collaboration Time and Feedback

Launched in 2009 through a cooperative agreement with NASA

in two school districts in Maryland;

Grew to 35 sites over the years including some in NH, MD, and

VA with variations in implementation and sustainability;

Partners have included NASA, NIH, Boeing, Northrop

Grumman, Chesapeake Bay Conservancy;

Independent evaluation by West Ed over the last three years;

Focused now on sharing what we have learned

Learning Studios

Learning Studios took three separate

efforts that may be seen to be

competing strategies, and have

integrated them in a systematic way.

(PLC + PBL) x OCE = Better Teaching, Faster

Educator

Teams

STEM

ExpertsStudents

Program Components

Program Components:▪ Professional Learning Communities (PLC)▪ Project-Based Learning (PBL)▪ Outside Content Experts (OCE)

Program Participants:

Working Together to Improve STEM Curriculum

Program Activities

Working Together to Improve STEM Curriculum

Embedded Evaluation Design

• Embed evaluation information gathering and reporting into program routines and expectations

• Provide design and evaluation capacity-building for program participants at their level-of-use

• Ground data collection and use in relevant curriculum design and timely assessment feedback to reduce evaluation burden

• Include teacher input and feedback in design cycles for tools

Expert-User Focus

Groups

Evaluation Closer to Practice

Nested Evaluation Design

External Program

Evaluation

Internal Evaluation and

Program Monitoring

Practitioner Inquiry

Student Assessment

and Feedback

Rich teaching and learning data for timely, formative feedback for students and teachers

Periodic, formative data from program

sources, key informants, and

observations

Systematic, accurate, reliable, and valid

formative and summative overall data with rich

exploratory cases

Evaluation Use Across the Levels

Program Evaluation

Team Member Evaluation Focus Tools

Teams of EDUCATORS • Collaborative

Culture (PLC)• Curricular

Improvement (PBL)• OCE Roles and

Characteristics

• Baseline and Interim Self-ratings• Collaboration Discussion Protocol• Curriculum Map• PBL Checklist• Teacher Final Survey• OCE Survey, Interviews, and

Observations

STEM Experts

STUDENTS • Student Learning• Student Interest

• Student Retrospective Survey

Internal Formative Evaluation | External Third-Party Evaluation

Internal and External Evaluation Working Together

Collaborative CultureCollaborative Culture

DomainInitial/Final

Ratings* Goals Strategies * Evidence

Maintaining an Inquiry Stance

❑ 0❑ 1❑ 2❑ 3❑ 4

❑ 0❑ 1❑ 2❑ 3❑ 4

Survey Responses (Survey Monkey)

Organize Data for Team Summary

Discussion Reports

Team Discussion: Take stock, set goals,

select strategies

Collaboration Discussion Protocol

Final Self-Ratings based on Evidence

* Collaboration Discussion Protocol provides some evidence for these goals

Systematically Improving Our Work

Collaborative Culture

NCLE Collaborative Teams (2012)

STEM Teachers in PLCs (2011)

Team Up (2010)EdWeek(2010)

Pearson LT Readiness Instrument

(2009)

De-privatizing PracticeCollective Responsibility Collective Responsibility Perseverance

Creating Collaborative Culture

Collective Responsibility TrustA Single School Subject

Collective responsibility Job-alike teamsPerseverance

Experience with collaboration

Maintaining an Inquiry Stance

Good Facilitation Self-directed reflection ProtocolsTrained peer facilitatorsPerseverance

Teacher Workgroup facilitatorCoach/Content expert

Using Evidence Effectively

Use of Student Data and Student Work

Authentic assessment

Shared AgreementsShared Values and Goals Shared Values and Goals Potential Buy-in

Supporting Collaboration Systematically

Leadership SupportTime

Strong leadership supportStable settings

Stable settings Site administratorAvailable settingsTiming/bandwidth

Informing the Design of Tools with the Literature


Gathering Data to Inform Our Work

Using Evaluation to Inform Their Work


CULTIVATING COLLABORATIVE CULTUREHS SUM 2013 Q4I. Deprivatizing Practice 3.25II. Creating Collaborative Culture 3.17III. Maintaining an Inquiry Stance 3.17IV. Using Evidence Effectively 3.38V. Shared Agreements 3.42VI. Supporting Collaboration Systemically

3.38

Q2: Reflections, Evidence, and Strategies

1. Which domain(s) did you choose to work on? Creating Collaborative Culture and Maintaining an Inquiry Stance

2. What progress have you made? Collaborative Culture & Common Planning Time ( the last 4 "A" Day). All teachers from the content areas (AFNR, Algebra, Biology, English, Environmental Science and FOT (Foundations of Technology) meet to elaborate on student progress as established this summer. The deadlines for assessments, submittal of evidence and workforce partnership meetings are listed in our "Mapping the Year" document per quarter. We've created a STEM Binder for our minutes and to include student artifacts.3. What are you planning to do? Continuing to meet weekly. Continued collaboration with our workforce partners and students. Reflection on student artifacts both future and present. Providing feedback to colleagues on lesson plans before they are presented to students. Differentiation of assessments for the different modalities of student learners.

Using Evaluation to Inform Our Work


Teacher Findings

Systematically Improving Our Work

PBL Ratings Base/Final

* Goals Strategies * Evidence

Student Voice and

Choice

❑ 0❑ 1❑ 2❑ 3❑ 4

❑ 0❑ 1❑ 2❑ 3❑ 4

* CMAP and PBL Checklists provide evidence for meeting these goals.

Curricular Improvement

PBL Survey with Self-Ratings

Take Stock, Set Goals, and Select

Strategies

Curriculum Maps (CMAP)

PBL ChecklistsTeaching and

Learning Reviews

Final PBL Survey with Self-Ratings

and Evidence

Informing the Design of Tools with the Literature

Curriculum Map

Understanding by Design (UbD)

Project-Based Learning(PBL)

DESIRED RESULTS

Significant Content

Driving Question

Need to Know

ASSESSMENT EVIDENCERevision and Reflection

Public Audience

LEARNING PLAN

21c Competencies

In-Depth Inquiry

Voice and Choice





DESIRED RESULTS

Big Idea | Enduring Understanding

Driving Question | Essential Question

NGSS: Science and Engineering

Practices

NGSS: Disciplinary Core Ideas (DCI)

NGSS: Crosscutting

Concepts

CCSS: Common Core Math/ELA Connections

ASSESSMENT EVIDENCE

NGSS: Performance Expectations

Baseline/Diagnostic Assessment

Formative Assessments

Summative Assessment

LEARNING PLANWorkforce

PartnerScience

TechnologyMath ELA/Arts

Week 1

Week 2…

Using Technology to Work Together


Using Data to Inform Our Work


Student Findings (WestEd)

Students’ views of changes in science investigation abilities

*Note: The difference in average score is significant at the p<0.001 level for all questions

Item

Before

Completing

the Projects

(Average)

After

Completing the

Projects

(Average)

Gain from

Before to

After*

Ability to Plan Projects

Planning how to do the project 2.98 3.94 0.96

Coming up with a problem or question 2.65 3.69 1.04

Designing a project that addresses the problem or question 2.83 3.78 0.95

Ability to Carry Out Projects

Carrying out the project in an organized way 2.81 3.77 0.96

Measuring and recording data 2.99 3.83 0.84

Knowing where and how to get needed information 3.08 3.90 0.82

Building things 3.10 3.74 0.64


Students’ views of changes in science investigation abilities


Item

Before

Completing

the Projects

(Average)

After

Completing the

Projects

(Average)

Gain from

Before to

After*

Ability to Make Sense of Data and Information

Analyzing data 2.83 3.68 0.85

Displaying data 3.07 3.80 0.73

Making sense of information 3.07 3.81 0.74

Making conclusions 3.11 3.81 0.70

Judging how well that something I built works 3.09 3.80 0.71

Communicating results to others 3.12 3.83 0.71


Students’ views of and interest in STEM

Item

Before the

Project

(Average)

After

Completing the

Project

(Average)

Gain from

Before to

After*

Views About STEM

I like math 3.28 3.60 0.32

I am interested in taking more math in school 3.07 3.45 0.38

I like technology and engineering 3.06 3.50 0.44

I am interested in taking more science in school 2.91 3.37 0.46

I am interested in taking more technology in school 2.97 3.34 0.37

The science in this class is relevant to everyday life 2.97 3.48 0.51

I like science 3.09 3.60 0.51



Students’ views of and interest in STEM

Item

Before the

Project

(Average)

After

Completing the

Project

(Average)

Gain from

Before to

After*

Interest in STEM After High School

I would be comfortable with a job that requires using math 3.06 3.33 0.27

I would be comfortable with a job that requires using

science

2.97 3.25 0.28

I am interested in becoming a mathematician 2.33 2.55 0.22

I am interested in going to college or university 4.23 4.48 0.25

I am interested in becoming a scientist 2.41 2.67 0.26

I am interested in becoming an engineer 2.62 2.85 0.23

I am interested in going to community college 2.18 2.49 0.31

I am interested in becoming a computer scientist or other

technology worker

2.46 2.67 0.21

Learning Studios Toolkit

Theory of Action

PLC’s focused on STEM PBL with Outside Content Experts

(PLC → PBL) OCE

Stories of Improvement

PLC: Professional Learning CommunitiesPBL: Problem-Based LearningOCE: Outside Content Experts

Tools, Case Stories, and Supports Along a Developmental Continuum

Teacher voice!

Thank you!

Elizabeth FosterVice [email protected]

Don GlassDirector of Evaluation and Field [email protected]

Learn more at nctaf.org

Education

CrICET: Learning Studios