www.edvantia.org

Organizational Consultant, Critical Friend,

and Evaluator: The Value and Challenge of Flexible

RolesKristine L. Chadwick, Ph.D.

Evaluator forThe Spatial Intelligence and Learning Center (SILC)The Science of Learning Center on Visual Language

and Visual Learning (VL2)

www.edvantia.org

SILC and VL2 Evaluations

• NSF encouraged centers to change evaluators

• Have served as SILC’s evaluator since its third year of operation (2008-2009)

• Served as VL2’s evaluator during its fourth year of operation (2009-2010) (and will continue in Years 6-10)

2

www.edvantia.org

What is SILC? (http://spatiallearning.org)

“The Spatial Intelligence and Learning Center (SILC) brings together scientists and educators from Temple University, Northwestern University, the University of Chicago, the University of Pennsylvania, and the Chicago Public Schools (CPS) to pursue the overarching goals of

– understanding spatial learning– using this knowledge to develop programs and technologies that will

transform educational practice, helping learners to develop the skills required to compete in a global economy

SILC participants include researchers from cognitive science, psychology, computer science, education, and neuroscience, as well as practicing geoscientists and engineers who are particularly interested in spatial thinking in their fields, and teachers in the CPS.”

3

www.edvantia.org

What is VL2?(http://vl2.gallaudet.edu)

“The purpose of VL2 is to gain a greater understanding ofthe biological, cognitive, linguistic, sociocultural, and pedagogical conditions that influence the acquisition of language and knowledge through the visual modality. Gallaudet University houses VL2, and brings together deaf and hearing researchers and educators from a variety of disciplines and institutions to study how language and literacy develop in deaf individuals.”4

www.edvantia.org

SILC and VL2 Evaluation Priorities

• Satisfy NSF evaluation “requirements”

• Present well at Year 4 high-stakes site visit

• Consult on center management issues

• Gather and report data on aspects of center PIs knew less about (e.g., outreach for SILC, training at affiliate IHEs for VL2)5

www.edvantia.org

Articulating Center Theory• Theory of the center, not its research per

se– Value added of center-mode research– Challenging for the center scientists

• Immediate and intermediate outcome categories– Centerness– Dissemination– Interdisciplinarity– Outreach– Training– Tools and resources

6

www.edvantia.org

Program Theory / Logic Model• Long-term Outcomes (> 10 years)

– Social Capital (reputation, networks)– Intellectual Capital (research advancement)– Physical Capital (tools and resources)– Human Capital (diverse interdisciplinary

researchers)• Impacts based on NSF goals

– advancing science– applying science to specific technological,

educational, and workforce challenges– enabling science to capitalize on new

opportunities and discoveries and respond to new challenges

7

www.edvantia.org

Evaluation Approach

• Operationalizing, measuring (or documenting) resources, activities, outputs, immediate outcomes in logic model

• Reflect their growth as centers• Consulting on center processes• Outsider viewpoint, critical friend

8

www.edvantia.org

NSF Requirements• Center-level evaluation

requirements not very clear– Show value added of center-mode

functioning– Present outcomes of center at site

visits• NSF need: To demonstrate to the

National Science Board the worth or value of the Science of Learning Centers

9

www.edvantia.org

Limited Resource Tensions

External Evaluator• Show merit or worth of

the center• Demonstrate how

center-mode functioning better than individual RO1 grants

• Focus on outcomes

Consultant & Critical Friend• Fix some process gaps• Work through the logic

model (are activities happening and resulting in expected outputs?)

• Attend to developmental stage of center

• Focus on process and some immediate outcomes10

www.edvantia.org

Not enough resources to attend to all needsCaption for this vacation photo: “Up a Tree”

11

www.edvantia.org

Conflicting Role Example

• SILC Year 3 Evaluation– Process focused– Organizational consulting– Conversation and informal feedback

and suggestions to Pis– SILC leadership satisfied– Site visit team wants outcome data

12

www.edvantia.org

Example• SILC & VL2 Year 4 Evaluation

– Outcomes oriented– Bibliometric analyses

• Limited corpus of publications• Not useful to center at this early stage

– Less feedback mid-year– PIs less satisfied– Fewer questions/critiques from site

visit team

13

www.edvantia.org

Balancing Tensions in a Most Unusual Way

• Years 6-7 will bring back focus on process (new 5-year funding cycle)

• Update logic model and key measures• Focus also on intermediate outcomes

(now appropriate in center’s lifecycle)• “SLC Evaluation Network” economies

of scale may result in better– center-specific evaluation– SLC program-wide evaluation

14

www.edvantia.org

Thank You!

Kristine Chadwick, Ph.D.Executive Director

Evaluation and Applied Research Initiatives

304.347.0429kristine.chadwick@edvantia.org

15