Evaluation and Research:Evaluation and Research:Dual Purpose Data CollectionDual Purpose Data Collection

Ann E. AustinAnn E. AustinProfessor, Michigan State UniversityProfessor, Michigan State University

ADVANCE PI MeetingADVANCE PI Meeting

March 5, 2013March 5, 2013

Key Questions addressed in This Key Questions addressed in This SessionSession

How to use evaluation data in How to use evaluation data in researchresearch

How to frame single institution data How to frame single institution data for publicationfor publication

Suggestion 1:Suggestion 1:

Take a dual purpose perspective from the start, rather than waiting until late in the project or after it is finished

– Take the time to assemble your project team, including internal and external evaluators, to discuss project goals, and ways to engage in both evaluation and research

Suggestion 2:Suggestion 2:

Consider the differences between evaluation and research questions

– Framing questions from the point of view of advancing knowledge (research) versus asking questions inform and assess project development, outcomes, and institutional practice (evaluation)

Suggestion 3:Suggestion 3:

Read literature that enables one to see connections, move from local examples to broader questions, and situate evaluation questions within larger contexts

– Consider the literature in sociology, educational psychology, political science, higher education, and women’s studies related to organizational change, leadership, faculty work and roles, gender issues, etc.

Suggestion 4:Suggestion 4:Use a conceptual framework to view issues and questions (which helps to situate what one is studying in a context beyond a single case)

– Conceptual frameworks highlight key questions, relationships, factors, and variables to consider in your work

– Conceptual frameworks help situate local questions or issues in ways that may be relevant to other institutions and to scholars studying related issues

Suggestion 5:Suggestion 5:

Take note of emerging questions and observations that may lead to broader research

– As the project develops, observe and note the issues that emerge, including challenges, barriers, and facilitators. Consider how your observations either confirm or contradict what you have seen in the related literature. These may be areas for shifting evaluation questions toward research questions.

Suggestion 6:Suggestion 6:

Ensure that IRB approvals are in place from the start so that data can be used for publication

– All data used in research publications must have been collected under appropriate IRB guidelines and approvals.

– A good practice is to have clear lines of communication with the IRB office and ensure that all data collection processes have been approved.

Suggestion 7:Suggestion 7:

 Take the time to step back and ask what evaluation results mean or imply, and whether others may find results useful

– Consider allocating some project team meeting to reviewing evaluation results, discussing their meaning, interpreting them in light of broader literature, and considering if the emerging findings may be useful beyond your institution.

Example: Example: Evaluation Leading to ResearchEvaluation Leading to Research

Internal and external evaluation work at an ADVANCE Internal and external evaluation work at an ADVANCE institution led to interesting questions about the institution led to interesting questions about the relationships between organizational characteristics and relationships between organizational characteristics and culture to interventions and strategies used and their impactculture to interventions and strategies used and their impact

These questions led to an ADVANCE IT project: These questions led to an ADVANCE IT project:

A. E. Austin & S. Laursen’s study concerning: A. E. Austin & S. Laursen’s study concerning: – Strategies and processes that have worked well in creating Strategies and processes that have worked well in creating

institutional environments conducive to the success of women institutional environments conducive to the success of women scholarsscholars

– Why and under what circumstance these interventions workedWhy and under what circumstance these interventions worked– How institutional culture and characteristics relate to choice and How institutional culture and characteristics relate to choice and

effectiveness of strategies effectiveness of strategies

Contact InformationContact Information

Ann E. AustinAnn E. Austin

Professor, Higher, Adult, and Lifelong EducationProfessor, Higher, Adult, and Lifelong Education

419A Erickson Hall419A Erickson Hall

620 Farm Lane620 Farm Lane

Michigan State UniversityMichigan State University

East Lansing, MI 48824East Lansing, MI 48824

Tel: 517-355-6757Tel: 517-355-6757

Email: aaustin@msu.eduEmail: aaustin@msu.edu