View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Institutional Researcher’s Credo
I realize that I will not succeed in answering all of your questions. Indeed, I will not answer any of them completely. The answers I provide will only serve to raise a whole new set of questions that lead to more problems, some of which you weren’t aware of in the first place. When my work is complete, you will be as confused as ever, but hopefully, you will be confused on a higher level and about more important things.
Developing & Sustaining a Culture of Institutional Research
Victor M. H. Borden, Ph.D.Professor of Educational Leadership and Policy Studies
Indiana University Bloomington
What’s Your Problem?
Select one current institutional issue that requires attention• What do you need to know to be better informed
to pursue that issue?
• What do you need to know to decide what course of action to take?
• How will you be able to tell if that action is working?
What is IR?
…research conducted within an institution of higher education to provide information which supports institutional planning, policy formation and decision making.• Saupe, 1990 – The functions of IR
What Does an IRer Do?
The institutional researcher serves higher education and, in turn, his institution through critical appraisal and careful investigation of its processes and programs.• Suslow, 1972– A Declaration on Institutional
Research
Who Does IR?
IR Staff
Anyone who gathers, evaluates, and interprets ‘data’ for use in decision making• Senior administrators and their senior staff
• Academic department chairs and program directors
• Administrative directors and managers
• Faculty working on committees and in administrative support roles
• Administrative assistants
Variations on U.S. IR Practice
From 1 to 25 person offices
Accountability
Enrollment management
Data administration and warehousing
Learning Assessment support
Survey research
Marketing and development
Planning and policy analysis
Other duties as assigned
An IR Office Case Study
IUPUI’s Office of Information Management and Institutional Research (IMIR)
IMIR mission
The mission of the Office of Information Management and Institutional Research (IMIR) is to provide and coordinate information support for planning, administering, and evaluating academic and administrative programs in ways that will continuously improve IUPUI.
IMIR Services and Functions
Management Reports and Analyses• developing for academic deans and other campus
administrators a series of management reports and analyses that integrate information from a variety of institutional and external data sources;
Ad Hoc Requests• providing academic and administrative managers with
information needed to address ad hoc problems and issues;
Information Management• creating organized, documented, and accessible data
resources based on institutional, survey, and external databases;
IMIR Services and Functions
Survey Research• conducting survey research to assess the expectations,
satisfaction, and outcomes of students, alumni, faculty, staff, employers, and other stakeholders;
Planning and Program Evaluation Support• providing direct support to specific campus, school, and
program evaluation and planning activities;
Information Technology Development• developing computer network-based systems for collecting,
accessing, and analyzing information in a more timely and cost-effective manner; and
Consulting• helping staff from other academic and administrative units to
conduct institutional research, reporting, and analysis
Principles of Effective IR (old school)
Rational, data-driven decision making processes yield more informed and successful decisions
Improved access to well-managed data sources provides decision makers with the tools they need to inform decision processes
To be useful, data must be turned into information, and information into knowledge
Data, information, and knowledge can all be managed The primary function of research and information
support is to provide information needed by decision-makers and decision-making groups
Heresy
Being “data-driven” is not, in and of itself, a good thing– e.g., “selective use” of evidence to support a
foregone conclusion Torture numbers long enough and they’ll confess to
anything
Effective use of data requires sharing diverse and often divergent perspectives– It’s not what the data say, it’s what you say about
the data– Some disagreement and dissent is important to
learning and innovation
Further Heresy
Building effective programs requires some level of irrationality and disorder• To learn from what we do requires that we
unlearn some things that we often don’t want to unlearn
As if doing this by ourselves were not difficult enough, we must do this together• Organizational learning requires communities
of practice
Beyond Data-Driven
Data-driven implies…– Rational, systematic testing of ideas through
inspection of facts– sequential, often individual decision-making
processLearning-driven implies…
– Going beyond what we already know and can do to gain new competencies
– Deconstruction and reconstruction of ideas and beliefs
– Becoming irrational to become re-rational
Beyond Data-Driven
We are engaged in the learning process as interdependent members of an organization (i.e., system)
Through collaborative, action-oriented, organizational learning, we learn (together) how to achieve organizational goals
Learning Organizations
…are organizations where people continually expand their capacity to create the results they truly desire, where new and expansive patterns of thinking are nurtured, where collective aspiration is set free, and where people are continually learning to see the whole together. (Senge, 1990)
Example: Learning Communities
Problem: Low retention rates at urban, commuter university associated with under-prepared new students, low grades, and non-existent campus life
Summer Retreat Involving Administrators, Faculty, Advisors, IR Staff– How do we increase engagement through the
classroom?– Constraint: can’t “block-schedule” working commuters
Common first-year seminar for all beginners– 100+ sections of 25 students, each with faculty member,
advisor, student mentor and librarian/technologist– Common outcomes template
E.G., LCs (2)
Bottom line evaluation shows modest impact• Improved retention and grades among “treated students”• Overall retention rate goes south
Further (qualitative) analyses reveal problems with “one-size-fits-all” approach• Focus groups; interviews; faculty fellows
Adjustments• Agreement with local CC to defer some students• School-based models to appeal to different interests• Companion supports: summer bridge, critical inquiry, structured
learning assistance, math assistance center• Block and thematic learning communities
E.G., LCs (3)
Improved Results• 10 percentage point increase in retention
over four years!• No improvement for two years!!
New problems—New approaches• Existing bridge programs adjusted to weed
out students who aren’t likely to succeedFurther increases in retention!!
The Collaborative Action-Inquiry Approach
An organizational learning framework for evidence-based practice
Action research and beyondFurther examples
Herbert Simon’s Challenge
What information consumes is rather obvious: it consumes the attention of its recipients. Hence, a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it. ~(Simon, 1971)
Kahneman & Tversky’s Challenge
Cognitive biases• We seek information based on what we know
and want to know• Biases and defense mechanisms prevent us
from processing information objectivelySelf-serving biasPrimacy and Recency
Challenges (cont.)
Decisions made within a social-organizational-political context• Rules and roles define/constrain process• Social and organizational relationships must
be accommodated
A Learning Paradigm
Planning, evaluation, and improvement all fit within a rational model
Learning incorporates uncertainty, ambiguity, and multiple styles
Individual learning and organizational learning are compatible concepts
Learning is mission critical for all educational institutions
Single- and Double-Loop Learning
Learning is the detection and correction of error (unintended consequences)
“Governing Variables” are those things what we feel are important to keep within acceptable limits
“Action Strategy” is what we do or plan to do to keep the governing variables within limits
“Consequences” are the intended and unintended outputs and outcomes• Intended: confirm our theory in use• Unintended: suggests error in our theory in use
Single-Loop Learning
Governing variables not called into question
Adjustments made to action strategies at best
Defense mechanisms can readily arise to maintain single-loop learning
GoverningVariables
ActionStrategies
Conse-quences
Double-Loop Learning
Questioning the role of the framing and learning systems which underlie actual goals and strategies
Reflection is fundamental• Basic assumptions are confronted• Hypotheses publicly tested• Falsification is sought• Ego is laid aside
GoverningVariables
ActionStrategies
Conse-quences
Model I and II Org Learning
Single- and double-loop learning at the organizational level
Model I: Organizational members prescribe to a common theory in use• Organizational policies and practices inhibit
changeModel II: Governing values, policies, and
practices promote double-loop learning
A Model I Learning Organization Governing Variables
– Tow the line– Win at all costs– Suppress negative feelings– Emphasize rationality
Action Strategies– Control environment and task unilaterally– Protect self and others unilaterally– Discourage inquiry
Consequences– Defensive relationships– Low freedom of choice– Reduced production of valid information– Little public testing of ideas
A Model II Learning Organization
Governing Variables– Valid information is most important– Free and informed choice– Shared internal commitment
Action Strategies– Shared control– Participation in design and implementation of action
Consequences– Minimally defensive relationships– High freedom of choice– Public testing of ideas
Jerome Bruner / Gilbert Ryle
• Learning about
vs.
• Learning to be
• Knowing that
vs.
• Knowing how
Learning To Be / Know How
Based on collaborative practice• Communities of practice
Knowledge as inseparable from the knowerEvidence as inputSharing interpretations as processCommon priorities and strategies as output
What’s important is not what the data say,
It’s what we say and do about the data
Action Research
Continuous cycle of data collection data analysis data feedback action plans data collection
Stakeholder empowerment through active and on-going participation
Data feedback meetings promote collaboration, dialogue, and collective analysis
Active learning and discovery fostered by critical reflection process
Data-driven action plans developed = research linked to action
Action Research
1. Research Question/Evaluation Focus
2. Data Collection
3. Analysis and Interpretation
4. Presentation and Reflection
5. Action Plan
6. Evaluation of impact (Go to 2)
7. Focus on new issue (Go to 1)
Linking Research and Action
Who does what?• Decides what actions are taken?• Is responsible for effective implementation?• Can devise appropriate evaluation protocols?• Has access to or can collect appropriate
evidence?• Reviews the results and decides what to do?
What can be done to get these people to work together and in concert?
Determining the Research Question
Traditional IR• Given to researcher
– Top-down directive
– Bottom-up request• Clarification of
request– Discussion of
context and use
Action Research• Developed together
– Requester or researcher
• Specific questions often deferred until vested parties brought together
Data Collection
Traditional IR• Researcher finds
and collects data• Researcher
accountable for integrity of information
Action Research• Stakeholders have
role– Collecting data– Learning about
nuances• Shared responsibility
for integrity
Data Analysis and Interpretation
Traditional IR• Researchers
responsible through dissemination
• May consult with stakeholders to gain insight into the results
Action Research• Stakeholders involved
in stages of data analysis
• Preliminary results presented and discussed– Further analyses
shaped by those discussions
Report Presentation and Dissemination
Traditional IR• Researcher
prepares and often presents results to stakeholders
Action Research• Presentation and
report writing responsibilities shared
• Presentations involve– active discussion – facilitation of action
plan development
Follow-up
Traditional IR• Some additional
analyses may be requested or perhaps some clarification
• Often the end of the process
Action Research• Stakeholders design
action plan based on results
• Data collection included in follow-up plan
• Further lines of inquiry established for next cycle of research
Example: Evaluation of New Student Orientation
Research Question and Evaluation Focus – reassessment of goals; incoming students’
needs; impacts on knowledge, attitudes, and behaviors
Data Collection– focus groups and questionnaires, sought
perspectives of all major stakeholdersData Reporting and Feedback
– meetings with orientation leaders and faculty stakeholders
Example: Evaluation of New Student Orientation
Development of Action Plans• facilitation of dialogue and data-driven
proposalsAction
• implementation of proposed changesAssessment – on-going formative evaluation;
re-administration of process and outcome instruments
Example: Indiana Project on Academic Success (IPAS)
Research-based inquiry for enhancing academic success
Four-stage method• Assessment• Organizing• Action Inquiry• Evaluation
Supported by use of state and institutional student tracking records
Stage 1: Assessment
Compare campus assessment information to statewide assessment results; identify possible challenges
Collect additional information from campus sources, such as prior reports and studies and focus group interviews
Organize teams of administrators, faculty, professional staff, and students to identify critical challenges on the campus
Prioritize the challenges, identifying two or three that merit special attention at a campus level
Stage 2: Organizing
Coordinate the assessment and inquiry process with campus-level planning and budgeting; integrate the challenges with strategic plans; coordinate budgeting to provide necessary support.
Appoint workgroups to address critical, campus-wide challenges; consider providing release time to team leaders to work on tasks for the campus.
Coordinate the inquiry process (activities of the workgroups) with campus planning and budgeting.
Stage 3: Action Inquiry
Build an Understanding of the Challenge– What solutions have been tried in the past, and how well
did they work? What aspects of the challenge have not been adequately addressed? What aspects require more study? Develop hypotheses about the causes for the challenges using data to test the hypotheses. Do the explanations hold up to the evidence?
Look Internally and Externally for Solutions– Talk with people on campus about how they have
addressed related challenges. Consider best practices for retention and how they might be adapted to meet local needs. Visit other campuses that have tried out different approaches to the problem. How well would these alternatives address the challenge at your campus?
Stage 3: Action Inquiry
Assess Possible Solutions• Consider alternatives in relation to the
understanding of the problem developed in Stage 3, step 1. Will the solutions address the challenge at your campus? How can the solution be pilot tested? If you tried out the solution, how would you know if it worked? What information would you need to know how well it worked?
Stage 3: Action Inquiry (cont.)
Develop Action Plans– Action plans should address the implementation of
solutions that should be pilot tested. Consider solutions that can be implemented by current staff. If there are additional costs, develop budgets for consideration internally and externally. (Remember, seeking additional funds can slow down the change process.) Develop action plans with time frames for implementation and evaluation
Implement Pilot Test and Evaluate– Provide feedback to workgroups and campus coordinating
team. Use evaluation results to refine the solution. Also, evaluation can be used as a basis for seeking additional funding from internal and external sources, if needed
Institutional Support for Organizational Learning
How do we organize/staff to support the “culture of institutional research?”
There are many kinds of supports possible• Institutional research and analysis• Measurement and instrumentation• Program review and evaluation• Quality enhancement• Performance management
Planning for Improvement
Planning & Budgeting 1. Mission, Vision, Goals developed 2. Unit goals aligned 3. Annual reports on web 4. Programs based on assessable goals,
with performance indicators 5. Biennial planning/budgeting hearings
conducted
Implementation
(Everyone on campus implements goals)
Evaluation
1. Academic and administrative program reviews
2. Evaluation of process effectiveness 3. Assessment of learning outcomes
in major in general education
4. Course evaluations 5. Student assessment 6. Constituent surveys 7. Management information and analysis 8. Program cost analysis 9. Web-based evaluation tools 10. Annual campus performance report 11. NCA accreditation
Assessable
Outcomes
Inst
rum
enta
tion
Data Collection
Analysis
App
licat
ion
of F
indi
ngs Culture
OfEvidence
Improvement 1. Reporting to internal constituents
2. Demonstrating accountability to external stakeholders
3. Applying findings in campus improvement initiatives
4. Proposing improvement initiatives
5. Improving assessment methods · Web-based
data
· Electronic portfolios
Planning for Improvement
Planning & Budgeting 1. Mission, Vision, Goals developed 2. Unit goals aligned 3. Annual reports on web 4. Programs based on assessable goals,
with performance indicators 5. Biennial planning/budgeting hearings
conducted
Planning for Improvement
Evaluation1. Academic and administrative program reviews2. Evaluation of process effectiveness3. Assessment of learning outcomes
• In major• In general education
4. Course evaluations5. Student assessment6. Constituent surveys7. Management information and analysis8. Program cost analysis9. Web-based evaluation tools10.Annual campus performance report11.General and specialized accreditation
Planning for Improvement
Improvement1. Reporting to internal constituents2. Demonstrating accountability to
external stakeholders3. Applying findings in campus
improvement initiatives4. Proposing improvement initiatives5. Improving assessment methods
• Web-based data collection/reporting• Electronic portfolios
How Might You Do it?