Upload
nathaniel-mitchell-george
View
213
Download
1
Tags:
Embed Size (px)
Citation preview
Turning Data Into Action WorkshopJune 26, 2014
2 ©2014 U.S. Education Delivery Institute
Kentucky prioritized gap closing as a major focus for the upcoming academic year
Policy Objective 4: Increase high-quality degree production and completion rates and close achievement gaps.
(Associate degrees)Graduation rate gaps: Underrepresented MinoritiesGap between the graduation rate of African-American, Latino, American Indian and Hawaiian/Pacific Islander students and the graduation rate of White and Asian students .Data Source: CPE Comprehensive Database (KPEDS)
3
2009-10 2010-11 2011-12 2012-130.0%
2.0%
4.0%
6.0%
8.0%
10.0%
12.0%
14.0%
16.0%
7.2% 7.3%7.7%
6.8%
13.0%14.1% 13.7% 13.5%
Underrepresented Minority
Not Underrep-resented Minor-ity
Policy Objective 4: Increase high-quality degree production and completion rates and close achievement gaps.
(Associate degrees)Graduation rate gaps: UnderpreparedGap between the graduation rate of students who did not meet statewide collegeReadiness benchmarks and those who did.Data Source: CPE Comprehensive Database (KPEDS)
4
2009-10 2010-11 2011-12 2012-130.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
7.5%
11.3%8.9% 8.6%
22.8%
26.3%23.7% 24.4%
UnderpreparedPrepared
Policy Objective 4: Increase high-quality degree production and completion rates and close achievement gaps.
(Associate degrees)Graduation rate gaps: Low Income Gap between the graduation rate of Pell grant recipients and non-recipients .Data Source: CPE Comprehensive Database (KPEDS)
5
2009-10 2010-11 2011-12 2012-130.0%
2.0%
4.0%
6.0%
8.0%
10.0%
12.0%
14.0%
16.0%
18.0%
20.0%
10.8% 10.9% 11.7%10.4%
13.9%
16.7%15.5%
18.2%
Low IncomeNot Low Income
Policy Objective 4: Increase high-quality degree production and completion rates and close achievement gaps.
(Bachelor degrees)Graduation rate gaps: Underrepresented MinoritiesGap between the graduation rate of African-American, Latino, American Indian and Hawaiian/Pacific Islander students and the graduation rate of White and Asian students (using the IPEDS six-year graduation rate). Data Source: CPE Comprehensive Database (KPEDS)
6
2009-10 2010-11 2011-12 2012-130.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
36.9% 36.0% 33.6% 33.3%
49.0% 49.7% 49.4% 51.0%
Underrepresented Minority
Non Underrep-resented Minor-ity
Policy Objective 4: Increase high-quality degree production and completion rates and close achievement gaps.
(Bachelor degrees)Graduation rate gaps: UnderpreparedGap between the graduation rate of students who did not meet statewide collegeReadiness benchmarks and those who did (using the IPEDS six-year graduation rate).Data Source: CPE Comprehensive Database (KPEDS)
7
2009-10 2010-11 2011-12 2012-130.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
28.0% 29.2% 28.6% 27.8%
54.7% 55.3% 54.5% 56.5%
UnderpreparedPrepared
Policy Objective 4: Increase high-quality degree production and completion rates and close achievement gaps.
(Bachelor degrees)Graduation rate gaps: Low Income Gap between the graduation rate of Pell grant recipients and non-recipients (using the IPEDS six-year graduation rate). Data Source: CPE Comprehensive Database (KPEDS)
8
2009-10 2010-11 2011-12 2012-130.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
46.2%
34.5% 34.5% 36.3%
56.7%52.9% 51.5% 52.0%
Low IncomeNot Low Income
9 ©2014 U.S. Education Delivery Institute
Today, we will cover…
Turning Data Into Action Workshop Agenda
9:00 Welcome & Meeting Overview
9:40 Understanding Our Capacity to Use Data to Make Decisions on Campus
10:45 Break
11:00 Understanding Current Progress and Identifying Institutional Factors
11:45 Lunch
12:30 Prioritizing and Focusing Our Efforts
1:30 Identifying Relevant Metrics
2:15 Break
2:30 Determining Who and How
3:30 Identifying Next Steps
4:00 Meeting Close
10 ©2014 U.S. Education Delivery Institute
The delivery approach produces results by focusing leaders on four fundamental questions
“delivery” (n.) is a systematic process through which leaders can drive progress and deliver results.
It involves asking the following questions consistently and rigorously:
1 What are we trying to do?
2 How are we planning to do it?
3 At any given moment, how will we know whether we are on track?
4 If not, what are we going to do about it?
11 ©2014 U.S. Education Delivery Institute
EDI supports implementation of education reform in a number of states and systems
Current EDI Partner SystemsK-12Higher EducationBoth
Our mission is to partner with K-12 and higher education systems with ambitious reform agendas and invest in their leaders' capacity to deliver results. By employing a proven approach, known as delivery, we help state leaders maintain the necessary focus to plan and drive reform.
12 ©2014 U.S. Education Delivery Institute
Our Higher Education Delivery Network consists of a growing number of systems across the country
EDI focuses on supporting systems in their implementation efforts to: Increase the percent of students who graduate with a degree
on time
Close equity gaps in postsecondary access
Close equity gaps in postsecondary success
Current Higher Education Partnerships
California State University System Colorado State University System University of Hawai’i System Kentucky Council on Postsecondary
Education Minnesota State Colleges and
Universities University of Missouri System
New Jersey Office of Higher Education Pennsylvania State System of Higher
Education Tennessee Board of Regents The University of Texas System University of Wisconsin System
13 ©2014 U.S. Education Delivery Institute
We will use this workshop to focus on using data as an action strategy to close gaps
Assess campus capacity to use data to drive decision making
Understand current progress on campus and in the Council, as it relates to achievement gaps
Discuss institutional factors contributing to outcomes
Refine key campus initiatives intended to improve outcomes
Identify metrics in order to monitor those initiatives
Identify who will be involved in monitoring and making mid-course corrections and how monitoring will occur
14 ©2014 U.S. Education Delivery Institute
What is our
current status?
Why are we in this
position?
What are we doing about it?
How are we going
to monitor progress
?
To meet our objectives, we will tackle the gap challenges by answering four questions
15 ©2014 U.S. Education Delivery Institute
There are simple steps you can take to make the most of the workshop!
Choose a team leader/facilitator
Suspend disbelief!
Ask yourself, “What can we do now, with what we have?”
Try not to get caught up in the minute details
Leverage the expertise at the table
Leverage the efforts underway at your campuses
LET’S GET STARTED!
16 ©2014 U.S. Education Delivery Institute
We believe that these behaviors are necessary for campuses to use data effectively in decision making
EDI’s tenets for data use
Campuses must:
Have clear targets against which to measure performance
Use timely data (both headline metrics and leading indicators) that are monitor progress on a regular basis
Disaggregate the data in order to isolate outcomes for specific groups of students
Establish routines that help teams come to a shared view of progress
Involve IR in relevant conversations and analyses to take advantage of existing expertise
Empower decision makers to seek and use relevant data to support students
17 ©2014 U.S. Education Delivery Institute
As we begin to plan how we’ll use data to make decisions, we need to be clear about some terms
Description
StrategyA coherent set of initiatives that are designed to maximize impact on your target metrics
InitiativeA specific change in the way a campus supports and educates its students
MetricAn outcome used to measure progress on an initiative, strategy, or both
Term
Depending on what your team is prepared to discuss, you can choose to either focus on one discrete initiative or on a strategy comprised of multiple initiatives during this workshop.
18 ©2014 U.S. Education Delivery Institute
Poll Everywhere: Data-informed Decisions and Your Campus
How well does your campus currently use data to drive/inform decision making?
We do it, and we do it well
We do it well in a few areas across campus
Analysis Paralysis – We have it, but we rarely move to action
“Who needs data? We go with our gut!”
Data? What data? This is a challenge area for our campus
19 ©2014 U.S. Education Delivery Institute
Workshop Agenda
Understanding Our Capacity to Use Data to Make Decisions
Followed by a break
20 ©2014 U.S. Education Delivery Institute
Our goal is to not only understand our status but to discuss why, and how we can ensure that we maximize our impact
Our graduation rate for African American students went up 4% this year!
Huh. Interesting.
Our graduation rate for African American students went up 4% this year!
That’s probably because the cohort of students that entered when we began the new multicultural office support program graduated this year.
Not sure though…we should find out how many students this program served and what the early indicators said.
Yep! Early indicators indicated that this program was improving retention and credits earned for African American students.
We should monitor and scale up that support program from now on, so we maximize the impact from this important strategy!
21 ©2014 U.S. Education Delivery Institute
What is our
current status?
Why are we in this
position?
What are we doing about it?
How are we going
to monitor progress
?
This workshop allows campuses to develop a plan to assess progress on key campus strategies by asking 4 questions
Teams will review their campus’ data to develop a shared understanding of student success
22 ©2014 U.S. Education Delivery Institute
What is our
current status?
Why are we in this
position?
What are we doing about it?
How are we going
to monitor progress
?
This workshop allows campuses to develop a plan to assess progress on key campus strategies by asking 4 questions
Teams will discuss the institutional factors that need to be addressed or leveraged to improve student performance
23 ©2014 U.S. Education Delivery Institute
What is our
current status?
Why are we in this
position?
What are we doing about it?
How are we going
to monitor progress
?
This workshop allows campuses to develop a plan to assess progress on key campus strategies by asking 4 questions
Teams will identify, clarify, and refine key campus strategies to address challenges
24 ©2014 U.S. Education Delivery Institute
What is our
current status?
Why are we in this
position?
What are we doing about it?
How are we going
to monitor progress
?
This workshop allows campuses to develop a plan to assess progress on key campus strategies by asking 4 questions
Teams will develop a routine to regularly assess progress on selected strategies
25 ©2014 U.S. Education Delivery Institute
What is our
current status?
Why are we in this
position?
What are we doing about it?
How are we going
to monitor progress
?
How much capacity do we have on campus to
answer these
questions?
Before we can answer these questions, however, we must reflect on our current capacity to use data to drive action
26 ©2014 U.S. Education Delivery Institute
EDI has created a rubric to help campuses think about their ability to use data to drive decisions
27 ©2014 U.S. Education Delivery Institute
The self-assessment tool helps campuses understand current capacity to answer these questions
Rubric: Capacity to Use Data for Decision Making on Campus
Rubric category This category asks us to think about…
Access and efficiency
Monitoring
Problem solving
The ease and speed with which data can be accessed or retrieved from campus systems
Culture
Actions taken to understand progress on campus goals and metrics
The processes we use to turn what we learn from campus data into action around campus strategies
The attitudes and reactions of campus faculty and staff toward making decisions using campus data
28 ©2014 U.S. Education Delivery Institute
Use the rubric to determine how you would rate your campus in each category
Category
Access and efficiency
Monitoring
Problem solving
Culture
Rating Rationale Summary
Capacity to use data to make
decision on campus
Red: Our campus must focus on improving in this area to use data effectively in decision making
Orange: Our campus must improve in this area, though in some aspects more than others
Yellow: Our campus should continue in the aspects we find strong, but focus on weak points
Green: Our campus is successfully using our capacity in this area to use data to make decisions
29 ©2014 U.S. Education Delivery Institute
Example: one campus realized that monitoring and problem solving needed improvement in order to use data effectively
Category
Access and efficiency
Monitoring
Problem solving
Culture
Rating Rationale Summary
Unified data system with IR support, but faculty and staff generally do not know how to make data requests or what questions to ask
Conversations around areas of improvement almost never focus on solutions; we do not have priority owners
Our strategic plan calls for regular monitoring, but meetings get cancelled; when they do happen, we don’t have helpful data available
We feel that faculty, staff, and campus leadership are generally confident in our data systems and want to use data to drive the work
Capacity to use data to make
decision on campus
Example of completed rubric with rationales
Red: Our campus must focus on improving in this area to use data effectively in decision making
Orange: Our campus must improve in this area, though in some aspects more than others
Yellow: Our campus should continue in the aspects we find strong, but focus on weak points
Green: Our campus is successfully using our capacity in this area to use data to make decisions
30 ©2014 U.S. Education Delivery Institute
Exercise: Capacity To Use Data on Campus Self-Assessment
▪Individually, read your rubric and consider what your “traffic light” rating would be in each of the four areas. Write your thoughts on the rubric.
▪As a team, discuss your ratings and determine where your campus is strongest, where it needs the most improvement, and why.
Questions to ask each other…What prevents us from being green?Do our ratings mean the same thing?Are we being too harsh? Too easy?
31 ©2014 U.S. Education Delivery Institute
Your Role in Strengthening Capacity
How can you leverage your role to strengthen your campus’ capacity to use data to make
decisions?
Record your answer on a card Share your card with your team members Place card on brown paper
32 ©2014 U.S. Education Delivery Institute
Workshop Agenda
Understanding Current Progress and Identifying Institutional Factors
Followed by Lunch
Break
33 ©2014 U.S. Education Delivery Institute
First, we will determine the level of challenge we face in serving our students and then discuss contributing factors
What is our
current status?
Why are we in this
position?
What are we doing about it?
How are we going
to monitor progress
?
34 ©2014 U.S. Education Delivery Institute
“Good data, while essential, is only a start—
you then have to use it!” – Instruction to Deliver
35 ©2014 U.S. Education Delivery Institute
There are two steps to understand the drivers of performance and evaluate relevant campus strategies
Identify the drivers of your
student success patterns
Identify and evaluate campus
strategies related to drivers of
performance patterns
▪ Pinpoint the root causes, or drivers of performance patterns
▪ Identify “usual suspects” when possible
▪ Utilize formal problem-solving tools when drivers are not evident
Description Why important
▪ A campus cannot know where to focus unless it identifies drivers
▪ Creating new or revamping initiatives without understanding existing strategies wastes resources
▪ Identify what the campus is currently doing to try and address drivers of underperformance
▪ Evaluate whether current initiatives are sufficient and recommend changes where necessary
1
2
36 ©2014 U.S. Education Delivery Institute
Traditionally, data show several factors that are likely to impact how groups of students progress and persist
Race & Ethnicity
Gender
Income or Pell Status
High School GPA
Academic Major
Courses Taken During First Year
Number of Credits Accrued at Various Points
Some of those factors include:
37 ©2014 U.S. Education Delivery Institute
Florida State University used data to understand student success and which students were lost along the way
Six-Year Graduation Rates 2002-2010, Underrepresented Minorities and White Students
2002 2003 2004 2005 2006 2007 2008 2009 201050%
55%
60%
65%
70%
75%
80%
60.0
71.5
64.5
74.1
Underrepresented Minority White
Why were some students successful while others were not?
What could Florida State do to make a difference for them?
Source: Florida State University
38 ©2014 U.S. Education Delivery Institute
In looking at attrition rates, the team considered a range of time points and student background characteristics
Yearly Attrition Rates by Cohort: White, Female, First-Time In-State Students
0.5%1.6%
0.9%3.3%
2.1%3.9%
6.7%9.8%
10.2%17.1%
MINMAX
1995-2005
Source: Florida State University
39 ©2014 U.S. Education Delivery Institute
Hispanic female Pell recipients exhibited very different attrition rate patterns, suggesting the need for continued support over time to avoid dropouts
Yearly Attrition Rates by Cohort: Hispanic, Female, Pell Recipient, First-Time, In-State
Students
6.3%17.6%
MINMAX
6.9%17.7%
-1.2%10.5%
0%4.1%
-1.4%3.9%
Source: Florida State University
1995-2005
40 ©2014 U.S. Education Delivery Institute
Black male Pell recipients exhibited different attrition rate patterns, suggesting the need for continued support over time to avoid dropouts
6.0%14.0%
MINMAX
3.3%15.3%
-0.9%13.6%
0%11.0%
0%11.1%
Source: Florida State University
Yearly Attrition Rates by Cohort: Black, Male, Pell Recipient, First-Time In-State
Students
1995-2005
41 ©2014 U.S. Education Delivery Institute
A number of other institutional factors also create unclear pathways to graduation
▪ Insufficient academic guidance/planning
▪ Policies to pose as barriers
▪ Misunderstanding degree requirements
▪ Inappropriate preparation/enrollment in prerequisite courses
▪ General education requirement confusion
▪ Course withdrawal or failure
▪ Switching majors
▪ Inability to enroll in necessary coursesat the appropriate time
42
Graduation Rate in Major by Introductory Course
Grade
The Difference Between Satisfying a Requirement and Being on Track
STRENGTHENING ADVISEMENT
43 ©2014 U.S. Education Delivery Institute
Let’s look at Kentucky Campuses Success Gap data to help review your campus’s student success patterns
Are our gaps shrinking? Growing?
What do we know about closing gaps in higher education attainment?
Colorado State University System | Kentucky Council of Postsecondary Education | Louisiana Board of Regents | Minnesota State Colleges and Universities | Mississippi Institutions of Higher Learning | Montana University System | New Jersey Higher Education | Pennsylvania State System of Higher Education | Rhode Island Board of Governors for Higher Education | State University of New York | State University System of Florida | Tennessee Board of Regents | The California State University System | University of Hawaii System | University of Louisiana System | University of Missouri System | University of North Carolina System | University of Texas System | University of Wisconsin System |
19 Systems, 300 Campuses, 3.5 Million Students18% of Public Undergrads | 866K URMs | 1 million Pell Recipients
44 ©2014 U.S. Education Delivery Institute
Include all students, not just first-time, full-time
Include part-time and transfer students, many of whom are low-income and underrepresented minorities
Report retention and success rates for low-income and transfer students at institution and in system
A2S Metrics Count All Students
45 ©2014 U.S. Education Delivery Institute
Are we increasing success rates for underrepresented students?
GAP: Are we increasing success rates for underrepresented students relative to peers?
Benchmarking Progress on Success
46 ©2014 U.S. Education Delivery Institute
The data handouts answer
these and other questions
about student success on
your campus.
Benchmarking Progress on Success
47 ©2014 U.S. Education Delivery Institute
What is the overall freshmen rate?
What is the rate for transfer students?
How have these rates changed over time?
Are we increasing overall graduation rates?
Change in the Freshmen graduation rate since 2005 + 12 pts + 11 ptsChange in the Transfer graduation rate since 2005
State University
Bachelor's Programs
SUCCESS: IS THE CAMPUS IMPROVING THE RATE AT WHICH UNDERREPRESENTED STUDENTS COMPLETE?
OVERALL GRADUATION RATES
Freshmen graduation rate in 2012 59% 77%Transfer graduation rate in 2012
48 ©2014 U.S. Education Delivery Institute
Non-Pell graduation rate in 2012 (cohort)
60% (2,840)
Change in the Non-Pell graduation rate since 2005
+ 10 pts
Gap between Pell and Non-Pell graduation rates
2 ptsChange in Pell/Non-Pell graduation rate gap since 2005
Narrowed 5 pts
LOW-INCOME FRESHMEN
Pell graduation rate in 2012 (cohort)
58% (1,472)
Change in the Pell graduation rate since 2005
+ 15 pts
43% 44%
49% 50%
55%58%50%
53% 54% 56% 57%60%
35%
45%
55%
65%
1999/05 2002/08 2003/09 2004/10 2005/11 2006/12Pell Rate Non-Pell Rate
Are we increasing graduation rates for underrepresented students?
What is the grad rate of Pell freshmen?
What is the grad rate of non-Pell freshmen?
How do rates for these student groups differ?
49 ©2014 U.S. Education Delivery Institute
Non-Pell graduation rate in 2012 (cohort)
60% (2,840)
Change in the Non-Pell graduation rate since 2005
+ 10 pts
Gap between Pell and Non-Pell graduation rates
2 ptsChange in Pell/Non-Pell graduation rate gap since 2005
Narrowed 5 pts
LOW-INCOME FRESHMEN
Pell graduation rate in 2012 (cohort)
55% (1,472)
Change in the Pell graduation rate since 2005
+ 15 pts
43% 44%
49% 50%
55%58%50%
53% 54% 56% 57%60%
35%
45%
55%
65%
1999/05 2002/08 2003/09 2004/10 2005/11 2006/12Pell Rate Non-Pell Rate
How have graduation rates for Pell and non-Pell freshmen changed over time?
Are we increasing graduation rates for underrepresented students?
50 ©2014 U.S. Education Delivery Institute
Non-Pell graduation rate in 2012 (cohort)
60% (2,840)
Change in the Non-Pell graduation rate since 2005
+ 10 pts
Gap between Pell and Non-Pell graduation rates
2 ptsChange in Pell/Non-Pell graduation rate gap since 2005
Narrowed 5 pts
LOW-INCOME FRESHMEN
Pell graduation rate in 2012 (cohort)
55% (1,472)
Change in the Pell graduation rate since 2005
+ 15 pts
43% 44%
49% 50%
55%58%50%
53% 54% 56% 57%60%
35%
45%
55%
65%
1999/05 2002/08 2003/09 2004/10 2005/11 2006/12Pell Rate Non-Pell Rate
How has the Pell freshmen rate changed over time?
How has the non-Pell freshmen rate changed?
Has the graduation rate gap between these groups
narrowed or widened over time?
Are we increasing graduation rates for underrepresented students?
51 ©2014 U.S. Education Delivery Institute
Discussion: What do your data tell you?
Review your campus’s progress using the Access to Success report card
We will then discuss the following:
Where are you making the most progress?
Where is the largest improvement needed?
What other data do you have and/or use to understand progress?
After reviewing your data, answer:
What institutional factors contribute to your current status?
52 ©2014 U.S. Education Delivery Institute
Workshop Agenda
Prioritizing and Focusing Our Efforts
Lunch
53 ©2014 U.S. Education Delivery Institute
We will determine which strategies are most likely to improve outcomes for our students
What is our
current status?
Why are we in this
position?
What are we doing about it?
How are we going
to monitor progress
?
54 ©2014 U.S. Education Delivery Institute
In order to better serve your student group, teams should develop a concrete action plan
“A goal without a plan is just a wish.”
-Anonymous
55 ©2014 U.S. Education Delivery Institute
Thinking through a strategy profile helps define the strategy and expectations about its impact
Description of strategy
What is the strategy? What initiatives make it up?
Definition of success
If the strategy were successful, how will we know?
Theory of action
Why and how do we believe that the strategy will help us to achieve the desired outcome? Express as one or more “If… then…” statements.
RisksBased in the delivery chain, what are the potential risks associated with this strategy? How will you mitigate them?
MilestonesWhen will the strategy begin and when will it end, or become “business as usual?”
Leading indicators/ feedback loops
What interim metrics can you use to measure progress? How will you know that the actors along the delivery chain are doing the things they need to do?
Scale and efficacy
Scale: how many students will be affected by the strategy? Efficacy: of those affected, how many new students will achieve the outcome we desire?
56 ©2014 U.S. Education Delivery Institute
We have already begun to address some of these questions today
Description of strategy
What is the strategy? What initiatives make it up?
Definition of success
If the strategy were successful, how will we know?
Theory of action
Why and how do we believe that the strategy will help us to achieve the desired outcome? Express as one or more “If… then…” statements.
RisksBased in the delivery chain, what are the potential risks associated with this strategy? How will you mitigate them?
MilestonesWhen will the strategy begin and when will it end, or become “business as usual?”
Leading indicators/ feedback loops
What interim metrics can you use to measure progress? How will you know that the actors along the delivery chain are doing the things they need to do?
Scale and efficacy
Scale: how many students will be affected by the strategy? Efficacy: of those affected, how many new students will achieve the outcome we desire?
The gap we intend to close and the students affected.
Focusing on key institutional factors will help close the gap.
57 ©2014 U.S. Education Delivery Institute
We can narrow our focus from all initiatives to those that focus on the right factors affecting our students
Narrow the list to only the relevant initiatives
All campus
initiatives
Those that target the
right outcome
“We have a lot going on at our campuses!”
Graduation rates for underprepared students
Underprepared students are more likely to need support in registering for class and applying for financial aid
Those that target the
right factors
affecting our
students
Priority strategy
58 ©2014 U.S. Education Delivery Institute
Once we understand the initiatives that comprise our strategy, we can clearly define success
59 ©2014 U.S. Education Delivery Institute
For the next several exercises, we will use the strategy profile worksheet
60 ©2014 U.S. Education Delivery Institute
Exercise: Defining Success
Describe your strategy with the objective/gap and target metric that you will focus on
− Example: This strategy will improve our success rates for our first-generation students by providing more individual support and expanding current programs that disproportionally affect this group of students.
Identify the initiatives included in your strategy and the students that you are counting on these initiatives to serve
Define what success looks like for each initiative by Spring 2015
− Example: We want this initiative to be implemented in all Natural Sciences departments by Fall 2015 and to see an increase in the retention of first-generation students in those departments by Spring 2015.
Identify which institutional factors these initiatives address
− Example: Our campus currently does not offer enough guidance to students who are unsure of the most important decisions required to plan for and earn an bachelor’s degree.
61 ©2014 U.S. Education Delivery Institute
Workshop Agenda
Identifying Relevant Metrics
Followed by a break
62 ©2014 U.S. Education Delivery Institute
Once we know our strategy, we can consider how we expect to impact our students and when
“Ultimately, it is up to your team and those responsible for implementing the intervention to make the best judgment they can—informed by as many facts as possible and backed by sound logic—of how specific interventions will work together to affect your target metric.”
—Taken from Deliverology 101
63 ©2014 U.S. Education Delivery Institute
Determining expected impact is essential for monitoring progress
Consider the planned progression of your performance indicator from your current levels to your goal over time. We want to create targets that will we achieve en route to achieving the goal.
2010 2012 2014 2016 2018 2020 2022 2024 2026 2028 2030
Goal!
Stu
den
t O
utc
om
es
64 ©2014 U.S. Education Delivery Institute
Considering impact and setting targets serves a number of important purposes
Purpose of estimating expected impact
To monitor and predict performance rather than react
Provide early identification of problems to enable corrective action
Allow leadership to see impact of initiatives on an ongoing basis and learn from past performance
To give confidence that planned initiatives are sufficient to reach the target
65 ©2014 U.S. Education Delivery Institute
To assess impact, we can estimate the timing, scale, and efficacy for our strategy
Scale of impact▪ Estimate the number of students who will
be reached by the strategy.
Efficacy of impact▪ Assess, for each 100 students who are
impacted by the strategy, the number who will achieve our intended outcome who would not have before.
Scale x Efficacy = IMPACT
Timing of impact▪ Consider when the initiatives within the
strategy will begin to have impact and the duration of that impact.
Assess your priority strategy on timing, scale, and efficacy
66 ©2014 U.S. Education Delivery Institute
Here are some questions to consider when discussing impact and timing of the strategy
▪ When will the strategy be scaled up and when will it be as effective as we plan?
▪ Do we expect our strategy to contribute the same impact towards our goal each year?
▪ Will the implementation of our strategy improve over time? Until when?
Impact
Timing
▪ Does our strategy reach enough students to contribute substantially to our goal?
▪ Will we be able to reach our goal given the expected impact of our strategy (once we factor other initiatives into account)?
67 ©2014 U.S. Education Delivery Institute
Exercise: Determining Impact
Answer the following for your strategy:
▪Scale: Determine the number of students who will be reached by your strategy by Spring 2015
▪Efficacy: Estimate the percent of students targeted by the strategy who will contribute to your target metric
▪Impact: What is the resulting estimated impact and when will it occur?
68 ©2014 U.S. Education Delivery Institute
We will use intermediate metrics and feedback loops to monitor our priority initiatives
What is our
current status?
Why are we in this
position?
What are we doing about it?
How are we going
to monitor progress
?
69 ©2014 U.S. Education Delivery Institute
We know what we want to improve, but how will we know if our high impact strategies are making a difference?
Questions to consider for student-related metrics:
▪What data do our initiatives impact most directly?
▪How do we typically predict movement in our target metric?
▪How long will it be until we expect to see movement in each of the metrics we have identified?
Questions to consider for process-related metrics:
▪How many people or departments will need to change their practice? Which ones?
▪How will we know whether business as usual has changed?
▪What resources will these people need in order to implement these important campus initiatives with fidelity?
70 ©2014 U.S. Education Delivery Institute
Example Leading Indicators for College Completion
Course Participation: Begin remediation in first term Complete remediation in first year Complete college-level math or
English in the first year Complete a college success course
Course Performance: High rate of course completion
(80%) Complete 20-30 credits in first
year
Student Enrollment Patterns: Earn summer credits Enroll full-time Enroll continuously, without stop-
outs On-time registration for courses
What percentage of students reach
each of the leading indicators?
What is the impact of reaching each of
the leading indicators on
success rates?
Campuses can use leading indicator analyses to dig into their student data to help determine actions
71 ©2014 U.S. Education Delivery Institute
Florida State evaluated its strategies against important progress metrics
Impact of Florida State’s Success Coaching Program
Program that provides support for students around these 7 “soft” factors that influence retention and graduation:
Commitment to graduation
Managing commitments Finances School community Academics Effectiveness Health & Support
Program Description Overall Impact
Source: Florida State University
72 ©2014 U.S. Education Delivery Institute
In addition to looking at traditional progress metrics, a team can track key actions to determine progress
Example of a campus’ action plan
Month Timeframe
Action Responsibility
January By end of January
Emails to students with 35 attempted hours who have not been accepted into a major
Individual Responsible
January Ongoing Update department Degree Audit reports Individual Responsible
January Ongoing Individual contact with students who have been placed on probation
Academic Section
January Ongoing Individual contact with students who have been placed on warning
Academic Section
February 1st week Offer Workshop: Students Taking Exploratory Paths to Success
Advising First
February 1st week in the month
Email to all F coded students w/100+ hours inquiring about graduation plans; email to all H coded students w/100+ hours inquiring about finishing/graduation plans
Individual Responsible
February 6th week of term
New transfer—How are you doing— deadlines
Individual Responsible
73 ©2014 U.S. Education Delivery Institute
We can also use a delivery chain to find important metrics and feedback loops
A delivery chain is the set of actors (people or organizations), and the relationships between them, through which a given system activity will be implemented.
A delivery chain has one question at its core: Starting from the policy intent of a leader in your system and ending with the front-line behaviors and practices that this policy is designed to influence, how – and through whom – does a system activity actually happen?
74 ©2014 U.S. Education Delivery Institute
Example: Using a delivery chain to find metrics and feedback loops for mandatory first-year remediation
Provost Faculty Senate
Enrollment Managemen
t Office
Student Affairs
Academic AdvisorsRegistrar
Information Technology
Entering Students
Continuing Students
Orientation Advisors
Provost’s Office
Faculty Affairs
Vice Provost
StudentsAcademic Departments
Inform
Info
rm
Info
rm t
o
up
date
IT
syst
em
Inform and train
Create policy Inform
and advise
Info
rm
Example metrics/feedback loops
▪Number of students requiring remediation
▪Average time to complete remediation▪Remedial course passing rates
▪Number of remedial courses offered▪Number of advisors trained in new
policy▪Number of project benchmarks hit
75 ©2014 U.S. Education Delivery Institute
Exercise: Identifying Relevant Metrics
As a team, discuss which leading indicators and feedback loops you will use to track progress.
Answer the following: What are the most important leading indicators or
feedback loops to track (related to outcomes and to process)?
Based on these data, how will we know if we need to take action?
When we do, what action will we take?
How frequently will these data be available?
What do we need to do to make these data available to us?
76 ©2014 U.S. Education Delivery Institute
Record your discussion on leading indicators and feedback loops using your strategy profile worksheet
77 ©2014 U.S. Education Delivery Institute
Workshop Agenda
Determining Who and How
Break
78 ©2014 U.S. Education Delivery Institute
Finally, we will determine a routine for monitoring the impact of our key initiatives on our key metric
What is our
current status?
Why are we in this
position?
What are we doing about it?
How are we going
to monitor progress
?
79 ©2014 U.S. Education Delivery Institute
One of the main benefits of routines is their ability to maintain focus despite distractions and barriers
“Fires” and other crises!
Changes in Leadership!
Institutional inertia!
Initiatives lacking robustness!
Cynicism!IMPLEMENT TO
IMPROVE STUDENT OUTCOMES
Shifting agendas!
ROUTINES
80 ©2014 U.S. Education Delivery Institute
Routines provide us with checkpoints where we can compare current progress with what we expected
▪ Regularly scheduled checkpoints to assess if implementation is on track
▪ A source of structure and discipline to keep the core goals on the agenda
▪ Structure to have honest and helpful conversations about where the work is and why and what’s next
What are routines?
▪ Monitor performance: Understand if campus is on track to deliver on its goals
▪ Diagnose problems: Surface issues that are inhibiting progress and analyze data to pinpoint causes
▪ Address problems: Provide a venue to discuss and decide how to overcome challenges
What purpose do routines serve?
81 ©2014 U.S. Education Delivery Institute
Effective routines share similar characteristics
Criteria for effective routines
Agree on a common purpose
Arrive at a shared view of performance and progress
Identify and solve problems
Encourage learning and collaboration
Identify and commit to clear next
steps
82 ©2014 U.S. Education Delivery Institute
The following characteristics should be present in any good routine
▪The right people– The person receiving the information (usually
president, provost or other campus leader)– The person responsible/accountable– Other key stakeholders
▪Relevant outcome-focused data– Leading indicators– Results showing progress on trajectories– Judgments on the likelihood of delivery
▪A discussion of challenges
▪Clear deliverables/next steps
▪Crisp, concise presentation of information
▪The right frequency and timing
▪The right format for those involved– Written note– Stocktake or in-person meeting– Formal report
Characteristics of a good routine:
83 ©2014 U.S. Education Delivery Institute
A “textbook” campus routine calls for five roles
Roles and responsibilities: Graduation Rates Delivery Plan and Stocktake
Provost“The Chief”▪Holds others accountable for results▪Asks tough questions that challenge and
support▪Actively engages in problem-solving
Team LeaderBroker▪Prepares Commissioner for meeting▪Designs agenda, keeps meeting on track
Team MemberAnalyst▪Prepares data and evaluations▪Works with goal team to prepare
Director of Academic AdvisingSponsor▪Holds overall accountability for the plan’s
success▪Answers the Commissioner’s questions and
actively solves problems
VP for Academic AffairsWorkhorse▪Holds day-to-day accountability for the plan’s
success▪Manages the team to implement the plan▪Works with delivery unit and provides evidence
to arrive at current assessment of progress, in consultation with sponsor
84 ©2014 U.S. Education Delivery Institute
When preparing for a routine, invest time to make relevant data easy to display and understand
Example: Enrollment of first time, full time students by semester and cohort
Fall 2003 Fall 2004 Fall 2005 Fall 2006
1 Fall1 Spring
2 Fall2 Spring
3 Fall3 Spring
4 Fall4 Spring
5 Fall5 Spring
6 Fall6 Spring
1 Fall1 Spring
2 Fall2 Spring
3 Fall3 Spring
4 Fall4 Spring
5 Fall5 Spring
6 Fall6 Spring
1 Fall1 Spring
2 Fall2 Spring
3 Fall3 Spring
4 Fall4 Spring
5 Fall5 Spring
6 Fall6 Spring
1 Fall1 Spring
2 Fall2 Spring
3 Fall3 Spring
4 Fall4 Spring
5 Fall5 Spring
6 Fall6 Spring
0
50
100
Percent of cohort enrolled
11.7
13.3 18.4
6.2
13.0
12.2
21.9
8.4
Pell and Non-Pell
Fall 2003 Fall 2004 Fall 2005 Fall 2006
1 Fall1 Spring
2 Fall2 Spring
3 Fall3 Spring
4 Fall4 Spring
5 Fall5 Spring
6 Fall6 Spring
1 Fall1 Spring
2 Fall2 Spring
3 Fall3 Spring
4 Fall4 Spring
5 Fall5 Spring
6 Fall6 Spring
1 Fall1 Spring
2 Fall2 Spring
3 Fall3 Spring
4 Fall4 Spring
5 Fall5 Spring
6 Fall6 Spring
1 Fall1 Spring
2 Fall2 Spring
3 Fall3 Spring
4 Fall4 Spring
5 Fall5 Spring
6 Fall6 Spring
0
50
100
Percent of cohort enrolled
13.9
11.3
13.5
10.3
15.3
9.4
20.9
7.6
URM and Non-URM
Non-Pell
Pell
Non-URM
URM
85 ©2014 U.S. Education Delivery Institute
Exercise: Determining Who and How
Determine the key people who will be involved in monitoring and how monitoring and decision making will
take place.
Answer the following:– Who from our team will lead this work?
– Who else needs to be involved?
– When and how often will we meet to review the progress of this strategy to improve our key outcome?
– What are our next steps? Who will do these things?
86 ©2014 U.S. Education Delivery Institute
Record your discussion on who and how using your strategy profile worksheet
87 ©2014 U.S. Education Delivery Institute
Workshop Agenda
Identifying Next Steps
88 ©2014 U.S. Education Delivery Institute
Our Next Steps…
Identify and write on cards your key takeaway and two
immediate next steps your team will take to actualize your
strategy profile
Thank Youwww.deliveryinstitute.o
rg
Ellyn [email protected]
Omari [email protected]
g
Duncan [email protected]