Implementing and Scaling an Online Early Intervention System in a Multi-Campus Environment

The first choice for faster results.

Implementing and Scaling an Online Early Intervention System in a Multi-Campus Environment

Norm DowneyPerkins Grant Director

Mary Kate QuinlanDirector of Learning Outcomes & Achieving the Dream

Dream 2013 Conference, Anaheim, CaliforniaThursday, February 7, 2013


Agenda

• CCAC and Early Intervention• The Challenges of Scaling• Statistically Driven Incremental Yearly

Improvement• Lessons Learned • Rewards of an Early Intervention System• Q & A

2Early Intervention - Dream Conference 2013


Community College of Allegheny County

• Serves Pittsburgh, PA metro– Four Campuses/Five Centers

• Annual unduplicated headcount– 32,000 credit – 36,000 non credit ‐

• 2,793 grads 2011 12‐• Among two year colleges:

– Largest producer of nursing grads– Second largest producer of allied health professionals



What is Early Intervention?

• Early Intervention (EI) is a web based system that allows faculty to refer struggling students for assistance via a secure website

• The staff members who receive the referrals attempt to contact referred students in order to link them to college resources

• The staff members then provide feedback to faculty and provide on-going support to the referred students

Early Intervention - Dream Conference 2013 4


Early Intervention’s Goals

• Reach struggling students as early in the term as possible

• Connect struggling students to college resources to resolve their academic issues

• Help students to maintain or improve their GPA

• Improve student retention rates



Which Students are Academically at Risk?

• Students who:– Have poor attendance– Are continually late for class or leave early– Don’t take notes– Are inattentive– Don’t participate

• Any student who in the instructor’s opinion is unlikely to be successful in a given course



Early Intervention is NOT…

• A substitute for good classroom management• A place to refer students with behavioral

problems or students who are potentially dangerous - those students should be referred to the Behavioral Intervention Team (BIT)



What’s Early?

• Early Intervention is best done as soon as a potential problem is recognized

• Earlier in the term there are more and better options to address student problems

• Early Intervention concludes about ⅓ the way into the term



The Challenges of Scaling

Scenario• EI is a successful practice that is benefitting a select population• You think it could benefit all studentsQuestion• What are some of the challenges to taking EI to scale?


Challenges of Scaling - Background

• EI had originally been a Perkins Grant initiative – Designed to serve only Career & Technical Education

students or roughly 50% of CCAC students– Scaling would expand EI to developmental students– Scaling meant “new” staff would suddenly be

handling EI, roughly tripling the staff involved• Timing of scaling– 4 months from scaling decision to implementation



Challenges of Scaling - Background

• One year after scaling to all students, three broad problem areas with EI were identified – Staff buy-in– Faculty buy-in– IT Development of data collection and reporting systems

• Priority was given to– Staff buy-in by developing systems to streamline Early Intervention

workloads– IT development to gather the necessary statistical evidence to

determine EI efficacy• Faculty buy-in and increasing referrals would come later



CCAC’s Hierarchy Of Student Services


Supportive Services

Pittsburgh Promise

KEYS - A Pennsylvania Welfare-to-Work Program

International Students

Success Coaches for Developmental Students

Veterans Services

Perkins Grant (All Career & Technical Education students, excluding those referred from developmental courses)

College Counselors (All students who don't fit one of the above service groups and CTE students referred from Developmental courses)


Challenges of Scaling – IT Issues

• Needed systems to:– Sort and assign referrals to correct student services group– Allow student services to provide feedback to faculty– Track referrals, staff responses and student outcomes

• First year lacked automated referral assignment, unified feedback systems and tracking systems

• Difficult to assess EI’s effectiveness because of a lack of statistical data



Challenges of Scaling – Staff Buy-in

• Confusion over the purpose of Early Intervention• Scaling of Perkins Grant initiative not well received

by all student services staff members– Concern over workloads and unequal work distribution– Concerns over perceived inappropriate referrals

• Faculty feedback issues– No easy mechanism to provide feedback– Lack of understanding why feedback is important



Challenges of Scaling – Faculty Perceptions

• Faculty confusion over Early Intervention’s intended purpose…– What is a good referral vs. a bad referral?– What’s meant by Early?

• Lack of timely and robust staff-to-faculty feedback created quality perception issues

• Confusion about EI process and student services structure



Incremental Improvement

Scenario• You expanded EI and identified several BIG challengesQuestion• How do you identify where to start?


Where to start?



Where to start?

• Staff buy-in, Faculty perception and the IT issues were strongly related around the area of streamlining workflows, capturing EI data and providing timely robust feedback to faculty

• Developing IT systems to better handle EI while providing additional staff training were identified as starting points

• We agreed to not attempt to grow the number of referrals until the systems these were in place to handle such an increase



Fixing the IT Issues

Problem• Sort and assign all referrals to

correct student services group

• Give student services simple system to provide feedback to faculty

• Track referrals, staff responses and student outcomes

Solution• Develop systems within

Datatel Colleague to sort and assign student referrals to services groups in the heirarchy

• Create Datatel screens to capture staff intervention data and report that back to faculty



Fixing the IT Issues

• EI scaled in Fall of 2008• Unified Datatel systems to automatically sort and

assign referral introduced in Fall 2010• Outcomes:– No student referral “falls through the cracks” – all are

assigned to staff– Allows staff to easily send feedback to faculty– Improves ability to gather and report EI statistics



Fixing the IT issues

• Same system used to sort referrals is used to:– Capture staff/student interactions– Issue feedback to faculty– Send e-mail to students referred for attendance problems

• Outcomes:– Provide systems for staff accountability– Gathers statistical evidence to determine EI’s efficacy– Improved faculty feedback and staff buy-in by reducing

workload through automation



Fixing Staff Buy-in

Problems• Confusion over EI’s purpose• Faculty feedback issues • Concern over workloads

and unequal work distribution

• Concerns over perceived inappropriate referrals

Solutions• On going training• Datatel screens to handle

staff/student interactions and issue feedback

• E-mails to students with poor attendance

• Workshops for faculty, ability to reassign referrals



Fixing Staff Buy-in – Outcomes

• Asked staff to develop scripts for talking to students and to create suggested work flows

• Reduced workloads through automation• Greater accountability through staff report cards• Outcomes:

– Increase in student referrals with documented staff interactions

– Increase in EI students with passing grades for referring course



Fixing Staff Buy-in – Report Cards



Fixing Faculty Perceptions

Problem• Confusion about EI’s

purpose…• Quality perception from

lack of staff-to-faculty feedback

• Confusion about process and student services structure

Solution• Workshops on classroom

management• Reminders to faculty

differentiate EI from BIT• Newsletter articles• Improved feedback• Yearly survey of faculty


Early Intervention - Dream Conference 2013

26


Sample text from automated feedback e-mailDear Walt, You referred the following student for early intervention: 9999999 Duck, Donald

Class Name: BUS-101-XX71

Location: XXTerm: 13SP

Informed of referral:Y

Late, poor attendance: NDoes not submit work on time:NDoes not take notes: NPoor participation/inattentive: NLow grades: YLow grades despite effort: Y

Comments:Instructor comments here.

Early Intervention action taken by Mouse, Mickey.Contact methods:

Telephone: Attempted 01/20/10

Email: Completed 02/15/10Postcard:Office Visit: Attempted

05/01/10 Action Taken/Type of Referral:

Personal? NCareer/Transfer? YAcademic? YTutoring? NSupportive Services? NCounseling? YJob Placement? Y

Comments:

Staff Comments here line oneStaff Comments here line twoStaff Comments here line three

PLEASE DO NOT REPLY TO THIS MESSAGE.

Shown as two columns for display purposes

The Instructor

The StudentStudent

Services Staff member


Fixing Faculty Perceptions – Outcomes

• Fewer inappropriate referrals for classroom management issues

• Increases in total student referrals • Increase in number of faculty making referrals• Based on faculty survey results– Improved opinion by faculty of EI’s effectiveness– EI referral time increase by one week to accommodate

faculty wishes



Lessons Learned

• What have you learned that you could apply to your own intervention?


Lessons Learned

• Timing of scaling• Preparing staff and faculty for scaling , thus gaining

buy-in• Building minimum IT systems before scaling• Importance of accountability• Incremental improvement based on survey data and

student outcomes



Rewards of an Early Intervention System


Assessing EI’s Efficacy

• CCAC’s homegrown EI Datatel system allows for the capture of – Referral data that links the referral to a specific student,

course and instructor– Intervention data that links the referral data to the staff

member’s intervention efforts and student’s final grade• Data is combined in an Access database to produce

reports and publish statistics.



† A student can be referred from more than one class – this is an unduplicated count* For Academic Challenges Referrals with staff interaction

Early Intervention Statistical OutcomesStatistic Term 10FA 11SP Year

Totals 11FA 12SP YearTotals

Yearly Difference

Total Referrals 681 610 1291 719 635 1354 + 63Unique Students Referred † 660 580 1240 684 593 1277 + 37

Faculty referring students 185 192 377 231 206 437 + 60

Referrals with Staff Interactions 196 104 300 267 215 482 + 182

# with A, B or C Final Grade * 86 39 125 91 117 208 + 83



Early Intervention Student RetentionStatistic Term Fall 2010 to Spring 2011 Fall 2011 to Spring 2012

Students referred for EI 77.2% 76.8%

College Retention Rate 67.7% 66.3%

Difference 9.5% 10.5%

33

• This table is a comparison of the retention rates for students referred for EI versus the overall college retention rates.

• The data indicates that students referred for EI have retention rates that are on average 10% greater than the rest of the student population.

Early Intervention - Dream Conference 2013


What’s next?

• Continue with staff training• Work to improve feedback quality and timeliness• Add new tools to the staff tool box – StudentLingo,

Math Café and Learning Commons• Conduct assessments of these new tools• Work on increasing faculty referrals• Outreach to students to better explain EI and it’s

goals



Questions, Answers and Discussion


Presenter Contact Information

Norm DowneyPerkins Grant DirectorCommunity College of Allegheny County808 Ridge Ave.,Pittsburgh, PA [email protected](412) 237-4670

Mary Kate QuinlanDirector of Learning Outcomes & Achieving the DreamCommunity College of Allegheny County800 Allegheny Ave.,Pittsburgh, PA [email protected](412) 237-3196


mailto:[email protected]

mailto:[email protected]

Documents

Implementing and Scaling an Online Early Intervention System in a Multi-Campus Environment