The Academic Program Review Bridging Standards 7 and 14 Middle States Annual Conference December 10,...

The Academic Program Review

Bridging Standards 7 and 14

Middle State’s Annual Conference

December 10, 2010

Presenters

Mr. H. Leon Hill

Director of Institutional Research

Dr. Joan E. Brookshire

Associate Vice President of Academic Affairs

Overview

• Framework to Address the APRs

• Structure/Challenges/Approach

• Examples of Metrics

• Current Action Plan

• Integration of “End User” Technology

• Next Steps

• Benefits of Our Approach

• Questions

Assessment Cycle-2005

What we had to build on

• Strong focus on programs.

• State mandated 5-year academic program review in need of revision.

• Institutional Effectiveness Model (IEM) with performance indicators benchmarked through State and National data bases.

Mission

Strategic Initiative:

Access & Success

Institutional Effectiveness

IEM

• Needed a way to assess how the College was performing on key metrics in relation to prior. years/semesters and compared to other institutions.

• Historical/Trend data

• Benchmark data– Pennsylvania & National Peers

• Institutional Effectiveness Model

Where we started

• Restructured the Academic Program Review process

• Incorporated the use of technology

Goal of the restructuring

• Measure student performance as evidence by results of assessment of student learning outcomes.

• Measure program performance as evidenced by comparison of program performance to overall college performance on specific key indicator (current and aspirational).

Challenges

• Usual issues with assessment in general.

• Faculty had little knowledge of the College’s performance indicators.

• Organizational separation of assessment of institutional and student learning outcomes.

Approach

Began by building it backwards from the IEM by mapping out specific core indicators to program data, making additions where needed.

Examples of Metrics Used for APR

TARGETS Caution Acceptable Aspirational

Graduation Rate

<19% 19%-23% >23%

TARGETS Caution Acceptable Aspirational

Transfer Rate <29% 29%-32% >32%

Definitions of Success & Retention

Success=Grades of (A,B,C & P)/(A, B, C, D, P, D, F, & W)

Retention=Grades of (W)/(A, B, C, D, P, D, F, & W)

Added a curricular analysis

• How well program goals support the college’s mission.

• How well individual course outcomes reinforce program outcomes.

• How well instruction aligns with the learning outcomes.

• Specific assessment results.

• Changes made based on the assessment findings.

• Evidence of closing the loop

• Changes made to the assessment plan.

Action Plan

• Outcomes expected as a result of appropriate actions steps.

• Timelines and persons responsible for each action step.

• Resources needed with specific budget requests.

• Evaluation plan with expected benefits.

Bottom Line

• Is there sufficient evidence that the program learning outcomes are being met?

• Is there sufficient evidence that the program is aligned with the college on specific key indicators?

The Framework

STANDARDS 1-7 Focus on Institutional Performance

Standard 7: Assessment of Institutional Effectiveness How well are we collectively doing what we say we are doing, with a specific focus on supporting student learning. Assessment must be included in Standards 1-6.

STANDARDS 8-14 Focus on Student Performance

Standard 14: Assessment of Student Learning Do we have clearly articulated learning goals, offer appropriate learning activities, assess student achievement of those learning outcomes, and use the result of assessment to improve teaching and learning and inform budgeting and planning. Also in Standards 8-13.

Strategic Analysis

Institutional data Link to the IEM

through use of common data sets

Curriculum Analysis Course assessment Program Assessment Core assessment

Planning and Budgeting (Standard 2)APR Action PlanAPR Annual ReportAnnual Academic Planning

Assessment Results

Curriculum Committee

President’s Office

Curriculum BOT & BOT

Addition of Technology

• Worked in concert with Information Technology to integrate iStrategy with ERP (Datatel).

• The implementation of this permitted end users to obtain the data needed for program assessment, without the middle man (IR and/or IT).

Next Steps in the Evolution of of College and Program Outcomes

Example of APR Report Card

Persistence Fall to Fall

2005 2006 2007 Full-time

Persistence Program 75% 50% 70% College 64.39% 63.16% 50%

Part-time Persistence

Program 25% 43.48% 33.33% College 33.52% 37.09% 34.41%

Persistence - These data are based on a cohort of first time students from a specific semester and follows their enrollment patterns one year out

Fall to Spring 2006 2007 2008

Full-time Persistence

Program 70% 80% 76.92% College 78.19% 80.98% 80.93%

Part-time Persistence

Program 69.57% 45.83% 46.15% College 45.12% 41.60% 39.00%

Persistence - These data are based on a cohort of first time students from a specific semester and follows their enrollment patterns from Fall semester to the following Spring semester

Graduation and Transfer

Degrees Conferred

2006 2007 2008 Three Year

Percent Change

Program 9 9 8 -11.11% College 1058 1046 1127 6.52%

Degrees Conferred – These are the actual number of degrees conferred, not the degrees earned

Pass Rates on Licensure Exams

Graduation and Transfer Rate: Cohort Entering 2005

Total

Students Degrees

Graduation Rate

(Within 3 Years) Transfer

Transfer Rate

(Within 3 Years)

Full-time Program 14 1 7.14% 5 35.71% College 1524 192 12.60%* 402 26.38%

Part-time Program 9 0 0% 4 44.44% College 990 26 2.62% 232 23.43%

First time students to College, no transfer credits *Acceptable 19-23%

Fiscal

Evidence of High Priority and Employment Predictions

• Examples of Course Success

Success in ACC 111

2003/FA 2004/FA 2005/FA 2006/FA

2007/FA

2008/FA

2009/FA

% Success 61.4% 57.1% 55.4% 55.3% 51.4% 44.2% 48.3%# Success 329 276 253 281 261 244 249% Non Success 38.6% 42.9% 44.6% 44.7% 48.6% 55.8% 51.7%# Non Success 207 207 204 227 247 308 267

Success in ACC 111

  2003/FA 2004/FA 2005/FA 2006/FA 2007/FA 2008/FA 2009/FA

% Female Success 63.3% 57.5% 58.8% 57.7% 57.3% 51.8% 58.7%

Female Success 145 111 104 123 114 115 105

% Male Success 59.8% 56.8% 53.2% 53.6% 47.2% 39.1% 42.1%

Male Success 180 163 149 158 145 127 133

Success in Math 010

  2003/FA 2004/FA 2005/FA 2006/FA 2007/FA 2008/FA 2009/FA% Success 53.6% 46.3% 47.3% 45.7% 44.8% 43.3% 47.4%

Success 310 266 276 293 297 288 344

% Non Success 46.4% 53.7% 52.7% 54.3% 55.2% 56.7% 52.6%

Non Success 268 309 307 348 366 377 381

Success in Math 010

  2003/FA 2004/FA 2005/FA 2006/FA 2007/FA 2008/FA 2009/FA% African American Success

42.6% 37.7% 38.5% 25.8% 26.9% 29.9% 34.7%

African American Success

43 46 40 33 45 56 51

% Caucasian Success

58.2% 51.8% 50.3% 52.7% 53.5% 48.4% 52.0%

Caucasian Success 202 184 180 217 206 180 141

Benefits

• Build a bridge between Standards 7 and 14.

• Better data.

• By putting data in the hands of faculty, have them actively engaged with using data in decisions/planning.

• IR time better used.

• Annual planning cycle developed.• Built a culture of assessment in

several of the academic divisions.• Curricular changes that align with

graduation initiative.• Curricular and program

improvement.• Created a college-wide model for

improvement of student learning.

Evolution of the Dashboard

• Creation of a Student Success DashboardMetrics:

Course level success and retention (Developmental and College-Level)

Persistence (fall to spring and fall to fall) Progression of various cohorts of students College level success in Math or English

after Developmental Math or English Graduation Transfer

Graphic Representation for the SSD

Graphic Representation for the SSD

Final Thoughts

It’s not perfect, but it works for us. Do the research on which tools

are appropriate for your college Assessment of the core curriculum Launching of assessment software It all starts with asking the right

question PRR 2010

Questions

Presenters

Mr. H. Leon Hill

hlhill@mc3.edu

Director of Institutional Research

Dr. Joan E. Brookshire

jbrooksh@mc3.edu

Associate Vice President of Academic Affairs