Data: What Do We Mean and What Kind Do We Need? Facilitator: Chris Hill, Grossmont College Presenters: Elaine Kuo, Foothill College Mallory Newell, De

Data: What Do We Mean Data: What Do We Mean and What Kind Do We and What Kind Do We Need?Need?

Facilitator: Chris Hill, Grossmont CollegePresenters: Elaine Kuo, Foothill College

Mallory Newell, De Anza College

State of Accreditation State of Accreditation TodayToday

American Council of Education recommendations on accreditation1.Transparency – accreditation documents; student outcomes2.Metrics and measures – improvement of data; common, comparative measures3.Defined pathways to student success – increased focus on matriculation and student support; incentives for completion4.New pedagogies and new entities – MOOCs; assure comparable quality of online learning5.Adaptive accreditation reviews – flexibility for expedited reviews of sounds institutions6.Enhanced gatekeeping – more pressure on accreditors and institutions to maintain quality between site visits

From ACCJC News, Fall 2012

2

State of Accreditation State of Accreditation TodayToday

From June 2011-June 2012, the ACCJC took the following institutional actions:1.28 institutions had a comprehensive evaluation2.12 institutions were reaffirmed3.10 institutions were placed on warning4.5 institutions were placed on probation5.1 institution was placed on show cause

From ACCJC Report of Institutional Actions from the June 2011, January 2012, June 2012 Commission Meetings 3

WHAT BRINGS YOU WHAT BRINGS YOU HERE?HERE?

4

Overview of sessionOverview of sessionWhat does ACCJC say about data

and evidence?What types of data/evidence are

needed?Where and how are data

discussed?How are data used for continuous

improvement?

5

WHAT DOES ACCJC WHAT DOES ACCJC SAY ABOUT DATA SAY ABOUT DATA AND EVIDENCE?AND EVIDENCE?

6

Good evidence used in evaluations has the following characteristics:

1. It is intentional, and a dialogue about its meaning and relevance has taken place.

2. It is purposeful, designed to answer questions the institution has raised.

3. It has been interpreted and reflected upon, not just offered up in its raw or unanalyzed form.

4. It is integrated and presented in a context of other information about the institution that creates a holistic view of the institution or program.

5. It is cumulative and is corroborated by multiple sources of data.

6. It is coherent and sound enough to provide guidance for improvement.

Characteristics of Characteristics of EvidenceEvidence

Source: ACCJC/WASC Guide to Evaluating Institutions, 20097

1. Use longitudinal data when possible

2. Use data in context

3. Look for both direct and indirect data

4. Do not oversimplify cause and effect of data

5. Use appropriate levels of data for appropriate levels of decisions

6. Perception is the reality within which people operate

7. Use of data should be transparent

8. Consider carefully when to aggregate or disaggregate data

9. Focus on data that is actionable

10.Consider implications and the “What if?

Data 101 - PrinciplesData 101 - Principles

From Data 101: Guiding Principles for Faculty (ASCCC, 2010)8

WHAT TYPES OF WHAT TYPES OF DATA ARE NEEDED?DATA ARE NEEDED?

9

Data SourcesData Sources External

◦ Accountability Reporting for California Community Colleges (ARCC)◦ Community College Survey of Student Engagement (CCSSE)◦ CSU Analytics/UC Statfinder◦ Department of Education – Data Quest◦ Career Technical Education Outcomes Survey◦ CCCCO DataMart◦ NCES (National Center for Educational Statistics)◦ CPEC (California Postsecondary Education Commission)◦ CalPASS

Internal◦ Annual Master Plan Updates◦ Degree and Transfer Data◦ Annual Factsheets by Program◦ Annual Program Review Data◦ Student Learning and Service Outcomes Assessments◦ Key Performance Indicators (KPIs)◦ Environmental Scans◦ Surveys◦ Ad hoc research studies

10

Building an Evidence-based Infrastructure Thinking about everything you do as a research study Working with your researcher and department to

collect data Sharing the data with your department and/or campus

Keeping Up with the Demand Setting aside a time each year to discuss the data

in your area Linking research to (resource) planning Making changes to your program based on the data

Turning Data into Action Ask questions, share it, refine it, ask for more Keep track of all the great things you are doing with

the data Share with others that may be data weary

Information Capacity Information Capacity ChallengesChallenges

From Fulks, Hasson, and Mahon - 2010 AI presentation

11

Helpful HintsHelpful HintsDisaggregate the data by ethnicity!Collect data and evidence on an

annual basis to inform your accreditation cycle (don’t have to collect it all in year 6).

Monitor the college’s progress on completing it’s planning agendas on an annual basis.

Assess the college’s institutional level outcomes on an annual basis to stay on track within your cycle, don’t forget about your AUOs as well.

12

Examples of evidence Examples of evidence related to student learning related to student learning

Information on Student Achievement: (i.e., student progress through the institution)

Is there evidence that the college has the capacity to collect, does collect, and uses in its own evaluation and planning processes, data on student achievement? Is there evidence that the college does so regularly? Is this data disaggregated by subpopulations where appropriate? This includes data on:

◦ Student demographics

◦ Student preparedness for college, including performance on placement tests

◦ and/or placement

◦ Student needs (i.e., local employment training needs, transfer education needs,

◦ basic skills needs, etc.)

◦ Course completion data

◦ Data on student progression to the next course/next level of course

◦ Data on student program (major) completion

◦ Data on student graduation rates

◦ Data on student transfer to a four-year institution

◦ Data on student job placements

◦ Data on licensure exams (scores, pass rates) 13

Examples of evidence Examples of evidence related to student learning related to student learning (cont’d)(cont’d)

Information on Student Learning Outcomes: (i.e., student mastery of the knowledge, skills, abilities, competencies, etc. identified by those designing the educational experience of the college).

Is there evidence that student learning outcomes are defined? ◦ By course ◦ By program ◦ By degree (including General Education requirements)

Is there evidence there was dialogue about the SLOs?◦ Prior to development◦ As part of developing integrated educational services and

courses/programs • As part of institutional self evaluation, planning, and improvement

◦ At the appropriate level of inclusion for the SLOs for courses, programs, and degrees (is it evident that SLOs are “tracked” from courses, through programs, and to certificate and degrees)

◦ In terms of how institutional processes can be oriented to better support learning. 14

Examples of evidence Examples of evidence related to student learning related to student learning (cont’d)(cont’d)

Information on Student Learning Outcomes: (i.e., student mastery of the knowledge, skills, abilities, competencies, etc. identified by those designing the educational experience of the college).

Is there evidence the SLOs are measured and the measurements are analyzed in order to:

◦ Inform pedagogy and help improve the educational services◦ Evaluate institutional effectiveness and plan institutional

improvements The rubrics created to describe SLOs and related measurement

strategies o The ways in which specific pedagogical practices are changed in response

to analyses of SLO attainment Analyses of SLO attainment used in the Program/Unit Review

process to improve student learning, programs, and services?

Is there evidence that students are learning?◦ Samples of student work◦ Copies of summary data on measured student learning

outcomes 15

Grades Percent Grades Percent Grades Percent Grades Percent

MATHD210. Pre-Algebra MPS FACE_TO_FACE 176 69% 54 21% 25 10% 255 100%

OTHER FACE_TO_FACE 7,916 57% 3,471 25% 2,608 19% 13,995 100%

MATHD212. Beginning Algebra MPS FACE_TO_FACE 869 76% 155 14% 124 11% 1,148 100%


MATHD114. Intermediate Algebra MPS FACE_TO_FACE 1,073 80% 163 12% 98 7% 1,334 100%


OTHER DISTANCE 405 40% 308 30% 299 30% 1,012 100%

MATHD010. Elementary Statistics MPS FACE_TO_FACE 982 88% 86 8% 53 5% 1,121 100%


OTHER DISTANCE 1,759 48% 855 24% 1,020 28% 3,634 100%

Pass Did Not Pass Withdrew Total

2001-2002 to 2011-12

From FHDA IR&P

Course Level Data:Course Level Data:Math Performance Success – Course Success Math Performance Success – Course Success Rates Rates by Ethnicityby Ethnicity

47%

77%

59% 57%62%

71% 73%67%

81%

70% 70%

78% 81% 84%

0%

20%

40%

60%

80%

100%

African American

Asian Filipino/PI Latino/a Native American

White Decline/Unk

Foothill 2011-12 Course Success Rates by Instructional Method and Ethnicity

Success Online Success Not online

Foothill total enrollment=141,538; African American online=2,972; African American not online=4,868; Asian online=10,970; Asian not online=29,902; Filipino/PI online=2,297; Filipino not online=5,831; Latino/a online=6,054; Latino/a not online=18,737; Native American online=358; Native American not online=879; White online=13,457; White not online=33,956; Decline to state/Unknown online=2,324; Decline to state/Unknown not online=8,942 Source: FHDA IR&P

Institution Level Data:Institution Level Data:Online Course Success Rates by EthnicityOnline Course Success Rates by Ethnicity

Non Developmental Students

Developmental Students

0%50%

FoothillDe Anza

23%

0%50%

17%

14% 23%

15%

15%

2012 CCSSE Cohort of full-time students

Full-time students who spend over 21 hours per week preparing for classes (studying, homework, rehearsing, reading, writing)?

Students who have taken or plan to take a developmental course. Developmental – DA=619, FH=290; Non Developmental – DA=306, FH=302

SSurvey Based Data – CCSSE: urvey Based Data – CCSSE: How many How many hours do you spend in a 7-day week hours do you spend in a 7-day week preparing for class?preparing for class?

De Anza Foothill

Ethnicity Respondents Percent Respondents Percent

African American 14 56% 19 58%

Asian /PI 387 58% 199 60%

Latino/a 112 52% 92 60%

White 129 53% 136 48%

Other 78 56% 46 45%Includes students who selected often or sometimes. Other includes Native American, Other, Decline to state. Total respondents DA=1,286; FH=904. DA: African American=25, Asian/PI=663, Latino=215, White=224, Other=139; FH: African American=33, Asian/PI=329, Latino=154, White=286, Other=102.

From FHDA IR&P, CCSSE 2012

SSurvey Based Data – CCSSE: urvey Based Data – CCSSE: How often How often do you use academic advising/planning?do you use academic advising/planning?

Program level data: Labor Projections and Program Program level data: Labor Projections and Program CompletionsCompletions

20

Radiologic Technology

Regional Radiologic Technology Training Providers

3 110*

Institutions Completions (2010-2011) *Based on IPEDS data.

Institution Degrees Certificates Total Completions

Institute of Medical Education 0 66 66

Foothill College 25 0 25

Canada College 19 0 19

Radiologic Technology Occupation Performance

Target Occupations Regional

Openings (2011) Median Hourly

Earnings Growth

(2012-2015) Job Postings§

As of 10/29/12 Radiologic Technologists and Technicians 106 $41.17 5.8% 0

From EMSI, 2012

21From Tim Nadreau, EMSI

Service Area Level Data: Service Area Level Data: Vocational Training ProjectionsVocational Training Projections

1/3 of jobs will require training involving up to one year’s worth of experience, training and/or instruction.

HOW ARE DATA AND HOW ARE DATA AND ASSESSMENT RESULTS ASSESSMENT RESULTS COMMUNICATED, COMMUNICATED, DISCUSSED, AND USED DISCUSSED, AND USED FOR CONTINUOUS FOR CONTINUOUS IMPROVEMENT?IMPROVEMENT?

22

Institutional Data Fills Two Important Institutional Data Fills Two Important GapsGaps

Elevated Level of

Organizational Awareness

Improved Processes & Performance

Knowledge Gap Performance Gap

Strategic Planning Function

Institutional Effectiveness & Student

Success Function

(Poor Planning) (Good Planning) (Good Performance)

Classroom and Service Area Assessment

Standard Level of

Organizational Awareness

Institutional Outcomes and Benchmarks


23

Review of data and evidence is most meaningful when it informs decision making at the proper place of practice

1,000 ftPerspecti

ve

100 ftPerspecti

ve

On the Ground

Institutional Strategies

Classroom Innovation

Program Improvements

• Resource Allocation

• Institutional Policies

• System Structures

• Program Alignment

• Program Redesign

• Program Curriculum

• Pedagogy

• Course Redesign

• Innovations in Learning

From Gregory Stoup, Pinpointing Areas to Improve (2012)

Where are data Where are data communicated communicated and discussed?and discussed?

25

Department College and/or District

Professional Development Activities

Committee/Council meetings

Department Meetings Institutional Planning Documents

Program Review Email/Newsletters

Student Learning Outcomes Assessment

Factsheets/Environmental Scans

Annual Planning Documents Institutional Website

Cañada College Student

Performance and Equity Dashboarddeveloped and maintained by

The Office of Planning, Research and Student Success


26

DETAILED TABLES1. Successful Course Completion Rates…………………….…………………………….2. Fall-to-Spring Persistence

……………………………………………………………………3. Fall-to-Fall Persistence

………………………………………………………………………..4. Student Success Rates during their first year……….…………..…………………5. Success Rates in Gen Ed Course

………………………………………………………….6. Success Rates in CTE Courses ……………………………………………………….

……..7. Success Rates in Pre-Transfer Courses ………………………………………….

……..8. Success Rates in ESL Courses ……………………………………………………….

……..9. Six Year Degree Completion Rates

……………………………………………………...10.Six Year Certificate Completion Rates

………………………………………………….11.Median Number of Years to Degree …….………………………….…….…………..12.Average # of Credits Accumulated after 1 Year ..……….………………………..13.Average # of Credits Accumulated after 2 Years ..……….……………….……..14.Pct Placed into BS Math & taking BS math in first term ……………………….15.Pct Placed into BS Math & taking BS math in first term ………………………16.Pct Placed into BS Math & taking BS math in first term ………………………

8101214161820222426283032343638


27

Five Year Trend in Succesful Course Completion Rates (Cañada College 2007/08 – 2011/12)

Why this matters: Course completion is perhaps the most widely used and reported indicator of student academic achievement. Higher levels of course completion are associated with higher levels of degree and certificate completion. This report highlights that for Cañada students course completion rates vary widely by both student ethnicity and student age.

Course Completion Rate: also referred to as the college-wide course pass rate and the college success rate is an aggregation of student course taking performance. The success rate is the percentage of grades awarded that indicate successful course completion, namely, a grade of A, B, C, P or CR.


28

Feedback from college constituenciesCollege-wide Succesful Course Completion Rate

Summary of the discussion of findings

College Planning Council. Reviewed on September, 2012.

Student Equity Committee. Reviewed on September, 2012.

Why the overall decline over this period? What are the underlying forces driving the trend and what are their various magnitudes? How much of this is due to demographic changes? Budget? Policy? Hiring patterns? Process reengineering?

The 25 percentage point gap by ethnicity is not acceptable. We must understand where we can best target efforts to pull up the lowest performing groups . We need to embark on a path of modest gains in terms of both closing the gap and improving the overall college average.

Next steps

College Planning Council.

Student Equity Committee.

OPRSS will identify completion rate trends within the college’s most enrolled courses. How do the rates of different courses compare? Did student enrollment in courses with traditionally lower completion rates over this period? CPC will review Program Review data that provides completion rate trends by dept.

OPRSS will provide completion information on cross sectional groups including ethnicity by gender, ethnicity by age, ethnicity by geography & ethnicity by financial aid status. The goal being to identify specific populations where focused interventions can be deployed and be most effective.

29From Gregory Stoup, Pinpointing Areas to Improve (2012)

Setting objectives related to Setting objectives related to college mission and goalscollege mission and goals

30

FOOTHILL COLLEGE Core Mission Workgroup Objectives for 2012-13

Institutional Goal Workgroup Objective

Map to Institutional Learning Outcomes Critical Thinking Computation Communication Community

Target

Resource Planning Estimated Cost $________________ Funding Source

Existing Potential Requested NA Timeline Target Date(s) ___________________

Metrics ● ● ●

Lead Role ● ● ●

Supporting Documentation ACCJC Standard _______ District Priority Educational and Strategic Master Plan Equity Plan PaRC Initiative Program Review Other_____________________________

References & Notes

Workgroup Participants

Documenting your process Documenting your process and progressand progress

31

FOOTHILL COLLEGE Core Mission Workgroup Reflections for 2012-13

Institutional Goal: Workgroup Objective Target Summary Completed In Process Not Initiated

Explain:

Successes ● ● ●

Challenges ● ● ●

Resource Planning Review Cost(s) $______________

Financial Personnel Technology Time Other ___________________

Progress Indicators (Metrics Update) References & Notes

Workgroup Participants

Example - Planning Agenda Example - Planning Agenda Progress and Completion Progress and Completion TemplateTemplate

Standard Planning AgendaGroup

Assignment*Date

CompletedDate to be Completed

I.A.3 As a component of the new planning process, the mission statement will be publicizedCollege Councilon a regular basis. The review of the mission statement will be integrated intothe planning process.

I.B.1 The college will implement the integrated planning process that incorporates College Counciloutcomes assessment results into institutional planning, and provide thetime and space for broad- based dialogue aimed at improving student learning.

I.B.3 The college will implement the integrated planning process that incorporates College Counciloutcomes assessment results into institutional planning, and student learning providethe time and space for broad-based dialogue aimed at improving student learning.

II.A.1 Working with other Planning and Budgeting Teams and College Council, the InstructionalCollege CouncilPlanning and Budget Team (IPBT) will review and modify the Annual Program PBTsReview Update and Comprehensive Program Review processes on a regular basis. Academic Senate

II.A.1.b Develop a Distance Learning course student evaluation, based on the Academic SenateFoothill-De Anza Faculty Agreement Article 6 and Appendix J2W. & FA

II.A.1.b Develop faculty training on effective online teaching strategies to improve Academic Servicesstudent success and retention.

II.A.1.c Institutional Research will continue its commitment to assisting faculty and staff in their College Councilassessment efforts at the course and program levels.

32

Developing the Research Agenda

1. What and who will be researched?

2. How is research tied to college plans, goals, initiatives and/or activities?

3. How will the information be used, by whom and how often?

4. Which methodology or approach will be used?

Turning Data into Information

1. What do the data tell us?

2. Which questions were fully answered by the research and which need more exploration?

3. What are reasonable benchmarks based on the research?

Taking Action on the Information

1. What interventions or strategies do we need to deploy in order to move the needle?

2. How should this information be shared and applied across the college?

Action Research Guided Action Research Guided QuestionsQuestions


33

WHAT IS YOUR WHAT IS YOUR BIGGEST TAKEAWAY BIGGEST TAKEAWAY FROM THIS SESSION?FROM THIS SESSION?

34