Program Evaluation andProgram Evaluation and
AssessmentAssessment
David Parker, University of ConnecticutDavid Parker, University of Connecticut
Elaine Elaine ManglitzManglitz, Clayton State University, Clayton State University
Sam Goodin, University of MichiganSam Goodin, University of Michigan
Steve Robillard, SR-PS, Inc.Steve Robillard, SR-PS, Inc.
Richard Riccardi, Southern Connecticut State UniversityRichard Riccardi, Southern Connecticut State University
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
TodayToday’’s Agendas Agenda
2:00 Introductions2:00 Introductions
2:10 Overview; Reasons for Conducting a Program2:10 Overview; Reasons for Conducting a Program
Evaluation (David) Evaluation (David)
2:35 What to Evaluate: CAS and AHEAD Program2:35 What to Evaluate: CAS and AHEAD Program Standards (Elaine) Standards (Elaine)
2:55 Using Survey Tools to Gather Data (Sam)2:55 Using Survey Tools to Gather Data (Sam)
3:15 Break3:15 Break
3:35 Use of Non-Survey Tools to Collect Data (Steve)3:35 Use of Non-Survey Tools to Collect Data (Steve)
3:55 Tools for Storing, Analyzing, and Presenting Data3:55 Tools for Storing, Analyzing, and Presenting Data
(Richard) (Richard)
4:15 Interactive Segment (Suggestions for Utilizing4:15 Interactive Segment (Suggestions for Utilizing
Evaluation Results; Final Q&A) Evaluation Results; Final Q&A)
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Reasons for Conducting a ProgramReasons for Conducting a Program
EvaluationEvaluation
David ParkerDavid Parker
University of ConnecticutUniversity of Connecticut
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Accompanying MaterialsAccompanying Materials
Materials accompanying this symposium session are available
online at AHEAD’s learning management system:
http://ahead.coursewebs.com
Enter username: assessment
Password: ahead2007
Click on AHEADSYM6 Login
This will bring you to a page with instructions for accessing your
copy of Program Evaluation of Postsecondary Disability
Services: From Theory to Practice and accompanying
materials.
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Reasons for ConductingReasons for Conducting
a Program Evaluationa Program Evaluation
Ensure Accountability
“Given the fiscal, administrative, and
legal accountability of OSDs, it is vital
that the personnel responsible for
ensuring equal educational access
conduct a periodic self-assessment of
their performance.”
- Dukes & Shaw(1999)
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Reasons for ConductingReasons for Conducting
a Program Evaluationa Program Evaluation
Inform Decision Making
Conduct an LD program evaluation to:
Justify program needs (staff, space, equipment) and
academic lines of reporting (academic or student
affairs)
Monitor student achievement
Document compliance with Section 504/ADA
Analyze effectiveness of program services
Assist in developing institutional policy (e.g., course
substitution policy)
- Brinckerhoff, McGuire, & Shaw (2002)
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Reasons for ConductingReasons for Conducting
a Program Evaluationa Program Evaluation
3) Promote Self-determination
When sensitive to context, program evaluations provide datathat help DSS professionals establish, implement, andevaluate a programmatic mission:
Establish descriptive data/baselines
Clarify problems
Document successes
Meet peer-driven standards of “best practice”
- Goodin, Parker, Shaw, & McGuire (2004)
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Reasons for ConductingReasons for Conducting
a Program Evaluationa Program Evaluation
“What are our peer institutions doing?”
Program Evaluation of Postsecondary Disability Services:From Theory to Practice
2) DSS Program Evaluations (Questionnaire Results)
Use of data
Planning to evaluate
Anticipated benefits
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Using Survey Tools to Gather DataUsing Survey Tools to Gather Data
Sam GoodinSam GoodinUniversity of MichiganUniversity of Michigan
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
The The ““RealReal”” Reasons to do Reasons to do
Program EvaluationProgram Evaluation
DSS staff do not have the time to reconfigure supportDSS staff do not have the time to reconfigure supportservice mechanisms on the whims of concernedservice mechanisms on the whims of concernedstudents, staff, supervisors or officials from the Office ofstudents, staff, supervisors or officials from the Office ofCivil Rights.Civil Rights.
We do not have the blood pressure and cholesterolWe do not have the blood pressure and cholesterollevels that will allow us to spend our entire professionallevels that will allow us to spend our entire professionallives putting out fires.lives putting out fires.
We do not have spouses, children and significantWe do not have spouses, children and significantothers who will forever tolerate our spending weekendsothers who will forever tolerate our spending weekendsand evenings creating pilot projects for which we findand evenings creating pilot projects for which we findno demand.no demand.
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Modular Mailed SurveyModular Mailed SurveyModule I General questions for all students
Modules II A-D Disability specific questions
Module III Demographic data
Please read each item and rate the availability and quality of each
service.
1 = poor 2 = fair 3 = adequate 4 = good 5 = excellent
___ Reader services (availability)
___ Reader services (quality)
___ Loan of Special equipment (e.g., recorder) (availability)
___ Loan of Special equipment (e.g., recorder) (quality)
___ Test/quiz administration (availability)
___ Test/quiz administration (quality)
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Phone SurveyPhone Survey
Its much easier to terminate a telephone interview abruptly,
saying something like, “Whoops! someone's at the door. I gotta
go.” or “OMIGOD! The pigs are eating my Volvo!”
- Babbie, 1992
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Evaluating Other ServicesEvaluating Other Services
When people ask “how big is your staff?” I am fond of telling
them I have a staff of 2,500. That usually gets a look (and a
stir) out of the questioner, but in fact, on our campus, that is the
approximate number of our faculty and staff.
- Harris, 1984
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Instruments for EvaluatingInstruments for Evaluating
Other ServicesOther Services
Other Campus Offices:
Admissions
Counseling
Parking Office
Health Service
Housing
Library
Career Center
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Evaluation of Service ProvidersEvaluation of Service Providers
…formal performance evaluation probably began in United
States in 1813 (Bellows & Estep, 1954) when army general
Lewis Cass submitted to the War Department an evaluation of
each of his men using such terms as “a good natured man” or
“knave despised by all.”
- Murphy & Clevland, 1995
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Instruments for EvaluatingInstruments for Evaluating
Service ProvidersService Providers
Readers
Interpreters
Notetakers
Professional and Administrative Staff
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
StudentVoiceStudentVoice.com
Each StudentVoice toolkit provides each studentaffairs unit with specific assessment programsfrom which they can choose. These programsinclude common instruments for comparison withpeer institutions and the opportunity to addcampus-specific questions to address preciseissues or concerns that help Directors monitor theusage, satisfaction and outcomes.
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Need a Title
How do you rate the ease of finding out about
Paratransit/Yellow Cab service?
How do you rate the ease of registering with SSD to be eligible
for these services?
Have you used the Yellow Cabs?
Please rate the following regarding your experience with the
Yellow Cabs: - The cab consistently came within the promised
time.
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Using Survey Tools to Gather DataUsing Survey Tools to Gather Data
Elaine Elaine ManglitzManglitzClayton State UniversityClayton State University
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
What to Evaluate: CAS andWhat to Evaluate: CAS and
AHEAD Program StandardsAHEAD Program Standards
AHEAD Program StandardsAHEAD Program Standards
–– http://www.ahead.org/resources.phphttp://www.ahead.org/resources.php
CAS Professional Standards for HigherCAS Professional Standards for Higher
Education, 6Education, 6thth edition edition
–– http://www.cas.edu/http://www.cas.edu/
AHEAD Program StandardsAHEAD Program Standards
Revised professional standards andRevised professional standards and
performance indicators for our fieldperformance indicators for our field
Developed with input from experts withinDeveloped with input from experts within
the fieldthe field
Can direct program evaluation andCan direct program evaluation and
development effortsdevelopment efforts
Supports staff development effortsSupports staff development efforts
CAS Professional StandardsCAS Professional Standards
Consistent format across all functionalConsistent format across all functional
areas, including disability support servicesareas, including disability support services
Tied to student learning and developmentTied to student learning and development
Self-assessment guidesSelf-assessment guides
Use to gauge program effectivenessUse to gauge program effectiveness
Use to create an action plan forUse to create an action plan for
improvementimprovement
Supports professional staff developmentSupports professional staff development
Recommendations for Using theRecommendations for Using the
Assessment ToolsAssessment Tools
Using both AHEAD program standardsUsing both AHEAD program standards
and CAS professional standardsand CAS professional standards
Tie to institutionTie to institution’’s vision and missions vision and mission
Implementation of evaluation efforts: noImplementation of evaluation efforts: no
one one rightright way way
Making the tools work for your office andMaking the tools work for your office and
your needsyour needs
Additional RecommendationsAdditional Recommendations
Assess/evaluate how well the disabilitiesAssess/evaluate how well the disabilities
service office meets the mission of theservice office meets the mission of the
UniversityUniversity
Assess/evaluate how well the disabilitiesAssess/evaluate how well the disabilities
service office meets the vision and missionservice office meets the vision and mission
of the general Division of which it is a partof the general Division of which it is a part
(Academic or Student Affairs)(Academic or Student Affairs)
Using Survey Tools to Gather DataUsing Survey Tools to Gather Data
Steve RobillardSteve Robillard
SR-PS, Inc.SR-PS, Inc.
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Why Use Non-Survey Data?Why Use Non-Survey Data?
If the only tool you have is a hammer, everyIf the only tool you have is a hammer, every
problem is a nail.problem is a nail.
TrendsTrends
Overall shift from qualitative toOverall shift from qualitative to
quantitative, or better yet, both.quantitative, or better yet, both.
Graduation, retention, student learnerGraduation, retention, student learner
outcomesoutcomes
What is available?What is available?
Looking in your own backyardLooking in your own backyard
The UniversityThe University
StateState
FederalFederal
Peer institutionsPeer institutions
ExamplesExamples
Staff activityStaff activity
CSP 100CSP 100
Designated MathDesignated Math
Extended TimeExtended Time
Using Survey Tools to Gather DataUsing Survey Tools to Gather Data
Richard RiccardiRichard Riccardi
Southern Connecticut State UniversitySouthern Connecticut State University
AHEAD 2007, Charlotte, North Carolina, USA July 17-21, 2007
Think Before You DigThink Before You Dig……
Identify the problem to be addressedIdentify the problem to be addressed
–– ExistenceExistenceHave you used the Yellow Cabs?
–– Opinion based (Satisfaction/Agreement)Opinion based (Satisfaction/Agreement)
How do you rate the easeHow do you rate the ease……
–– Time basedTime based
How long have you used the service?How long have you used the service?
–– CombinationCombination
Is the service Is the service betterbetter than before? than before?
What is What is ““better?better?””
The Old Wheel Is RoundThe Old Wheel Is Round……
The New Wheel Is RoundThe New Wheel Is Round
Your own system (electronic, paper)Your own system (electronic, paper)
Student Information SystemStudent Information System
–– Extracts (IT, IR, Assessment, self)Extracts (IT, IR, Assessment, self)
School profiles (own, peer schools)School profiles (own, peer schools)
State/Federal data (IPEDS, US News)State/Federal data (IPEDS, US News)
Noel Levitz, NSSENoel Levitz, NSSE
Collecting DataCollecting Data
Survey ToolsSurvey Tools
–– SurveyMonkeySurveyMonkey
–– Snap SurveySnap Survey
–– ZoomerangZoomerang
–– ExploranceExplorance
–– StudentVoiceStudentVoice
–– Paper and #2 pencilsPaper and #2 pencils
–– TelephoneTelephone
Issues with Survey ToolsIssues with Survey Tools
Ease of useEase of use
AccessibilityAccessibility
Web-basedWeb-based
AudienceAudience
SecuritySecurity
Data ownershipData ownership
ITIT
Survey design (reliability, validity)Survey design (reliability, validity)
Analyzing DataAnalyzing Data
Microsoft ExcelMicrosoft Excel
–– Universal (PC, Macintosh)Universal (PC, Macintosh)
–– Easy to useEasy to use
–– Some statistical power (Pivot Tables, t-Test,Some statistical power (Pivot Tables, t-Test,
z-Test, Chi-Square, Pearson correlation)z-Test, Chi-Square, Pearson correlation)
Power ToolsPower Tools
–– Microsoft Access, Microsoft Access, FilemakerFilemaker, Oracle, , Oracle, MySQLMySQL
–– SPSS, SASSPSS, SAS
Excel Excel –– Pivot Tables Pivot Tables
Outcome Data
Fail No Grade Pass Withdraw Total Students
Term Subject Course Section Students % Students % Students % Students %
Fall 1993 CSP 100 02 2 9.1% 0.0% 19 86.4% 1 4.5% 22
Fall 1994 CSP 100 01 2 9.5% 0.0% 18 85.7% 1 4.8% 21
Fall 1995 CSP 100 01 1 4.5% 1 4.6% 20 90.9% 0.0% 22
Fall 1996 CSP 100 01 3 13.0% 0.0% 20 87.0% 0.0% 23
Fall 1997 CSP 100 01 3 12.5% 0.0% 20 83.3% 1 4.2% 24
Fall 1998 CSP 100 01 4 19.0% 0.0% 17 81.0% 0.0% 21
Fall 1999 CSP 100 01 2 13.3% 1 6.7% 12 80.0% 0.0% 15
Fall 2000 CSP 100 01 2 7.7% 0.0% 23 88.5% 1 3.8% 26
Fall 2001 CSP 100 02 0.0% 0.0% 22 95.7% 1 4.3% 23
Fall 2002 CSP 100 01 0.0% 0.0% 17 94.4% 1 5.6% 18
Fall 2003 CSP 100 01 1 4.3% 0.0% 21 91.3% 1 4.3% 23
Fall 2004 CSP 100 01 1 5.0% 0.0% 19 95.0% 0.0% 20
Grand Total 21 8.1% 2 0.8% 228 88.4% 7 2.7% 258
Legend (groupings by grade): Fail = D- or F, Incomplete = I or I+, No Grade = N or R, Pass = D or better, Withdraw = W, WP, or WF
Outcomes for CSP 100 - Fall 1993 to Fall 2004
A Picture Can Be Worth an FTEA Picture Can Be Worth an FTE……
Student/FT Staff Ratio
0
50
100
150
200
250
2003 2004 2005 2006 2007
Year
Stu
den
ts
Year FT Staff Students Served Student Ratio
2003 6 450 75
2004 5 575 115
2005 5 600 120
2006 4 700 175
2007 4 800 200
Presenting InformationPresenting Information
Know Your AudienceKnow Your Audience
–– ““Just give me the bullets.Just give me the bullets.””
–– How does your data fit with the mission and vision ofHow does your data fit with the mission and vision ofuniversity?university?
Know Your PurposeKnow Your Purpose
–– Set baselineSet baseline
–– Improve serviceImprove service
–– Increase fundingIncrease funding
KISS MethodKISS Method
–– If it needs explaining, it needs to be simplified.If it needs explaining, it needs to be simplified.
The Assessment CycleThe Assessment Cycle
Problem + Data + Standards =Problem + Data + Standards =
InformationInformation
Information + Context =Information + Context =
KnowledgeKnowledge
Appling the Knowledge Gained =Appling the Knowledge Gained =
AssessmentAssessment