Upload
roger-lloyd
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Student Learning Outcomes:
How Counseling/Advisement and Select Student Services Programs Impact Students’
Self-Development
2004 NACADA PACIFIC REGION 9 CONFERENCE
Pasadena, California April 21 – 23, 2004
Esau Tovar Merril A. Simon
2
Contact Information
Esau Tovar, M.S.Faculty Leader/Counselor, Assessment CenterSanta Monica College1900 Pico Blvd. Santa Monica, CA 90405(310) 434-4012 [email protected] A. Simon, Ph.D.Assistant Professor, Educational Psychology and CounselingCalifornia State University Northridge18111 Nordhoff St. Northridge, CA [email protected]
3
Presentation Abstract
As the number of students attending community colleges continues to rise, we face ever-increasing budgetary constraints while experiencing a demand to serve students’ diverse needs without sufficient resources to do so. As we respond to accreditation standards focusing on the assessment of student learning outcomes and of demands for greater accountability, it is imperative that advising and student services programs systematically measure the impact of our services. This presentation will focus on the development and use of a 67-item student services survey measuring the use of college-wide student services and the degree of self-development experienced by students as a result of attending college. Specific contributions to students’ self-development (e.g., values, self-management) by counseling/advising and other services will be highlighted.
Why Assessment?
5
Need Identified (SMC)
Primarily resulting from institutional accreditation self-study;
New accreditation standards; Need to measure student learning
outcomes in both academic and student services; Focus on usage and satisfaction with
services; and Contribution of student services to
students’ self-development (affective development)
6
Why Conduct Student SupportServices Assessment?
Schuh & Upcraft (2001) cite the following as reasons: Survival Quality Affordability & Cost Effectiveness Strategic Planning Policy Development & Decision
making Politics Accreditation
7
What is Assessment?
Upcraft & Schuh (1996) indicate that:
“Assessment is any effort to gather, analyze, and interpret evidence which describes institutional, divisional, or agency effectiveness.”
8
What Does Assessment Model Consist of?
Schuh & Upcraft (2001) recommend the components of: Tracking—Utilization of WHAT & by WHOM; Needs Assessment Satisfaction of Services Student Cultures & Campus EnvironmentOutcomes Assessment (effect for learning,
development, student success—users vs. non-users)
Peer institution ComparisonNational Standards—CAS Cost Effectiveness
9
Reporting
Implications of
Findings
Data Analysis
Whom?
Instrumen-tation
How?
Data CollectionWhom?
Best Assessment
Method
Where?
Purpose
DefineProblem
Assessment Process
Assessment Process: Schuh & Upcraft (2001)
10
Assessment is Iterative
Mission/Purposes
Objectives/Goals
Outcomes
Gather Evidence
How well outcomes are achieved?
Interpret Evidence
Decisions to improve; enhance SLOs; inform stakeholders; planning; budgeting.
Specific Community College
Example
12
Counseling DepartmentMission Statement
The Counseling Department is committed to promoting student success by providing a broad range of innovative services that address the educational, developmental, psychological, and social needs of Santa Monica College students. We actively contribute to the broader academic mission of the College through our instructional services and by building a multicultural learning community.
Values StatementThe Counseling Department places priority on the following values as it pursues its mission by fostering:
• Responsiveness to the developmental growth and changing needs of students and staff. • Innovation and creativity in the services we offer. • Interpersonal respect at all levels.• Reinforcement of student responsibility, self-direction, and decision-making skills.• Diversity as reflected in our programs, services, and staff composition. • Commitment to the ethical standards of the counseling profession.
13
Assessment in Counseling
Interest in measuring the contributions of student services—particularly counseling—to students’ self-development while in college.
Interest in operationalizing select components of the Santa Monica College Counseling Department’s Mission Statement.
Moving beyond assessment of utilization/ satisfaction with services.
14
Assessment in Counseling (cont.)
Many student services designed and driven by student development/environmental models: Student retention (Tinto, 1993) Student engagement (Astin, 1993) How College Affects Students (Pascarella & Terenzini,
1990)
Designing new curricula emphasizing both cognitive and non-cognitive components: e.g., American Cultures requirement
Items written were based on problems or issues discussed with students in a counseling setting.
15
Survey Characteristics
67 Items with four sections:1. Background information;
2. Awareness and usage of, and satisfaction with twenty student services programs;
3. Importance and agreement to items relating to counseling services, course enrollment/registration services, financial aid, safety and involvement; and
4. Degree to which students’ college experiences contributed to their self-development.
16
Survey Characteristics (continued)
Internal Consistency – Chronbach’s alpha:Total Survey .99
Awareness/usage of student services .87
Satisfaction with student services .99
Importance of counseling, course enrollment, financial aid, safety/ involvement
.95
Agreement of counseling, course enrollment, financial aid, safety/involvement
.96
Self-Development Full Scale .94
17
Sampling Protocol
Administered in-and-out of classroom:
Randomly selected classes from across disciplines (academic & vocational courses; excluded non-graded; did not sample early-ending or online courses).
• Collected 1,156 (81% of total surveys )
Also administered by 20 different Student Services Programs as students presented for services
• Collected 277 (19% of total surveys )
18
About Santa Monica College(at time of initial assessment)
Spring 2003 Enrollment:
Population: 27,850 graded
students
Gender: 57% female, 43% male
Status: 30% full-time; 10% F1-Visa
Race/Ethnicity: 37% White,
27% Latino, 20% Asian, 9%
African American, 4% Other, 3%
Filipino
19
Demographics of Respondents
26% F1-Visa
Goal: 86% AA
Degree/Transfer; 8% Career Certificate; 4% Personal Growth; 1% Basic Skills.
Age: M = 24.5
SMC Attendance: 40% 1-2 semesters 31% 3-4 semesters 15% 5-6 semesters 14% Over 3 years
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
Male Female Total
American Indian Asian/Pacific IslanderBlack/African AmericanLatinoWhite Other
20
Awareness of Student Services
66
29
65
28
65
32
62
32
48 46
0
10
20
30
40
50
60
70
Not aware of service Aware of service but never used it.
TRIO/Student Support ServicesStudent Success ProjectVeterans' Programs Pico Partnership ProgramPsychological Services
Percentage of Least Aware
21
Utilization of Student Services
20
29
11
59
18 19
12
50
10 11
16
37
25
8
2
34
12 13
4
29
0
10
20
30
40
50
60
70
Used service once
Has used service a few times
Has used service a lot
Most Used
General CounselingTransfer/CounselingFinancial AidAssessment CenterHealth Services
Percentage of Most Used
22
Utilization of Student Services
3
2 2
7
4
2
1
7
3
1 1
6
2
1
2
6
2
1
3
6
1 1 1
3
0
1
2
3
4
5
6
7
8
Used service once
Has used service a few times
Has used service a lot
Least Used
Student Success Project Women's CenterPsychological Services Pico Partnership ProgramTRIO/Student Support ServicesVeterans' Programs
Percentage of Least Used
23
Satisfaction with Student Services
59
17 17
52
35
1821
52
4 4
15
26
51
64
21 21
48
5 4
19
27
45
0
10
20
30
40
50
60
Verydisatisfied
Somewhatdisatisfied
Neutral Somewhatsatisfied
Very satisfied
Disabled Students Programs & ServicesLatino Center/AdelanteAfrican American Collegians Center/Black CollegiansTRIO/Student Support ServicesHealth Services
Percentage of Most Satisfied
24
Satisfaction with Student Services
13 13
2826
20
11 12
17
22
39
10 11
20
16
43
7 7
3229
25
64
21 21
48
0
10
20
30
40
50
60
Verydisatisfied
Somewhatdisatisfied
Neutral Somewhatsatisfied
Very satisfied
Veterans' Programs Financial AidPsychological Services Scholarship OfficeTRIO/ Student Support Services
Percentage of Least Satisfied
25
Importance of Counseling Services
0
10
20
30
40
50
60
70
80
No importance Little importance Moderate/ averageimportance
Great importance Very greatimportance
Counselors provide me with accurate information on courses and degree requirements.
Counselors help me keep my educational plan current.
Counselors refer me to other sources for assistance when necessary.
Counselors are usually available to me by appointment.
Counselors help me identify majors of study that fit my current interests.
Counselors help me identify career areas that fit my current skills/ interests.
Counselors are usually available to me through walk-in counseling.
Counselors help me identify ways to improve my study skills and habits.
By Decreasing
Level of Importance
26
Agreement Rating by Counseling Service
0
10
20
30
40
50
60
Strongly disagree Somewhatdisagree
Neutral Somewhat agree Strongly agree
Counselors are usually available to me by appointment.
Counselors provide me with accurate information on courses and degree requirements.
The counseling staff are helpful and courteous.
Counselors refer me to other sources for assistance when necessary.
Counselors help me keep my educational plan current.
Counselors help me set realistic goals to work toward.
Counselors are usually available to me through walk-in counseling.
Counselors help me identify majors of study that fit my current interests.
Counselors help me identify career areas that fit my current skills/ interests.
Counselors help me identify ways to improve my study skills and habits.
By Decreasing Agreement
Contributions of Select Student
Services to Students’ Self-Development
28
Factor Analysis for Self-Development Items (1 of 5)
Items subjected to: Principle Components Factor Analysis Varimax Rotation
Criteria for factor retention: Initial Eigenvalues > 1 Scree Plot test Item loading > .40 Theoretical justification of item
loading on factor
International students’ responses excluded
29
Self-Development Factor Analysis (2 of 5)
Extent to which experiences at Santa Monica College has contributed to students’ self-development.
(1 = very negative effect; 5 = very positive effect) Extracted Communalities
56.Understanding my career prospects .6757.Developing my self-confidence .7558.Developing a sense of personal identity .7559.Recognize my potential for success .7260.Helping me cope with change .7461.Helping me handle stress .6162.Helping me develop a personal code of values and ethics .7263.Understanding people of diverse cultures, values, and ideas
.7464.Working with groups of people .7765.Developing leadership skills .6866.Taking responsibility for my own behavior .6867.Developing time management skills .61
30
Self-Development Factor Analysis (3 of 5)
Initial Eigenvalues Extraction Sums of Squared Loadings Rotation Sums of Squared LoadingsComponent
Total % of Variance Cumulative % Total % of Variance Cumulative % Total % of Variance Cumulative %
1 7.676 63.965 63.965 7.676 63.965 63.965 4.651 38.762 38.7622 .754 6.282 70.247 .754 6.282 70.247 3.778 31.485 70.2473 .610 5.087 75.3344 .480 4.001 79.3345 .457 3.812 83.1466 .417 3.474 86.6207 .359 2.994 89.6158 .327 2.726 92.3419 .267 2.227 94.56810 .246 2.048 96.61611 .209 1.744 98.35912 .197 1.641 100.000
Total Variance Explained
31
Self-Development Factor Analysis (4 of 5)
32
COMPONENT1 2
Q56 Understanding my career prospects .785Q57 Developing my self-confidence .779 .373Q58 Developing a sense of personal identity .760 .417Q59 Recognize my potential for success .753 .393Q60 Helping me cope with change .729 .454Q61 Helping me handle stress .683 .376Q67 Developing time management skills .608 .493Q63 Understanding people of diverse cultures, values, and ideas .818
Q64 Working with groups of people .815Q65 Developing leadership skills .456 .687Q66 Taking responsibility for my own behavior .469 .677Q62 Helping me develop a personal code of values and ethics .557 .638
Self-Development Factor Analysis (5 of 5)
Rotated Component Matrix
Understanding/Management of
Selfα = .92
Developing Personal Valuesα = .90
Self-Development
Full Scaleα = .94
33
Self-Development by Counselor
Task Performance
Self-DevelopmentFull Scale
Understanding/Managing of Self
DevelopingPersonal Values
Pearson Correlation N Pearson Correlation N Pearson Correlation N
Self-Development Full Scale 1.00 1210 0.97 1210 0.94 1210Understanding/Managing of Self 0.97 1210 1.00 1239 0.82 1210Developing Personal Values 0.94 1210 0.82 1210 1.00 1253Keeping educational plan current 0.42 1051 0.42 1075 0.38 1080Setting realistic goals 0.42 1058 0.41 1081 0.38 1088Identifying career areas based on interest/ skills 0.41 1058 0.4 1082 0.38 1089Identifying majors fitting current interests 0.41 1048 0.4 1072 0.37 1079Helping improve study skills and habits 0.38 1025 0.36 1047 0.36 1053Referrals to other services 0.37 1052 0.37 1075 0.33 1083Accuracy of information on course/degree reqs. 0.36 1088 0.36 1112 0.32 1121Counseling staff curteousness 0.36 1073 0.36 1098 0.32 1104Counselor availability: appointment 0.35 1086 0.34 1111 0.3 1121Counselor availability: walk-in 0.3 1103 0.3 1127 0.26 1137Note: correlations in descending order by Full ScaleAll correlations significant at p < .001
Construct validation for the self-development scales is further enhanced by the statistically significant correlations with a variety of “key” tasks performed by counselors. Correlations are based on students’ level of agreement
with each item.
Influence of College Attendance on
Self-DevelopmentEffect of Ethnicity, Length of
Attendance, and Utilization of Student Services
35
Vectors of Development (Chickering & Reisser, 1993)
Developing Competence; Managing Emotions; Moving Through Autonomy Toward
Interdependence; Developing Mature Interpersonal
Relationships; Establishing Identity; Developing Purpose; Developing Integrity.
36
Interpreting Findings
All analyses described herein usedmultivariate analysis of covariance (MANCOVA), controlling for: Students’ Age Cumulative GPA
37
Effects of Ethnicity on Self-Development
Analyses indicated that age impacts degree of self-development to a statistically significant degree.
Significant effects for Ethnicity on:
Self-Development Full Scale Understanding/Management of
SelfSubscale
Developing Personal Values Subscale
38
Self-Development Differences by Ethnicity (1 of 2)
African American and Latino students consistently expressed statistically significant higher levels of development on the full scale and two subscales compared to Asian and White students.
Full Scale:• Asians (AS) differed from African Americans*** (AA)
& Latinos*** (LA);• White (W) from AA*** & LA***.
Understanding/Managing of Self:• AS from AA* & from LA***;• White (W) from AA**, & LA***.
Developing Personal Values:• AS & W differed from African Americans*** (AA) &
Latinos*** (LA);• White (W) from AA*** & LA***.
Note: Bonferroni Comparisons
* p < .05 ** p < .01 *** p < .001
39
Self-Development Differences by Ethnicity (2 of 2)
Semesters Self-DevelopmentFull Scale
Understanding/Managing of Self
DevelopingPersonal Values
Asian/Pacific Islander 45.82 26.80 19.02Black/African American 51.04 29.49 21.56Latino 50.12 29.29 20.84White 45.47 26.71 18.76Other 47.68 27.86 19.82
Note: Covariates appearing in the model are evaluated at the following values: AGE = 24.40; GPA = 2.8951.(ns)
40
Effects of Length of College Attendance on Self-Development
Analyses indicated that age impacts degree of self-development to a statistically significant degree.
Significant effects on: Self-Development Full Scale Understanding/Management of Self
Subscale Developing Personal Values Subscale
41
Self-Development Differences
by Length of College Attendance
First year students expressed significantly lower levels of self-development compared to 3rd year and 4th+ year students;
Findings consistent with theory (Vectors of Development; Chickering & Reisser, 1993)
Semesters Self-Development Full Scale
Understanding/ Managing of Self
Developing Personal Values
1 - 2 45.97*, ** 26.86* 19.10* 5 - 6 48.22 28.14 20.03 3 - 4 48.48* 28.45 20.03 6 + 49.33** 28.80* 20.53*
Note: Bonferroni Comparisons: * p < .05, ** p < .01
Covariates appearing in the model are evaluated at the following values: AGE = 24.40; GPA = 2.8951.(ns)
42
Effects of Utilization of Counseling Services on Self-
Development
Analyses indicated that age impacts degree of self-development to a statistically significant degree.
Significant effects on: Self-Development Full Scale Understanding/Management of Self
Subscale Developing Personal Values
Subscale
43
Effects of Utilization of Counseling Services on Self-
Development
As may be expected, the greater the number of times a student used counseling services, the greater their level of self-development.
Differences: Full Scale: 1 & 4**; 1 & 5***; 2 & 4**; 2 & 5***; 3 & 5**UMS Subscale: 1 & 4**; 1 & 5**; 2 & 4*; 2 & 5**; 3 & 5*DPV Subscale: 1 & 4**; 1 & 5***; 2 & 4*; 2 & 5**; 3 & 5*
*p < .05; **p < .01; ***p < .001
Covariates appearing in the model are evaluated at the following values: AGE = 24.40.
Self-DevelopmentFull Scale
Understanding/Managing of Self
DevelopingPersonal Values
1. Not aware of service 45.14 26.47 18.672. Aware of service but never used it 46.50 27.20 19.303. Used service once 46.79 27.27 19.514. Has used service a few times 48.38 28.30 20.085. Has used service a lot 49.78 28.95 20.84
44
Effects of Utilization of Special Programs on Self-Development
Students participating in SMC’s select “special programs” (e.g., African American Collegiate Center, Latino Center, Pico Partnership) expressed significantly higher levels of self-development than non-participants.
Findings support the premise that the very nature of their function and the added services they provide (social, cultural, affective) further assist students.
***p < .001
Covariates appearing in the model are evaluated at the following values: AGE = 24.01, GPA = 2.9126.
Semesters Self-DevelopmentFull Scale
Understanding/Managing of Self
DevelopingPersonal Values
Did Not Use Service 46.65 27.25 19.40Used Service 51.27*** 30.01*** 21.26***
45
Effects of Utilization of Special Programs on Self-Development
EOPS participants differed to a significant degree from non-EOPS participants
EOPS students required to meet with counselor three times per semester.
*p < .05; **p < .01; ***p < .001
Covariates appearing in the model are evaluated at the following values: AGE = 24.03, GPA = 2.9132.
Semesters Self-DevelopmentFull Scale
Understanding/Managing of Self
DevelopingPersonal Values
Did Not Use Service 47.00*** 27.48** 19.52***
Used Few Times 48.67 28.68 19.99*
Used a Lot 51.25*** 29.70** 21.55*, ***
46
Effects of Financial Aid on Self-Development
Analyses indicated that age impacts degree of self-development to a statistically significant degree.
Significant effects on: Self-Development Full Scale Understanding/Management of Self
Subscale Developing Personal Values
Subscale
47
Effects of Financial Aid on Self-Development
Students using financial aid services—presumably receiving financial aid—expressed significantly higher degrees of self-development, compared to students not receiving it (finding consisted with other studies).
Differences: Full Scale: 1 & 4*; 1 & 5***; 2 & 5**UMS Subscale: 1 & 5**DPV Subscale: 1 & 4*; 1 & 5***; 2 & 5***; 3 & 5*
*p < .05; **p < .01; ***p < .001
Covariates appearing in the model are evaluated at the following values: AGE = 24.01.
Self-Development Full Scale
Understanding/ Managing of Self
Developing Personal Values
1. Not aware of service 44.48 26.15 18.33 2. Aware of service but never used it 46.64 27.33 19.32 3. Used service once 46.63 27.28 19.35 4. Has used service a few times 47.80 27.85 19.94 5. Has used service a lot 49.15 28.44 20.71
48
Effects of Feeling Safe on Campus on Self-Development
Analyses indicated that age impacts degree of self-development to a statistically significant degree.
Significant effects on: Self-Development Full Scale Understanding/Management of Self
Subscale Developing Personal Values
Subscale
49
Effects of Feeling Safe on Campus on Self-Development
Students feeling safe on campus expressed significantly higher scores on self-development. Safetyis a contributing factor to a supportive learning environment.
Differences: Full Scale: 2 & 4***; 2 & 5***; 3 & 4 *; 3 & 5***; 4 & 5***UMS Subscale: 2 & 4***; 2 & 5***; 3 & 4 *; 3 & 5***; 4 & 5***DPV Subscale: 2 & 4***; 2 & 5***; 3 & 5***; 4 & 5***
*p < .05; **p < .01; ***p < .001
Covariates appearing in the model are evaluated at the following values: AGE = 23.80.
Self-Development Full Scale
Understanding/ Managing of Self
Developing Personal Values
2. Disagree 41.10 23.93 17.17 3. Neutral 43.39 25.30 18.10 4. Somewhat agree 45.14 26.40 18.74 5. Strongly agree 49.68 29.03 20.64
50
Conclusions (1 of 3)
College attendance significantly and positively impacts students’ self-development, particularly for: African Americans & Latinos; Those attending college for a longer
period of time; Students receiving counseling
services, including ethnic and SES-based programs (e.g., Latino Center, EOPS);
Students receiving financial aid assistance;
Students feeling safe on campus.
51
Conclusions (2 of 3)
Students particularly credit college attendance as positively impacting their ability to: Understand people of diverse values
and cultures; Taking responsibility for their own
behavior; Helping shape a personal identity; Enhancing their self-development; Recognizing the potential for success.
52
Conclusions (3 of 3)
There is a need for student services to move beyond simply conducting usage and satisfaction studies.
Assessment of student learning outcomes should also emphasize non-cognitive domains.
Assessment should be ongoing and embedded into student services’ ongoing evaluations—not simply program reviews.
Assessment should be used to improve service delivery.
53
Select References
Astin, A. W. (1993). What Matters in College: Four Critical Years Revisited. San Francisco: Jossey-Bass.
Chickering, A. W., & Reisser, L. (1993). Education and identity . San Francisco: Jossey-Bass.
Komives, S. R., Woodard, D. B., & Associates (2003) (4th ed.). Student Services: A Handbook for the Profession. San Francisco: Jossey-Bass.
Kuh, G., Schuh, J., Whitt, E., Andreas, R., Lyons, J., Strange, C., et al. (1991). Involving colleges: Successful approaches to fostering student learning and personal development outside the classroom. San Francisco: Jossey-Bass.
Pascarella, E., & Terenzini, P. (1991). How college affects students: Findings and insights from twenty years of research. San Francisco: Jossey-Bass.
Schuh, J. H., Upcraft, M. L., & Associates. (2001). Assessment practice in student affairs: An application manual. San Francisco: Jossey-Bass.
Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition. Chicago: The University of Chicago Press.
Upcraft, M. L., & Schuh, J. H. (1996). Assessment in student affairs: A guide for practitioners. San Francisco: Jossey-Bass.