Upload
bailey-howcroft
View
214
Download
0
Embed Size (px)
Citation preview
Student Engagement and Mission Student Engagement and Mission Effectiveness: Effectiveness: Looking for Links between Looking for Links between
Outcomes and Institutional GoalsOutcomes and Institutional Goals
Ellen M. Boylan, Ph.D.Association for Institutional Research
2005 Forum
San Diego, California • May 29 – June 1
22
Learning Environment Institutional
Mission, Goals and Objectives
Student Engagement
HIGHER EDUCATION
ENVIRONMENT
Catholic colleges
BackgroundBackground
Sadlak (1986) asserts that the institutional mission statement influences student achievement…it describes the desired academic and social development skills to be possessed at end of the educational experience.
How does an institution measure the effect of its mission on students’ educational experience? Can it be measured? Do students’ perceptions of mission differ by institution?
33
Research QuestionsResearch Questions
1. Do student ratings of engagement and learning environment differ by student perceptions of institutional mission?
(a) Are the Research Mission Questions reliable and valid measures of student engagement and perception of learning environment?
(b) What is the relationship between the Research Mission Questions and NSSE benchmark scores for Consortium schools?
2. Do students and administrators share the same perceptions of institutional mission?
44
Study DesignStudy Design Select the National Survey of Student Engagement (NSSE) 2004 instrument (The College Student Report) to test
first year and senior students
Assemble an NSSE 2004 Catholic College
Consortium for mission research
Develop Research Mission Questions
Administer the NSSE 2004 with added Research Mission Questions to Consortium students
Administer a sub-set of Research Mission Questions to Consortium college administrators
Analyze results: factor analysis, correlation and regression, and t-test analysis of means
Marywood University Institutional Review Board Approval 2004-041
55
Sample Mission StatementSample Mission Statement Concepts common to the Consortium Colleges’ mission Concepts common to the Consortium Colleges’ mission
statements were drawn out, reduced, and formulated into 20 statements were drawn out, reduced, and formulated into 20 Research Mission QuestionsResearch Mission Questions
A Catholic university sponsored by the Congregation of the Sisters, Servants of the Immaculate Heart of Mary, Marywood University roots itself in the principle of justice and a belief that education empowers people. Enacting its ideals, Marywood offers students a welcoming and supportive community that encourages men and women of all backgrounds to shape their lives as leaders in serviceleaders in service to others. Proud of its liberal arts tradition and host of professional disciplinesprofessional disciplines, Marywood challenges students to broaden their understanding of global issues and to make decisions based on spiritual, spiritual, ethical, and religious valuesethical, and religious values. Marywood calls upon students to seek their full potential and invites all to engage in a lifelong process of learninglearning. Witnessing the efficacy of teaching and scholarship, Marywood educates students to live responsibly in a diverse and diverse and interdependent world.interdependent world.
66
Professional Development
PLeadership
L
AcademicExcellence
A
Community/Service
C
Ethical/Spiritual
DevelopmentE
DiversityD
The joy of learning is faithfully nurtured...power of learning.
Higher morals, and social conscience are valuable
develop...personal competencies
Collaborativeenvironment
Commitment to build a just and peaceful society
Develop moral character
Promotes leadership to local and boarded communities
Promotes service to local and boarded communities
Emphasizes on the growth of the human person
development of leadership
community of learning
spirit of servicepromotes just and ethical relations; values
diverse ages, backgrounds, and cultures
preparation for responsible living and meaningful work
Responsible academic freedom;provides intellectual preparation
Promote moral, spiritual, religious vales to students; moral prep
Imagination and spirit
intellect and spirit
academicexcellence
Service and leadershipcommunity
Imagination and Spirit
Prepare forleadership
Develop intellectually
Lives of serviceDevelop spirit & heart; values-based education
Prepare forprofessional careers
Intellect and spirit
commitment to serving others through a liberal arts education
Service opportunities in the community
diversity
professional disciplines
education empowers
calls upon students to reach their full potential…lifelong learning
leaders in servicespiritual, ethical, and religious values
live responsibly in a diverse and interdependent world
Career preparationResponsibility for decisions
Intellectual inquiryCommitment to service
personal growth diversity
Mission Statement Concepts - NSSE 2004 Catholic College Consortium
77
AdministrationAdministration
The NSSE Survey with attached Research Mission Questions was administered to 3,827 first-year and senior students at 14 Catholic Colleges in the United States in Spring 2004.
88
Institution DemographicsInstitution Demographics
Gender of students 2 single-sex institutions Percent female – 61 to 85
Size of Institutions Large - 1 Small - 2 Medium - 12
Location of Institutions Urban – 5 Suburban - 9
99
Characteristics of Characteristics of InstitutionsInstitutions
GenderGender
TOTAL %
INSTITUTION M F A 24 76 B 20 80 C 27 73 D 15 85 E 46 54 F 23 77 G 3 97 H 33 67 I 29 71 J 19 81 K 39 61 L 27 73 M 18 82 N 0 100
Total 26 74
1010
I nstitution
% Roman Catholic
Urban/ Suburban
Small
G 64 S
N 42 U
Medium
A 36 U B 51 S C 47 U D 51 S E 60 S F 35 S I 93 S J 61 S K 68 S L 36 S M 42 U
Large
H 54 U
Characteristics of Characteristics of InstitutionsInstitutions
Size and locationSize and location
1111
Research Question 1a.Research Question 1a.
Are the Research Mission Questions reliable and
valid measures of student engagement and perception of
learning environment?
1212
Reliability and factor analysisReliability and factor analysis
Reliability analysis of Research Mission Questions (20 items). Three items removed. Mission Perception Inventory (MPI) is produced (17 items). Factor analysis of MPI conducted. Four subscales produced: Sense of Mission Respect for Diversity
Individual Actions
Religious Practice
1313
Mission
Perception
Inventory(MPI)
(17 items)
Senseof
Mission(8 items)
Respect for
Diversity(5 items)
IndividualActions(2 items)
ReligiousPractice(2 items)
Research
Mission
Questions
(20 items)
Derivation of Mission Perception Inventory Derivation of Mission Perception Inventory (MPI) and four subscales(MPI) and four subscales
from Research Mission Questionsfrom Research Mission Questions
1414
Sense of Mission ScaleSense of Mission Scale (8 items (8 items = .867) = .867)
The mission is widely understood by students
Social and personal development is important
Ethical and spiritual development is important
This institution’s religious heritage is evident
Professors here discuss the ethical implications of what is being studied
This institution offers opportunities for volunteering and community service developing leadership skills students to strengthen their religious
commitment
1515
Respect for Diversity ScaleRespect for Diversity Scale (5 items (5 items = = .854) .854)
The faculty, staff, and students here respect different religions respect different races, cultures
Students free to express their individual spirituality
Different sexual orientations are accepted
The environment encourages appreciation of diversity
1616
Rotated Factor Matrix
1Sense ofMission
2Respect
for Diversity
3IndividualActions
4ReligiousPractice
Social and personal development of students is an important part of the mission at this institution.
.719 .266 .129 .049
Ethical and spiritual development of students is an important part of the mission at this institution.
.710 .181 .066 .169
This institution offers opportunities for: Volunteering and community service
.674 .141 .149 .030
Developing leadership skills. .656 .271 .160 .000
At this institution, there are opportunities for students to strengthen their religious commitment.
.604 .296 .029 .131
The heritage of the founding religious community… is evident here. .601 .201 .032 .078
The mission of this institution is widely understood by students. .515 .189 .089 .053
The professors at this institution discuss the ethical implications of what is being studied.
.477 .329 .111 .141
The faculty, staff, and students here are respectful of people of different religions. .290 .786 .029 .064
The faculty, staff, and students here are respectful of people of different races and cultures.
.361 .747 .012 .104
Students feel free to express their individual spirituality here. .289 .656 .064 .051
People of different sexual orientations are accepted socially here. .119 .621 .070 -.081
The environment here encourages students to develop an appreciation of diversity.
.293 .616 .067 -.005
How important is it to you that you accomplish the following: Volunteering in community service.
.185 .043 .693 .184
Influencing social values. .106 .081 .649 .118
Within the past week… participated in a religious service? .103 .001 .111 .660
Within the past week…spent time in private prayer or meditation? .099 .024 .153 .619
1717
Research Question 1b.Research Question 1b.
What is the relationship between the Research Mission Questions
and NSSE benchmark scores for Consortium schools?
1818
NSSE Benchmark ScalesNSSE Benchmark Scales
Level of Academic Challenge
Active and Collaborative Learning
Enriching Educational Experiences
Supportive Campus Environment
Student-Faculty Interactions
1919
Scale N
Items Cronbach
MPI 17 .880
Sense of Mission 8 .867
Respect for Diversity 5 .854
NSSE
Level of Academic Challenge (LAC) 11 .717
Active & Collaborative Learning (ACL) 7 .743
Enriching Educational Experiences (EEE) 11 .611
Supportive Campus Environment (SCE) 6 .761
Student Faculty Interaction (SFI) 6 .719
NSSE Beliefs & Values (NSSE/BV) 12 .807
Reliability of Scales Consortium data
2020
Scale MPI
Sense of
Mission
Respect for
Diversity LAC1 ACL2 EEE3 SCE4 SFI5
MPI
-Sense of Mission .768
-Respect for Diversity .712 .581
NSSE
1Level of Academic Challenge .308 .274 .187
2Active & Collaborative Learning .297 .224 .167 .481
3Enriching Educational Experiences .329 .305 .150 .412 .489 4Supportive Campus Environment .507 .526 .462 .272 .264 .277 5Student-Faculty Interactions .307 .282 .173 .460 .584 .549 .331
NSSE/Beliefs & Values .611 .551 .401 .455 .481 .659 .541 .509
Significant Correlations by Consortium Significant Correlations by Consortium
ScaleScale
***All correlations are significant at p<.001 level, 2-tailed, df=3,825.
2121
Institution Scale Correlations by LevelInstitution Scale Correlations by Level MPI Scales with NSSE ScalesMPI Scales with NSSE Scales::
Correlation is significant at the * p<.05 level (2-tailed), df=12; **p <.02 level (2-tailed), df=12;
***p<.01 level, 2- tailed, df=12; ****p<.001 level, 2-tailed, df=12,
Scale LAC ACL EEE SCE SFI
NSSE/ BV
Mission
First .21 .09 .63**
.84**** .31 .83***
Senior .01 -.01 .61** .64 .34 .82***
Diversity
First .10 .25 .66** .28 .70*** .50
Senior -.28 .10 -.32 .06 -.43 .04
MPI
First .19 .19 .61** .85**** .31 .84****
Senior -.18 .04 .36 .59* .02 .85****
2222
Research Question 1Research Question 1
Do student ratings of engagement and learning environment
differ by student perceptions of institutional mission?
2323
Procedure for regression analysisProcedure for regression analysis
Conduct regression analysis using Sense of Mission and Respect for Diversity as dependent variables, entering all NSSE scales as a group.
Check regression coefficients for significance.
Repeat analysis employing stepwise regression, entering scale(s) with significant beta as indicated.
2424
Standardized Regression Coefficients: Standardized Regression Coefficients: Sense of Mission and Student Class Level by NSSE Sense of Mission and Student Class Level by NSSE
ScalesScales Scale
β
t
p FirstYear .103 4.51 .000 Level of Academic Challenge
Senior .074 3.07 .002 FirstYear .010 .39 N.S. Active & Collaborative
Learning Senior -.046 -1.82 N.S. FirstYear .111 4.75 .000 Enriching Educational
Experiences Senior .163 6.44 .000 FirstYear .464 21.85 .000 Supportive Campus
Environment Senior .452 21.04 .000 FirstYear -.012 -.48 N.S. Student-Faculty Interactions
Senior .066 2.43 N.S.
2525
Model Summary – Sense of Mission
Regression analysis
Change Statistics
R Square Change F Change df1 df2
Sig. F Change
First Year Supportive Campus Environment .276 711.00 1 1866 .000
Supportive Campus Environment + Level of Academic Challenge + Enriching Educational Experiences
.027 35.76 2 1864 .000
Senior Supportive Campus Environment .278 660.11 1 1711 .000 Supportive Campus Environment
+ Enriching Educational Experiences + Level of Academic Challenge
.044 55.01 2 1709 .000
2626
Standardized Regression Coefficients: Standardized Regression Coefficients: Respect for Diversity and Student Class Level by NSSE Respect for Diversity and Student Class Level by NSSE
Scales Scales
Scale
β
t
p FirstYear .070 2.91 .004 Level of Academic Challenge
Senior .049 1.90 .058 FirstYear .027 1.00 N.S Active & Collaborative
Learning Senior .045 1.62 N.S FirstYear .017 .69 N.S. Enriching Educational
Experiences Senior -.019 -.70 N.S. FirstYear .436 19.43 .000 Supportive Campus
Environment Senior .442 19.04 .000 FirstYear -.013 -.47 N.S. Student-Faculty Interactions
Senior -.016 -.53 N.S.
2727
Regression analysis
Model Summary – Respect for Diversity
Change Statistics
R Square Change F Change df1 df2
Sig. F Change
First year Supportive Campus Environment .215 511.46 1 1862 .000
Supportive Campus Environment + Level of Academic Challenge .006 14.85 1 1861 .000
Senior Supportive Campus Environment .209 451.25 1 1710 .000 Supportive Campus Environment
+ Level of Academic Challenge .003 5.96 1 1709 .015
2828
Procedure forProcedure for post hoc post hoc analysis analysis
Institution scale correlations suggest differences between institutions may exist.
Explore this by comparing means on Sense of Mission and Respect for Diversity by level and institution.
2929
Post hocPost hoc - Mean Sense of Mission scores - Mean Sense of Mission scores
by institution and student levelby institution and student level
Sense of Mission
3.6
3.8
4.0
4.2
4.4
4.6
4.8
A B C D E F G H I J K L M N
Institution FY Senior
Mean
3030
Post hocPost hoc - Mean Respect for Diversity scores - Mean Respect for Diversity scoresby institution and student levelby institution and student level
Respect for Diversity
3.2
3.4
3.6
3.8
4.0
4.2
4.4
A B C D E F G H I J K L M N
Institution FY Senior
FY Mean
Sr Mean
3131
Research Question 2 Research Question 2
Do students and administrators share
the same perceptions of institutional mission?
3232
Student and administrator means Student and administrator means by scale and itemby scale and item
FYN=1868
SRN=1705
Admin.N=60 T* Sig.*
SENSEOF MISSION 4.03 4.01 4.28 3.36 .001
RESPECT FOR DIVERSITY
4.01 3.95 3.95 0.05 N.S.
RELIGIOUS PRACTICE 1.51 1.53 1.79 4.71 .000
*Comparison of senior and administrator means only
3333
Findings and ConclusionsFindings and Conclusions
Reliability analysis and factor analysis confirmed that the Mission Perception Inventory (MPI) is a reliable and valid measure of student perception of institutional mission ( = .85-.88).
Factor analysis of the MPI resulted in four reliable subscales, Sense of Mission, Respect for Diversity, Individual Actions, and Religious Practice.
The four subscales produced indicate the MPI reflects these concepts which are included in Consortium college mission statements.
The high reliability of subscales indicates that the MPI can be employed with confidence as measures of student mission perception in subsequent studies.
3434
Findings and ConclusionsFindings and Conclusions
The MPI, Sense of Mission, and Respect for Diversity scales correlate with the NSSE benchmark scales, particularly Supportive Campus Environment (SCE).
Respect for Diversity was significantly related (for first year students only) to SCE, Enriching Educational Experiences (EEE) and Student-Faculty Interactions (SFI) scales.
3535
Findings and ConclusionsFindings and Conclusions
Supportive Campus Environment (SCE), followed by Enriching Educational Experiences (EEE) and Level of Academic Challenge (LAC), are most closely related to Sense of Mission.
SCE contributed about 21% of the variance to the Respect for Diversity scale and was significant for both first year and senior students.
3636
Findings and ConclusionsFindings and Conclusions Post hocPost hoc
The institutions showing higher senior than first year scores on the Sense of Mission scale are medium in size, selective.
Institutions showing higher senior means compared to first year, have low to moderate percentages of Roman Catholic students, and/or are about 75% female.
The two schools with the lowest senior mean scores for Respect for Diversity have the largest percentage of students self-identifying as Roman Catholic.
3737
Findings and ConclusionsFindings and Conclusions Research Question 2Research Question 2
The mean score differences between administrators and seniors is significant for nine items.
Administrators perceiveSense of Mission conceptsand behaviors as more widelyexpressed on campus than students perceive they are.
Not unexpectedly, mean scores for administrators and students differed significantly on the Religious Practice scale.
3838
ConclusionsConclusions
Factor analysis and analysis of reliability indicate the Mission Perception Inventory (MPI) has extremely high reliability and four correlated, independent and strong factors.
There is a significant relationship between students’ Sense of Mission and Respect for Diversity and the Supportive Campus Environment they experience; and to a lesser extent, the academic challenges and enriching educational experiences provided them by higher education.
Sense of Mission is rated significantly higher by college administrators than students; however, Respect for Diversity ratings are not significantly different.
3939
For further information…
Ellen M. Boylan, Ph.D.Planning and Institutional Research
Marywood University570.348.6203