IMAGINE THE POSSIBILITIES: THE NEW U STUDENT EXPERIENCEASSESSMENT PLAN

Introduction

The Office of Undergraduate Studies and the Office of Student Affairs launched the New U Student Experience (NUSE) campaign in the fall of 2012 in response to President David Pershing’s charge to strengthen undergraduate education and the University of Utah’s New U strategic vision. The campaign promoted a transformative educational experience that motivates students to have an impact and make a contribution to society upon graduation.

More specifically, NUSE aims to:

Revitalize the University of Utah’s undergraduate mission by creating a compelling and attractive vision for potential students and parents, enhancing the enrollment process by providing a seamless navigation through the system (recruitment, admission, financial aid/scholarships, orientation and advising), engaging students in powerful learning environments both in and outside the classroom, and assuring a rewarding, meaningful and successful experience for those who attend (p. 1, “Imagine the Possibilities: The New U Student Experience”).

The specific elements of NUSE are:

1. First-Year Learning Communities2. Support for Student Success 3. Deeply Engaged Learning Experiences4. Broad and Deep Knowledge5. Capstone or Culminating Experiences

Assessment Methods Because of the complexity of this approach and its span across many programs and offices, assessing the impact of NUSE requires the examination of data from many different programs and activities. This document will summarize quantitative and qualitative data that together tell the story of students’ participation in NUSE and its impact on their educational experience. This document will also be updated as programs are changed or added.

Quantitative MeasuresQuantitative measures will include:

Existing Institutional Data such as retention and graduation rates Specific questions from the University of Utah’s Graduating

Student Survey Questions from the National Survey of Student Engagement

(NSSE) Data on participation in the various Undergraduate Studies and

NUSE programs mentioned in this plan.

Qualitative Measures Focus groups and learning portfolios will capture student and faculty feedback and investigate specific questions about the impact of NUSE and the achievement of learning outcomes.

1. Develop a learning portfolio (integrated with our learning management system) that will be used to measure the achievement of General Education learning outcomes through examples of student work that are reviewed using the VALUE rubrics - which were specifically developed by AAC&U to assess the Essential Learning Outcomes framework. The portfolio will also ask students to reflect on their work and describe in their words how they think they achieved the learning outcomes through that particular activity.

2. Create a cohort of students in a unique yearlong opportunity to provide feedback about NUSE programs and initiatives from the student vantage point. This group, the New U Student Experience Cohort (NUSEC) will consist of 15-20 students, ideally representing diversity of race, gender, income, geographic location, prior collegiate schooling (transfer and non-transfer), admissions index/5% group, athletes, etc. NUSEC students will provide regular feedback about their day-to-day experience at the University. The NUSE assessment team will conduct student surveys and prompt students with specific questions on a regular basis but also allow them open-ended opportunities to observe and point out their significant experiences on campus.

Research QuestionsThe assessment of NUSE will address the following questions:

1. What programs/services/opportunities discussed in NUSE were created and made available to students?

2

2. To what degree did students participate in the NUSE programs and opportunities?

3. To what extent did students benefit from NUSE programs and opportunities?

4. What are the fall-to-fall retention rates for the University for each year that NUSE has been in place?

5. What are the four and six-year graduation rates for the student cohorts impacted by NUSE?

These research questions, and the data used to answer them, will help evaluators and program administrators:

track the accomplishment of goals and the development of new programs

understand how students learn about, participate in, and are impacted by NUSE programs

examine the impact of NUSE programs through the lens of students, institutional data, and the Essential Learning Outcomes

improve the design and delivery of these programs in the future.

Assessment Plan Organization - This assessment plan is organized into five sections – one for each element of the plan. Within each section is a description of the programs or initiatives that contribute to that element of the campaign, specific goals for those programs, and metrics that are used to measure success. Many of the metrics will overlap, as we expect that many of these approaches will help achieve multiple outcomes.

Results – As we gather results that can help inform this plan, they will be posted on the New U Student Experience Assessment Plan web site: http://ugs.utah.edu/new-u/NUSE-assessment-plan.php.

I. FIRST-YEAR LEARNING COMMUNITIES

Programs: The University of Utah has been operating some form of a learning community for over 25 years. The impact that belonging to a community has on students’ retention, progression, and achievement is significant. Starting with the LEAP program over 25 years ago and continuing now with learning communities throughout campus, Undergraduate Studies believes that every student having a learning community experience during their first year is paramount among our goals.

Block U - The Block U program bundles general education courses and provides support for student success and integrated

3

learning in students’ first year at the U.  BlockU students participate in a learning community, learn with excellent teachers, have the support of peer mentors, advisors, librarians and student success advocates, and engage in a meaningful capstone project.

Diversity Scholars - Diversity Scholars (DS) and Transfer Diversity Scholars (TDS), are geared to serve underrepresented students and students of color. Diversity Scholars enroll in a two-semester course that connects students with mentoring opportunities at a local elementary and middle school.

Honors Living and Learning – Provides a mandatory advising program within each students’ first year at the U and thereafter until they have selected a major.

LEAP – The Learning, Engagement, Achievement, and Progress (LEAP) program at the University is a two-semester cohort program that meets three curricular requirements (two general education and one bachelor degree requirement). Students stay with the same classmates, instructor, and peer adviser and they can choose from a variety of disciplinary LEAP sections – Fine Arts, Engineering, Architecture, Business, etc…-- in order to get some intellectual exposure to a field they may be interested in.

New University Scholars - New University Scholars is a two-year program. It consists of a community of like-minded students navigating General Education together. New University Scholars take part in a learning community pathway in their first year. In addition, they take part in monthly dinners and events and have access to visiting scholars to the University. In the second year, the main focus is on leadership and students have the opportunity to take part in the ULEAD program. New University Scholars receive a $2,000 scholarship for the school year.

Definitions: First year learning communities are experiences where students who are in their first year at the University – whether they are freshman straight from high school or transferring from another institution – proceed through a curriculum of more than one course for more than one semester with the same cohort of classmates.

Metrics: We expect students who participate in first-year learning communities will be more likely to be retained to their second year and will also graduate in less time than students who do not participate in first-year learning communities. We will examine these metrics as part of our overall assessment of retention and graduation rates for all MUSE programs.

4

II. SUPPORT FOR STUDENT SUCCESS

Programs: All of the elements of NUSE are designed to help students succeed, and that is the main focus of this assessment plan – measuring to what degree NUSE programs help students be successful. This section will show the definitions, goals, and metrics of two areas of student success:

Overall student success: retention, graduation and students’ perception that they have had a meaningful experience

Specific goals related to the University of Utah’s new Student Success and Empowerment Initiative. This initiative, launched in Fall 2012, empowers students to identify direction, engage with high impact activities/services, and accomplish goals.

1. Overall Student Success and Engagement

Definition: For assessment purposes overall student success is defined in three ways: student retention to the second year, timely graduation, and having a meaningful experience at the University.

Metrics: “Retention” is measured by the percentage of students who return for the fall semester after their first full year of college.

“Timely graduation” is the amount of time it takes students to graduate and one metric that is used nationally to compare institutions is a six-year graduation rate. Our goal is to decrease the amount of time it takes students to graduate. In 2011, the University, through the Utah System of Higher Education (USHE), committed to specific retention and graduation goals for the next five years:

1. An increase in the one-year retention rate for first time freshmen to 88%

2. An increase in six-year graduation rates for all cohorts to 60%.

Both retention and six-year graduation rate will be updated and published on the NUSE Assessment Plan web site: http://ugs.utah.edu/new-u/NUSE-assessment-plan.php.

“Meaningful Experience” will be measured using questions from the University of Utah’s Graduating Student Survey and the National Survey of Student Engagement (NSSE). Students’ responses will be tracked over time as well as compared to those in other institutions responding to the survey. Here is an example of some of the questions and statements that students are asked to react to in those surveys:

5

If you could start over, would you choose to attend the U?Does the institution emphasize studying and academic work?Do you discuss ideas from readings or classes with others outside of class?Are students actively involved in their learning, individually and working with others?How challenging is your institution’s intellectual and creative work?Do students work with faculty members inside and outside the classroom?Do students feel the institution is committed to their success?How often did you work harder than you thought you could to meet an instructor's standards or expectations?

NSSE data will allow the assessment team to track changes in these important metrics from year to year as well as compare University of Utah students’ responses to several different comparison groups across the nation. NUSE’s goal is to significantly improve the percentage of students who agree with these statements over the next five years. A chi-square statistical test will be used to compare the percentages over that period of time.

Definition 2: “Engagement” is defined as student participation in activities on campus that take place outside the classroom – clubs, student government, service projects, and high impact programs such as research, study abroad, and so forth.

Metrics: Engagement will be measured by examining the questions on the Graduating Student survey and the NSSE that are relevant to engagement. Some of these questions include:

Can you identify three faculty who could give you a reference? Did you participate in a learning community? Did you participate in a practicum, internship, field experience,

co-op experience, or clinical assignment?

NUSE’s goal is to significantly improve the percentage of students who agree with these statements over the next five years. A chi-square statistical test will be used to compare the percentages over that period of time.

2. The Student Success and Empowerment Initiative (SSEI)

The goals of SSEI are as follows:

6

1) Promote student success through identifying short and long-term goals that include degree completion.

2) Identify high-impact educational experiences that result in a degree that is unique to the student.

3) Introduce planning tools and strategies that foster action in accomplishing student goals but respect flexibility as students experience intellectual development.

4) Encourage utilization of campus and community services that enhance the undergraduate experience and empower students to achieve excellence.

5) Engage students in using the “4 Year Plan” to create a holistic educational experience.

To measure progress on these goals, the SSEI plans to utilize the following assessments:

1) Quantitative data that tracks student use of our servicesa. Number of initial student contacts with Student Success

Advocates (SSAs) across campus b. Number of follow-up contacts initiated by SSAsc. Number of follow-up meetings between students and SSAsd. Annual survey of students in our contact list to determine

the following:i. Satisfaction with SSEI services

ii. Involvement in high impact educational experiences resulting from contact with SSAs

iii. Utilization of campus services and resources resulting from contact with SSAs

iv. Progress on degree completion2) Qualitative assessment of processes and meaningful experiences

a. Annual program assessment involving a systematic examination of processes within the SSEI to determine the effectiveness and experience of offering our various services

b. Open-ended survey questions on the survey described in item 1.d that ask students about their experience of interacting with SSEI staff and processes

c. Annual focus group with students who have interacted with our initiative to gather more in-depth information regarding how the SSEI impacted their overall experience at the University.

7

III. DEEPLY ENGAGED LEARNING EXPERIENCES

Deeply engaged learning experiences or high impact programs (HIP) are a key element in NUSE. They are the experiences through which students seek out new ways of learning, explore interests, engage in new conversations, and are transformed. They typically happen outside of traditional classroom settings and in the context of the real world off campus, the research laboratory, or in community based research.

Programs: On the University of Utah campus, one category of deeply-engaged or high impact programs are MUSE programs. The goal of the MUSE Project is to assure that all students have the option to participate in a signature experience during their education, which could include: studying in another country for a semester, engaging in an internship with a senator in Washington, DC, conducting research in a genetics lab, or volunteering for a non-profit organization to build a house. The MUSE program advocates for, coordinates, and provides information to students about these kinds of experiences.

More importantly, MUSE hosts the “theme year” each year to engage students, faculty and staff across campus in a discussion of a common book, gather together for a campuswide, large-scale lecture, MUSE nights out or dinners with presenters, faculty and students, and faculty lectures and microgrants. MUSE promotes intimate, reflective and deep learning.

Definition: These programs are called “high impact” because of the research on student success which shows that students who are meaningfully engaged in co-curricular experiences are more likely to remain enrolled in school and graduate in a timely fashion (some of these programs are also listed under first-year learning communities). The high impact programs that are part of NUSE include:

Bachelor of University Studies (design your own major) Bennion Community Service Center Block U General Education Cohorts (pre-scheduled, thematic

groupings of courses to meet General Education requirements) Honors College (Living and Learning residence hall programs,

Praxis Labs, Service Programs) Integrated Minors Learning Abroad Learning, Engagement, Achievement, and Progress (LEAP:

first-year and multiple-year experience cohort programs) MUSE Project theme year events or participation in MUSE

Scholars

8

National Student Exchange Office of Outreach and Engagement Student Success and Empowerment Initiative (SSEI) Undergraduate Research Opportunities Program (UROP) University College Advising (Advising and Major Exploration)

Metrics: As mentioned above, these high impact programs are expected to increase students’ engagement, attachment to the University, and sense of belonging, which we believe will translate into higher retention and graduation rates. The assessment of these outcomes, as has already been described in earlier sections, will be measured by questions from NSSE on engagement, retention and graduation rates, and questions on satisfaction, pride, and sense of belonging from the Graduating Student Survey. Direct qualitative evidence from students will be gathered from a cohort of students called the New U Student Experience Council (NUSEC), which was formed in the fall of 2012 to provide the NUSE Assessment Team with feedback about its programs and was in existence for two years.

IV. BROAD AND DEEP KNOWLEDGE

Broad and deep knowledge refers to the General Education curriculum and the way that it connects with and supports curricula in the majors. The NUSE campaign promises that students will participate in powerful learning environments and efforts to re-imagine General Education, including the way that it connects to majors, and is a vital part of the NUSE.

Re-Imagining General Education began in the Fall of 2011 with the appointment of a Vice-Presidential Task Force. The goal of the Task Force was to offer a set of recommendations about the content and structure of a General Education Program that will prepare students for meaningful participation in a 21st Century world.

The work of the Task Force began with two Town Hall meetings with over 200 faculty, staff and students in attendance and a 90-minute meeting with 45 undergraduate students. Input from those meetings was supplemented by broad reading of General Education reform efforts at other universities, review of best practices in General Education, and focused study of our current General Education Program. The Task Force met for the entire 2011-12 academic year at the end of which a preliminary statement was composed. That preliminary statement is now being used to structure a series of faculty focus groups throughout the campus community.

9

Programs. Two new General Education programs have emerged from the NUSE campaign. Both were being piloted in the Spring of 2013.

Definitions:

Block U Program. Offers students a thematic way to navigate General Education and is designed with many student success best practices: set schedule of 15 credit hours, book scholarship, learning community, peer mentors and student success advocates, and a capstone experience with an applied learning experience. We developed curriculum in the Fall and Spring of 2012-13 and piloted three Block Us in the Fall of 2013: Sustainability, Creativity and Community, Global Citizenship.

Integrated minors. Integrated minors take advantage of changes in intellectual development by offering student choice in the selection and pacing of their General Education coursework. Integrated minors are also flexible and well matched with more highly structured majors like Engineering and Business. Like the Block U, Integrated minors offer students a thematic way to navigate General Education and also include many student success best practices, including learning communities, peer mentors and peer advisors, MUSE experiences, and capstone courses.

Metrics. Progress will be assessed on three target goals.

1. Enrollment and retention: We will track enrollment growth in each program from year 1 to year 2 and track retention of students who participate in one of these programs as compared to students who do not participate in the program.

2. Student Satisfaction: Data from our Student Course Feedback instrument (delivered to every student at the end of every course) will allow us to track student satisfaction with the experience.

3. Essential Learning Outcomes: Once a learning portfolio product is tested and implemented, contributions to learning portfolios will provide assessment of progress on targeted learning outcomes in learning community courses. In the meantime, the Office of Undergraduate Studies, which oversees General Education is beginning to pilot test the use of the rubrics with artifacts of student work that are being voluntarily submitted by General Education faculty. Read more about this effort below.

10

In 2008, the Undergraduate Council selected the American Association of Colleges and Universities’ (AAC&U) 15 Essential Learning Outcomes (ELO) as the outcomes for the University of Utah’s General Education program. AAC&U designed the ELO’s as qualities that students, as they leave higher education, should have in order to be engaged citizens of the 21st century. The learning outcomes are grouped into four areas1:

A. Knowledge of Human Cultures and the Physical and Natural World

1.  Knowledge of human cultures and the physical and natural world through study in the sciences and mathematics, social sciences, humanities, histories, languages, and the arts.

B. Intellectual and Practical Skills 2.  Inquiry and analysis3.  Critical thinking4.  Creative thinking5.  Written communication6.  Oral communication7.  Quantitative literacy8.  Information literacy9.  Teamwork10. Problem solving

C. Personal and Social Responsibility 11. Civic knowledge and engagement12. Intercultural knowledge and competence13. Ethical reasoning and action14. Foundations and skills for lifelong learning

D. Integrative Learning 15. Integrative learning, including synthesis and advanced accomplishment across general and specialized studies

Metrics: The assessment of ELO’s will provide a lens through which Undergraduate Studies views the impact of its programs on these important outcomes. AAC&U has developed a set of rubrics to measure these outcomes, which will be used in the assessment. These are available on the http://ugs.utah.edu/gen-ed-reqs/outcomes.php web page.

Baseline data will be established in the first year of the study. The new General Education review process asks departments/faculty as they apply for General Education designations to pick several outcomes that they expect students in the course to accomplish, and then identify the assignment(s) from their syllabus that will be used to measure the achievement of those outcomes.

1 A full description of the Essential Learning Outcomes can be found in Appendix B

11

A pilot test of a process to collect assignments and assess them with the appropriate AAC&U rubric was done in Spring 2013. The pilot asked faculty of General Education courses to submit examples of student work on assignments identified as meeting the learning outcomes selected for the course. Faculty participation was voluntary. The Office of Undergraduate Studies paid and trained a team of faculty to use the AAC&U VALUE rubrics to assess these student artifacts on the achievement of the learning outcomes,. This pilot test will also include a formative assessment that examines the design of this assessment process, an examination of interrater reliability and the rubrics, and the data produced.

The final design of an overall assessment of the ELO’s in General Education will be implemented after a successful pilot and the accomplishment of suitable interrater reliability among the reviewers on the assessment team. The overall assessment will require that a process be implemented that collects an example of student work on each assignment in each General Education course that is identified as meeting the ELO’s selected for that course. This process will be phased in over a number of years, aligned with the ongoing five-year review that is done for every one of the approximately 900 courses that carries a General Education designation. After assignments begin to be collected from each course, a meaningful sample of assignments from two learning outcomes will be pulled and assessed each year with the appropriate rubrics.

Once the full assessment is implemented with artifact collection, benchmark scores will be created with the rubrics (which generate scores ranging from 0 to 4). Subsequent cohorts will be compared to this benchmark. Because the rubric scale is ordinal and the purpose of the rubrics is largely qualitative, statistically significant improvements on the rubric scores is not one of the goals. However, the distribution of scores achieved each year (the percentage of scores assigned to each of the 0-4 ratings) will be used as discussion points in the larger conversation about the achievement of the learning outcomes over time. Overall, data from the assessment of General Education learning outcomes and from learning portfolios examined in the Block U pilot or other areas, will be used in the University’s ongoing discussion of the importance and achievement of Essential Learning Outcomes in the student experience. General Education is a great place to start with this examination because it literally affects all students. If certain outcomes are not accomplished or addressed at the level that the Office of Undergraduate Studies and the assessment team thinks is

12

appropriate, it will use information from courses and assignments that were successful to promote best practices in achieving outcomes. UGS will also conduct trainings or seminars for General Education faculty that are deemed appropriate to help faculty help students to improve the achievement of the outcomes.

V. CAPSTONE EXPERIENCES

Programs:

The Capstone Initiative

The culminating experience of the New U Student Experience, the capstone, asks students to integrate the learning they have experienced in General Education and study in the major and through deeply engaged learning experiences in a project or experience that anticipates their future life after graduation and that has a real world application.

Capstone’s Make it Real program might be an individual student project, a collaborative project, or a project that occurs in the context of a capstone class. Capstones can occur as a culminating project in a major like the Political Science senior seminar project or the integrated design project in the School of Architecture, or, capstones of more general integrative nature can be hosted by the Capstone Initiative office. Students can apply for mini-grants of between $1,000 and $5,000 to help their projects materialize.

Metrics: Assessment of learning outcomes in capstone projects will be completed through a variety of different approaches. First, graduation rates will be studied for students participating in capstone projects. While students will typically complete their capstones toward the end of their time at the University, being able to show that those completing Capstones are not taking longer than others will be important. The Graduating Senior Survey will be used to assess the impact of capstones on students’ perception of their integration of knowledge, sense of belonging, and their commitment to community engagement. During the academic year, 2015-16, the UGS Engage Portfolio faculty team will develop a systematic approach for a direct method of assessment of learning that occurs in deeply engaged learning experiences, including capstones.

13

APPENDIX A

Table 1: The New U Student Experience Assessment Plan Timeline: Academic Year 2012-13

ACTIVITYSeptember 2012

October November December January 2013

February March April May

Block U The Block U and Integrated Minor Cohorts: Program Design.

Block U and Integrated Minors Curriculum Development and Marketing for Fall 2013 launchIntegrated

MinorsNUSEC Focus Group

Bi-Monthly (Dec 3, 2013)

NSSE Bi-annualE-Portfolios E-portfolios platforms researched, planned, and designed. Students will populate their e-porfolios throughout their Block U

experience, starting in Spring 2013 and continuing through Fall 2013. Review of portfolios will happen at the end of each semester.

Gen Ed Learning Outcome Rubric Study

Instructors solicited to submit student assignments from fall and springs semester courses. A pilot rubric evaluation will begin in Summer 2013 with a faculty committee.

Retention & Graduation

Annual Institutional Data Report

NUSEC=New U Student Experience CouncilNSSE=National Survey of Student Engagement

APPENDIX B – LEARNING OUTCOMES In Fall 2008, the University of Utah’s Undergraduate Council adopted the Essential Learning Outcomes (ELO) Framework for the U’s General Education program. Beginning in Fall 2012, the Office of Undergraduate Studies (UGS) is aligning all of its programs under the ELO’s. The American Association of Colleges and Universities (AAC&U) developed this framework with the input of hundreds of educators from around the country, including representatives from our campus and other PAC-10 schools. Schools in the national Voluntary System of Accountability (VSA), which the University of Utah is a member of, now accept this framework as an effective way to meet the VSA’s learning outcome requirement.

ESSENTIAL LEARNING OUTCOMES

These learning outcomes are intended to develop students who are informed and prepared citizens of the 21st Century. They include:

A. Knowledge of Human Cultures and the Physical and Natural World

1.  Knowledge of human cultures and the physical and natural world through study in the sciences and mathematics, social sciences, humanities, histories, languages, and the arts.

Focused by engagement with big questions, both contemporary and enduring

B. Intellectual and Practical Skills (click to see rubrics for measuring each outcome), including

2.  Inquiry and analysis 3.  Critical thinking 4.  Creative thinking 5.  Written communication 6.  Oral communication 7.  Quantitative literacy 8.  Information literacy 9.   Teamwork 10. Problem solving

Practiced extensively, across the curriculum, in the context of progressively more challenging problems, projects, and standards for performance

C. Personal and Social Responsibility, including

11. Civic knowledge and engagement 12. Intercultural knowledge and competence 13. Ethical reasoning and action 14. Foundations and skills for lifelong learning

Anchored through active involvement with diverse communities and real-world challenges

D. Integrative Learning

15. Integrative learning, including synthesis and advanced accomplishment across general and specialized studies

Demonstrated through the application of knowledge, skills, and responsibilities to new settings and complex problems

The University of Utah’s Undergraduate Council selected these fifteen learning outcomes for its General Education Program in the Fall of 2008. Moreover, these outcomes should form the core of a student’s entire University experience.

16