Annual Report 2015-2016: Key Activities, Accomplishments

and Staff Excellence

The mission of the Office of Assessment, Evaluation, and Research is to: (1) Provide a coordinated approach for data collection and use to foster data-informed decision-making for programs and services within Student Affairs, (2) Serve as the data resource for Student Affairs, (3) Align assessment outcomes with strategic objectives at both the divisional and departmental level, and (4) Serve as a liaison with other institutional assessment and research entities to promote a culture of assessment at the University of Utah.

The values that guide our work are: Integrity, Professionalism, and Collaboration.

Table of Contents Key Activities ................................................................................................................................................................................................ ............. 1

Strategic Planning ....................................................................................................................................................................................... 3

Assessment ................................................................................................................................................................ .................................... 7

Research ................................................................................................................................................................................................ ...... 21

Reporting ................................................................................................................................................................................................ ..... 21

Plans for the Future .............................................................................................................................................................................................. 24

Anticipated Challenges ................................................................................................................................................................ ........... 24

Anticipated Opportunities ................................................................................................................................................................ .... 25

Staff Excellence ................................................................................................................................................................ ....................................... 26

1

I. Key Activities The Office of Student Affairs Assessment, Evaluation and Research (AER) provides coordinated support for these activities within the Division of Student Affairs. During the past year the entire Division of Student Affairs has been engaged in a focused process to make greater connections between programs and services and the assessment of these activities. As part of this larger organizational mapping process, AER has outlined its key activities, goals and outcomes for AER as follows:

Key Activities Goals Outcomes

Strategic Planning

1. Facilitate ongoing strategic planning for the Division of Student Affairs to align with institutional strategic planning efforts.

2. Assist each department

within Student Affairs with strategic plan development in a manner that aligns with the Division's strategic plan.

3. Collaborate

institutionally to develop a strategic plan for research and assessment.

1.a. Departmental key activities have been developed into division-level key activities. These have been aligned with the UU Four Big Goals and are the foundation for Student Affairs Strategic Planning efforts. 1.b. Strategic Planning process initiated for Student Affairs.

2. Defined key activities, goals and outcomes for each Student Affairs department. Once division-level strategic plan is further developed, department level work will resume. 3. a. Managed institutional surveys. b. Led NWCCU 7-year accreditation Standard 4 and formed Assessment Council.

2

Assessment

1. Re-develop the assessment process for Student Affairs to focus on outcomes.

2. Facilitate assessment planning for the Division of Student Affairs and each department within the Division.

3. Increase assessment of

processes and services that involve multiple areas.

4. Implement standardized utilization data collection for the division

1. Continued to monitor efficacy of new process

2. Completed assessment plans for each unit of Student Affairs and the Division.

3. Focused on Enrollment Management Processes and mapping

4. Worked through

PeopleSoft feed issues and began soft roll-out to Division.

Research Develop 1-2 research studies annually.

Focused research plans with collaborators identified.

Reporting

1. Develop a division-wide annual report.

2. Increase informal

reporting of data through website and newsletter.

3. Develop 1-2 professional publications each year.

1. Annual report will be drafted for SALT review by January 2015;

2. Established assessment newsletter for Student Affairs.

3. This goal has not

been achieved.

Staff Excellence and Education

1. Support professional development for AER staff.

2. Provide educational opportunities for Student Affairs and other data entities on campus to develop best practices for assessment and share

1. Each staff member will have a professional development plan with associated costs, goals and outcomes.

2. Provide training for student affairs staff on relevant AER topics.

3

research findings related to students.

A discussion of each key activity, associated goals and outcomes follows. Strategic Planning

Goal One: Facilitate ongoing strategic planning for the Division of Student Affairs to align with institutional strategic planning efforts. Goal Two: Assist each Student Affairs Department with strategic plan development in a manner that aligns with the Division’s Strategic Plan. Over the past two years, AER has worked closely with Student Affairs departments to document key activities (key focus areas) as related to assessment and budget to prepare for a Division-wide strategic planning process that aligns with the University of Utah Four Big Goals:

1. Promote student success to transform lives 2. Develop and transfer new knowledge 3. Engage communities to improve health and quality of life 4. Ensure long-term viability of the University

4

Proposed Student Affairs Strategic Goal Map with the University of Utah Four Big Goals (draft 4/27/2016)

AER has analyzed the key activities identified by departments in relation to the current strategic plan. This analysis was provided to the Student Affairs Leadership Team (SALT) to provide contextual information to shape the strategic planning process. This approach creates a feedback loop from the departments as well as alignment with the Four Big Goals. Given the role of Student Affairs, the figure above suggests a strategy for relative weighting of Student Affairs alignment with the Four Big Goals. The following chart shows the alignment with the previous plan, updated strategic objectives and alignment of key activities.

Big Goal 1: Student Success Student Affairs:

• Student Engagement & Success

• Student Health & Wellness • Strategic Enrollment

Management • Diversity and Inclusion • Staff Excellence • Partnerships • Assessment & Best Practices • Facilities & Resource

Management

Big Goal 3: Engage

Communities Student Affairs

• Strategic Enrollment Management

• Diversity & Inclusion • Partnerships • Facilities & Resource

Management

Big Goal 2: Generate New Knowledge Student Affairs:

• Student Engagement & Success

• Student Health & Wellness

• Staff Excellence • Assessment and Best

Practices

Big Goal 4: Ensure Long Term Viability Student Affairs:

• Strategic Enrollment Management

• Diversity & Inclusion • Partnerships • Staff Excellence • Facilities & Resource

Management • Assessment & Best Practices

5

Existing Student Affairs Strategic

Objectives (2009) Updated Strategic Objectives 2016

Alignment of Division Level Key Activities

1. Student Engagement: Develop students as a whole through the cultivation and enrichment of the body, mind and spirit.

1. Student Engagement and Support

• Student Success & Support • Engagement & Leadership • Campus Community & Safety •

2. Strategic Enrollment Management

• Student Success Support • Diversity and Inclusion • Recruitment & Access • Staff Excellence • Community and Partnerships

3. Student Health and

Wellness • Student Health and Wellness

3. Commitment to Diversity: Promote diversity on campus through effective programming and active recruitment of staff and students.

4. Diversity and Inclusion • Diversity and Inclusion • Recruitment & Access • Staff Excellence • Campus Community & Safety • Community & Partnerships

2. Professionalism: Provide education that ensures all staff are properly trained to provide professional and competent service. 4. Exemplary Staff: Recruit and retain highly qualified staff.

5. Staff Excellence • Staff Excellence • Diversity & Inclusion

5. Collaboration: Partner with faculty, staff and external constituencies to foster student development and enhance the greater community.

6. Partnerships • Community & partnerships • Student Success and Support

6. Assessment, Evaluation and Research: Utilize a coordinated assessment, evaluation and research approach to promote data driven-decision-making. 7. Leading Edge Technology: Provide, maintain and utilize technology to enhance student services, assessment, and communications. 8. Best Practices: Promote the effective use of best practices in Student Affairs departments, programs and services.

7. Assessment and Best Practices

Supports all key activities

8. Facilities and Resource management

• Facilities and resource management

6

• Partnerships • Student Success & Support

Outcome for Strategic Planning Goals One and Two: Key Activities for the Student Affairs identified and established as the foundation for Student Affairs strategic planning process that align with the University of Utah Four Big Goals. This year a process for strategic planning has been initiated and will continue into 2016. Departmental key activities inform the 2016 Student Affairs strategic planning process as well as the ongoing assessment and budget planning.

Goal Three: Collaborate institutionally to develop a strategic plan for research and assessment.

AER continues to serve as the main institutional survey management organization and provides support across the institution to minimize oversampling of students by coordinating and managing sample requests. The demand for this assistance has increased and this is a valuable service provided to the institution for AER. AER is the managing organization for Campus Labs Baseline, which is the survey administration and management tool utilized by Student Affairs, Undergraduate Studies and the Office of Budget and Institutional Analysis. While the majority of projects generated are from Student Affairs, AER staff continue to utilize time to assist outside organizations with account management, project management, reporting tools and hardware. AER continues to collaborate with other entities across the institution to produce data and share assessment outcomes.

0%

10%

20%

30%

40%

50%

60%

70%

80%

Campus Labs Usage by Area2013-2014 2014-2015 2015-2016

7

With the NWCCU Accreditation visit in 2016, Stacy Ackerlind formed and led the University of Utah Assessment Council in preparation for the visit. The charge for that committee is being defined by senior leadership and Dr. Ackerlind as well as the AER team will continue to offer leadership and support for institutional assessment initiatives in alignment with Student Affairs. Outcome: AER has continued to support the institution with sampling and managementfor institutional survey projects. Greater clarity about AER’s role is desirable for future planning and resource allocation to institutional efforts.

Assessment and Evaluation Goal One: Re-develop assessment process for Student Affairs to focus on outcomes. Over the past year, AER has refined the Student Affairs Assessment model to provide a clear focus for departments and the division as a whole.

Student Affairs Assessment Model

Student Life Cycle Data: Data about students from initial contact through retention that may be data-mined for its own value or connected to other data sources to assess the student life cycle it its entirety. This data is foundational in understanding how students navigate through Student Affairs and the institution.

Utilization Data: Any type of captured activity with a student that may come from a variety of systems but relies on connection to the student record. This type of data answers core questions such as (1) How many students were served, (2) when they utilized the program, service or event (3) how often they attended and used the program, service or event (4) who those students are

Learning Outcomes(Focus on direct measures)

General Outcomes (1-Retention, 2-Graduation, 3-Engagement,

4-Needs/Satisfaction, 5-Benchmarking)

Utilization(Focus on who is utilizing services and

programs)

Student Life Cycle Data(Focus on students through Enrollment

Management Data and Analytics)

8

(demographics). This data type may come directly from the student record (PeopleSoft) or from systems such as the following: Utouch, Campus Labs Forms, Hobsens, Titanium, UIT webforms, EMR, and other office specific solutions. Utilization data for assessment purposes is provided by AER to departments by two strategies:

• U-Touch / U Career Path: is a solution that provides basic frequency data and should be used to provide a summarized overview (via canned report) when a direct interface with PeopleSoft is not available or desired. U-Touch / U-Career path should not be used if another existing system meets the department’s needs. U-Touch / U Career Path reports are provided at the end of the data collection period and represents a level one analysis. This data in itself may be sufficient to answer the assessment issue or question or if more in-depth analysis are needed can form the foundation of a datamining/analysis strategy.

• Datamining: is a solution where data that is already captured by another system is mined to provide a report that includes basic frequency data (level 1) or more in-depth analysis (level 2) and may inform a larger analytics model. The department may provide the data to AER for analysis or AER may pull the report directly depending on the importance of the data to larger division level strategic efforts. There may also be some restrictions/limitations on data-mining within certain systems due to the nature of the data contained.

General Outcomes Data: General outcomes data provides evidence about the degree to which a particular departmental or divisional outcome has been meet. Learning Outcomes Data: This data provides evidence around the degree to which student learning has occurred as defined by the department and the division. Direct evidence of learning is richer data and as such requires more investment of time. Indirect methods may also be used to assess student learning. Goal Two: Facilitate assessment planning for the Division of Student Affairs and each department within the Division. This was the second year of the planned assessment process where assessment projects are initiated based on assessment plan that includes outcomes that is accepted by AER rather than assessment projects requested on-demand. This change was made to better manage AER staff time and ensure that any assessment project was connected to the departmental key activities, goals and outcomes rather than generated based on personal interest. In the past, even with limited FTE, a great deal of time was spent working with multiple individuals and was not an efficient strategy given the staffing level of AER. Departmental assessment projects are now facilitated through departmental assessment liaisons rather than multiple individuals within a department. Previous “Assessment Request” Model New “Assessment Planning and Initiation Model”

9

As the model above illustrates, the number of contacts with AER has been reduced and has streamlined communication and workflow. The following individuals served as Assessment Liaisons for their departments during 2015-2016.

Department Assessment Liaison for 2015-2016 Admissions Ann Buffington Asia Campus at Songdo Jean Oh ASUU Erica Andersen Bennion Center Jennifer "JJ" Jones Campus Recreation Julian Gomez Career Services Eric Bloomquist Center for Disability Services Chris Green Center for New Student & Family Programs Nomani Satuala Center for Student Wellness Kassy Keen Child Care Shauna Lower; then Virginia Despain Conference and Guest Services TBA Contract Administration TBA Counseling Center Tony Kemmochi Dean of Students Katie Cartee Dining Services Melody Anderson Diversity Council TBA Financial Aid Karen Henriquez Fraternity and Sorority Life Nick Robbins Housing and Residential Education Daysha Moes; Christine Fleming Learning Success Center Leslie Giles-Smith LGBT Resource Center Gabriella Blanchard Money Management Center Tiffany Davis Network Support TBA Office of Inclusive Excellence Belinda Otukolo Saltiban Professional Development Committee Elizabeth Duszak Registrar Michael Santarosa

10

Scholarships Carol Bergstrom Student Affairs Stacy Ackerlind Student Health Suzanne Martin & Mark Pfitzner Student Leadership and Involvement Tasha Myers TRIO Kyle Ethelbah Undergraduate Advancement Erica Marken Union Ashlee Christofferson Veterans Support Roger Perkins; then Amanda Keller Women's Enrollment Initiative Nedra Hotchkins Women's Resource Center Nedra Hotchkins * Note: These liaisons are the liaisons who served for most or all of the year. If there were changes mid-year, these are indicated with italics.

Due to staffing changes as well as AER staff time dedicated to the NWCCU Seven Year Report and site visit, AER approached the planning process in a more segmented manner than desired using the prior year plans due to time constraints. The outcome was similar to the past year with some projects continuing to be requested that were not on the plan.

Updated Assessment Process Model

As was anticipated with more focused plans, the number of assessment projects decreased. For the 2015-2016 year, the count of 194 includes only projects AER spent time on, excluding declined projects, projects that were never requested (e.g., department did not respond to initiation request) and projects that were not needed (i.e., department responded to initiation request by letting us know it was no longer needed). It does include all of the postponed projects, some of which AER spent time on and some of which were originally planned but postponed before any work was done on them.

Assessment Plan

Project Initiation

Development (AER & Dept)

Data CollectionData Review

Project Sumary

Review for next cycle

Project requested (not on plan)

11

Over this past year, the proportion of completed or “in process” assessment projects was very similar to the previous year. Departments cancelled fewer projects and also had a lower non-initiation rate. The assumption is that with more focus and use of existing data, departments would be able to scale back and prioritize their assessment efforts better. This is some initial evidence that must continue to be evaluated with AER’s more focused planning process.

211194

80 90107

141

201 186 196

0

50

100

150

200

250

AER Assessment Project Trend Data

Note: There was a process change in 2014-2015 to develop assessment plans for the year.

12

6344

2

31

16

161

164

0

25

50

75

100

125

150

175

200

225

250

275

FY14-15 FY15-16

AER Assessment Project StatusIn process/completeDept. CancelledAER declined*On plan but not initiated by dept.

* AER began tracking declined projects during FY15-16.

13

Assessment projects continue to be primarily surveys followed by analyses projects (using a variety of data sources such as student records and existing data sets) and projects that involve working with existing departmental data collection processes to enhance usability of data (reflected in the chart below as “other”).

Over this past year, a greater number of projects were facilitated for Enrollment Management, followed by Student Development. Approximately 5% of AER’s projects were institutional or were conducted as a courtesy to a college or other department outside of the Division.

8090

107

141

201 196211

161170

516 1

10 164

6

0

25

50

75

100

125

150

175

200

2007-20082008-20092009-20102010-20112011-20122012-20132013-20142014-20152015-2016

AER Assessment Project Trend DataTotal Survey Projects Focus Groups Ucard Utilization Data Analyses / Report Other*

* Other projects include AER support of internal data collection and sampling projects.

14

Outcomes: The data for this year related to the new process represents similar trends as seen in the past. An “Assessment Boot Camp” has been planned to increase the knowledge of the division about assessment to shift the culture to one that is more outcomes focused. The assessment planning process will also be revitalized in conjunction with the strategic planning process.

Goal Three: Increase assessment of processes and services that involve multiple areas. Over the past year through AER’s close alignment with Enrollment Management, emphasis was placed on better connection of student life-cycle data. Due to the complexity and demands of those offices as well as personnel changes within AER, this work is ongoing and is the highest priority. Because of the need to focus on Enrollment Management and limited staff, greater collaborative efforts were not initiated across the division. This goal will be addressed in 2016-2017 with the establishment of the Assessment Cooperative. Outcomes: Reports established on Student Life Cycle Data. Planning is occurring to continue to deepen this effort in 2016-2017.

Goal Four: Implement standardized utilization data collection for the Division of Student Affairs

In February 2016, Student Affairs Assessment, Evaluation and Research (AER) office started the process of implementing the Symplicity CSM (Career Services Manager) program in all of Student Affairs. A Multi-School Environment (MSE) was purchased with the goal of utilizing the kiosk function of the program to capture utilization in various locations and events across the division of Student Affairs. The MSE development began in February with a goal of going live by summer 2016. To determine which offices would best benefit from utilization data and to determine which offices would be likely pilot candidates, AER deployed a survey to Student Affairs Directors and/or Assessment Liaisons. Responses from offices that had more than one

11.3%

16.0%

30.9%

27.8%

7.7%

2.6% 3.6%

0%

5%

10%

15%

20%

25%

30%

35%

Business &AuxiliaryServices

Dean ofStudents

EnrollmentManagement

StudentDevelopment

StudentAffairs &

Division-wide

Institutionalprojects

OutsideProjects

2015-2016 AER Assessment Projects by Reporting Line

15

respondent were combined. Twenty five responses were analyzed and each department provided a response. Following is an overview of this data.

The majority of offices within the Division of Student Affairs currently have need of utilization data that can be gleaned from the kiosk/swipe technology offered by the Symplicity CSM/MSE program. Three offices have indicated that they have their own system for utilization data tracking: Admissions, Office of the Registrar and the LGBT Resource Center. Four offices indicated that they were not sure if they had need of a utilization data program. Seventeen offices have indicated that they had a need for a utilization data program. Six offices agreed to participate in the pilot during summer 2016. Due to issues with the feed from PeopleSoft, the system did not go live until August, which created chanllenges. Kiosks in Symplicity CSM/MSE The Symplicity CSM/MSE system is a robust system that operates within three capacities. The components are Events, Counseling and Kiosks. The system functions as a CRM for the creation of Career Services-related events such as On-Campus Recruiting, Career Fairs and Information Sessions. The system functions as a CRM for Counseling by allowing staff to track student action within the system and write notes as part of the digital file on each student. The Kiosk function serves to track student attendance through lobbies, events, workshops and

4

19

2

0 5 10 15 20

Different System in Place

Extremely important

Moderately important

Slightly important

Not at all important

How Important is This Type of Data to Your Area?

5

17

0 2 4 6 8 10 12 14 16 18

Unsure

Yes

Need for utilization data?

16

custom made events. The kiosk feature of the Symplicity CSM/MSE will be used to track student utilization across the Division of Student Affairs. Kiosks work by using a student’s UNID or U Card number, which in turn allows the database access to bio-demographical data populated by PeopleSoft. Kiosks can be individually tailored to the needs of each department, allowing for collection of data in a clean and concise manner. Utilization Strategy Needs

17

Utilization Data Collection Strategy Needed

Stationary Kiosks: These are fixed-position kiosks that can be employed in a lobby, or other entrance point, to collect student utilization data. A stationary kiosk can be any desktop computer, laptop, or tablet or mobile device with a reliable internet connection that is dedicated to the collection of student information. Mobile Kiosks: Mobile kiosks are kiosks that can be employed anywhere across campus using a laptop or handheld device, such as an iPad or iPod Touch. Use of mobile devices requires careful planning and execution. There are several different ways to collect mobile kiosk data, including options that do not require an internet connection. Kiosk Needs In an effort to roll out Symplicity CSM/MSE across the division, we needed to ascertain which types of kiosks are most needed by each office, how many kiosks (if they were doing a large event), or if they wanted to have printable name tags. That information, along with consulting with each office, should yield a framework of the technological needs of implementing this project.

1716

11

0

2

4

6

8

10

12

14

16

18

Stationary Mobile Both

Which Type of Strategies are Needed by Your Areas?

18

In thinking about the flow and space within your office, what type of device(s) would work best? Would a computer or laptop be most beneficial? Would a tablet or iPad be beneficial? Would either work? How many do you anticipate needing? Would you like to have the ability to print name tags for events?

19

Average Programs & Device Estimates On average, how many programs does your area produce per year that could benefit from this type of utilization data? How many devices would you need to optimize traffic flow?

20

Hardware Options Symplicity CSM/MSE kiosks can be employed on any online browser, be it stationary desktop computer, laptop computer, or tablets. Smartphones and iPod Touches are also usable as a check-in device to pull a student ID or ISO number into a kiosk for utilization data. The needs of an office are entirely dependent on what kind of hardware they currently use for other purposes and whether or not they envision having dedicated hardware for kiosk purposes or plan to use one or more devices in a flexible manner, employing the hardware only as needed.

21

Student Affairs Assessment, Evaluation & Research is taking a proactive stance on the needs of the offices within the division. In order to better understand the needs of each office, we have asked what more information aside from student record data would be helpful as well as any concerns that department may have.

Beyond Student Record Data Departments were also asked to share other concerns and needs that AER should address in the division-wide plan. Needs beyond PeopleSoft such as if a student is married/partnered, number of children as well as options for self-identification of variables such as gender, sex, sexual orientation and race ethnicity were suggested. Those who responded were very positive about the ability to understand who is utilizing their programs and services. The question was raised around whether this is expected for each department to use. As part of our assessment model, this data is critical. However, the roll-out will be soft to ensure that we can understand the nuances of each department and ensure that any utilization data collected meets their needs. Implementation of Utilization Data Collection The PeopleSoft feed to Symplicity was a much greater challenge than anticipated. Chuck Masoka, our information coordinator in his role shared between AER and Career Services spent the majority of his time working with UIT and Symplicity on different aspects of developing the feed. In addition a great deal of time of Elizabeth Duszak (Assistant Director) and Stacy Ackerlind (Director) was spend on mapping the feed, accounting and contractual issues. The project is dependent on getting a clean data feed from PeopleSoft into the Symplicity CSM platform. The Symplicity CSM system is currently in use by hundreds of people each day. The system is live and contains over 83,000 student records, over 7300 registered employers, and over 9700 employer contacts. Altering a live system in use is a delicate operation that requires much logic and planning for a successful implementation. Once we are certain that the system is operational, with minimal technical bugs, we will begin the process of rolling out the project to the rest of the division. Pilot offices will be the first with access to kiosks and systems reporting and we will begin automating the reporting cycle, with a goal of one month from beginning of data collection. The tool remains open to all of the division, but AER would like to first consult with each office to discuss how and when the tool will be operational for them. We are confident that this will bring a solution to offices in the division that are looking for a way to expand their understanding of students using their services and attending their programs. Outcome: Utilization data collection has been established via soft-rollout due to extended delays as described above. Evaluation of the use of this tool are in process as we plan for the future.

Research

Goal: Develop one to two research studies annually.

AER developed the Campus Climate Survey on behalf of the Academic Senate, under the leadership of Lori McDonald and Bill Johnson. The survey was administered, data analyzed and a formal report prepared for the Academic Senate and Senior Leadership.

Outcome: One complex institutional study was completed on behalf of the Academic Senate.

Reporting

Goal One: Develop a division-wide annual report. This goal is underway with the initiation of the Student Affairs Strategic Plan. A report focused on the past plan is being developed.

22

Outcome: This goal is in process. Goal Two: Increase informal reporting of data through website and newsletter. The Student Affairs “Assessment Highlights” was initiated. Three issues for 2015-2016 are available here: http://studentaffairs.utah.edu/assessment/newsletter.php. The newsletter includes data of interest to the division, a focus on Assessment Liaisons and upcoming AER events and trainings. Three reports are also available on the AER website under the “Closing the Loop” section: http://studentaffairs.utah.edu/assessment/assessment-results/reports.php. Outcome: This goal was met with the establishment of the Assessment Highlights Newsletter and reports. Goal Three: Develop 1-2 professional publications each year This goal has again not been met. Greater effort needs to be dedicated towards this goal and/or the goal itself may need to be re-evaluated. Outcome: A publication plan was not completed or a plan to publish developed.

Staff Excellence and Education

Goal One: Support professional development for AER staff In order to utilize best practices to support the Division in creating a culture of evidence, providing opportunities for learning and continued skill development are critical for the AER team. Each staff member completed a professional development plan that was used as the basis for evaluation. Outcome: Each staff member has a professional development plan with associated costs, goals and outcomes for 2015-2016. Goal Two: Provide educational opportunities for Student Affairs and other data entities on campus to develop best practices for assessment and share research findings related to students. Assessment Coffee Hour Seminar Series In 2015-2016, the Assessment Coffee Hour seminar series was continued for Student Affairs Staff. The attendance at the events varied greatly with a very high number of staff attending the session offered by Vice President Snyder and Senior Vice President Watkins.

Topic Number of Attendees Fulfilling Our Promises to Students: Fostering and Demonstrating Student Success AER hosted watch party for the ACPA Presidential Symposium, (September 29, 2015)

14

The Accreditation Coffee Hour: A Conversation, presented by Dr. Barbara Snyder, Vice President for Student Affairs, and Dr. Ruth Watkins, Senior Vice President for Academic Affairs. (October 27, 2015)

55

Graduating Student Survey: Understanding the Student Experience, presented by Dr. Elizabeth Duszak, Assistant Director; Dr. Stacy Ackerlind, Special Assistant to the Vice President for Student Affairs and Director; and Emery Vigil, Graduate Assistant, Student Affairs Assessment, Evaluation & Research (January 26, 2016).

22

23

National Student Financial Wellness Study, presented by Ann House, Coordinator of the Personal Money Management Center; Tiffany Davis, CPFC, IRS Certified Tax Preparer (February 16, 2016).

13

Sexual Misconduct at the U: A preliminary look at the Campus Climate Survey, presented by Dr. Lori McDonald, Dean of Students; and Dr. Elizabeth Duszak, Assistant Director Assessment, Evaluation & Research (April 20, 2016).

43

Evaluation data was collected for each event, compiled, reviewed, shared with presenters and used to plan subsequent events. Assessment Technology Trainings Assessment as an organizational practice, by necessity, needs to be integrated into many individual roles with a department. To assist our colleagues in using data that supports their work, AER offers trainings on data analysis using Campus Labs Baseline (the survey and data management system for the Division). The following trainings were offered during this reporting period primarily to Student Affairs staff, although AER also reaches out to Undergraduate Studies and OBIA to include any staff who might also benefit from training. All of the trainings other than the Project Writing Lab, were well attended and received positive evaluations. The Project Writing Lab was designed to assist Assessment Liaisons in completing project summaries as this was determined as a need by AER staff, so we were not expecting this result. More discussion will occur around this topic with the Assessment Liaisons to understand their needs and develop a different approach.

Topic Audience Dates Number who attended

Project Writing Lab Liaisons August 10 & 14, 2015 0 Student Response System Anyone August 19, 2015 12 Assessment Beyond Surveys Liaisons September 17, 2015 8 Student Response System Anyone March 10, 2016 6 Campus Labs Baseline Basics Anyone March 17, 2016 3

Assessment of Campus Labs Baseline Trainings Evaluations were administered following each of the trainings. Attendees who completed the evaluation rated the trainings positively. Trainings will continue to be offered to Student Affairs staff and to others outside of the division.

24

Outcome: Trainings were provided and evaluation data are used to update elements of the training. This data will inform the training plan for 2016-2017.

II. Plans for the future The role of Assessment, Evaluation and Research has evolved over time to meet the needs of the Division as well as address gaps at the institution. With the charge to provide Enrollment Management analysis and reports as well as implement a system for utilization data, AER spent considerable time on data mapping to ensure accurate data for both needs. AER has continued to address institutional survey needs and increased capacity to provide meaningful data visualizations. AER has also played a critical role in the Division’s strategic planning process.

A. Anticipated challenges 1. Prioritization: While our overall number of projects decreased, the complexity of our projects increased

tremendously. This was especially true for the new role of Enrollment Management analytics. This complexity will likely continue to increase, with the demand for actionable data. With institutional demands of AER for initiatives such as Civitas, it will continue to be a balancing act and require us to continue to ensure that what is being requested of us, truly meets a priority. Our concern has been that if we do not provide the data on demand, that departments will develop “work-arounds” that are problematic in terms of accuracy, integrity and/or adherence to policy. As we’ve delved into more student record data, we have seen an abundance of work-arounds that are problematic at the department level and present a segmented view at the division-level. A major goal for us is to create a detailed data map of the Division to ensure the best and most accurate data are being used to help retire some of these inefficient and ineffective strategies that are currently in place.

0

1

2

3

4

5

6

7

8

9

Training Evaluations

Very useful Useful Slightly useful Not useful

25

2. Data consistency and integrity: With the mapping process of student life cycle data for Enrollment

Management, AER has gained a great deal of insight as to how data is pulled from PeopleSoft and reported by different departments across the division. There are many “functional silos” with data that have developed to meet operational needs yet, present some challenges for connecting data across the life cycle. With gaining greater insight into how data is currently used and the development of meaningful analytics and routine reports for Enrollment Management leadership, AER has is working closely with the Business Intelligence team to ensure the data we use for analytics is the most accurate data. While this is critical, it is also incredibly time consuming and has created a strain with our routine workflow.

3. Utilization data process: As we began the process of rolling out Symplicity MSE as the Division’s utilization

data strategy, we encountered numerous challenges that were unanticipated. The MSE product is an adjunct to the Career Services Management platform. Our understanding had been that the feed for the CSM portion was functional and we would be adding to an existing feed. This was not accurate and required the majority of Chuck Masoka’s time, a great deal of Elizabeth Duszak’ s and Stacy Ackerlind’ s time to work closely with UIT, Symplicity and Career Services to make the feed functional. Because of the delays, AER did a soft roll-out of the utilization data. While this is an improvement to what had been in place, it will require focused time and effort to scale up. We have also learned that many departments have utilization data that is not being utilized effectively. This has been informative and deepened our mapping process.

4. Survey Management Software: With more emphasis on data that can be used effectively by departments,

the division and the institution as a whole, AER staff time needs to be used differently and focused more on analysis and thorough conceptualization of complex projects in addition to the mapping described above. We are exploring different survey management tools that will allow more flexibility and seamless integration of data into dashboards. Dr. Ackerlind has led the exploration process of different survey tools and has will begin the review process through the Integrated Student Team (IST).

5. Institutional surveys and lack of policy: AER has continued to meet the institutional need for coordination

and administration of surveys. While this has likely reduced some oversampling of students, it does not address the number of surveys that are sent out through the colleges, departments or even by students to students. AER will continue to explore how we might serve the institution through leadership to develop a strategy that will hopefully lead to meaningful policy development and clarity of our role. With the new technology being explored, there may be some ways to merge these two needs.

B. Anticipated opportunities: 1. New Student Affairs Strategic Plan: With a greater focus on outcomes within the strategic plan, this will

provide AER a more well-defined focus for assessment and prioritization. It will also sharpen our focus for reports to address accomplishment of specific goals.

2. Implementation of new Student Affairs Assessment Model: With expanded capacity to collect utilization data and additional AER staff, our focus will be to implement the model across the division more fully. This will provide a consistent assessment approach at the program, department and divisional level.

3. Improved data usability and access: With access to the Student Data Warehouse, PeopleSoft and our

collaboration with the Business Intelligence group, AER is able to provide analytics on existing data which supports our assessment model. This greatly supports our ability to expand the reporting for Enrollment Management and for other areas who have needed more integration of utilization and student records data to inform their work.

26

4. Further strengthen Student Affairs reporting and budget process: With the establishment of the Four Big Goals, the new institutional budget process and the new reporting and budget process within Student Affairs, AER has the opportunity to support our leadership through refining data elements and showing the impact of Student Affairs programs and services. With the partnership with the Associate Vice President for Business and Auxiliary Services, Jerry Basford, AER can take an active and supportive role in refining this process for our division.

5. AER leadership with surveys and survey policy: With our leadership around survey technology and management of institution student surveys, we have an opportunity to work closely with our executive leaders to determine a vision for student surveys that will provide necessary data and protect students from excessive requests for information. As we continue to explore the new tools, this is providing structure to support conversations as well as consider integration with other institutional tools such as Tableau and Civitas.

C. Grants and contracts: None received in 2015-2016.

Staff Excellence

A. Professionalism Each AER staff member collaboratively develops a professional development plan that is used to evaluate performance across the year for both departmental and personal goals. Staff members are encouraged to enhance core job related skills as well as broader skills within student affairs. Dr. Ackerlind is currently the National NASPA Assessment, Evaluation and Research Knowledge Community Co-Chair Elect. This position entails responsibilities at both the national NASPA conference and the Assessment and Persistence conference. Following are the professional development conferences attended by staff members:

Staff Member Conferences Attended

Stacy Ackerlind, Director • Tableau Conference, 2015 • Utah NASPA Conference, 2015 • NASPA Western Regional Conference, 2015 • Qualtrics Summit, 2016 • NASPA National Conference, 2016 • NASPA Assessment and Persistence

Conference, 2016

Elizabeth Duszak, Assistant Director • Tableau Conference, 2015 • Utah NASPA Conference, 2015 • Qualtrics Summit, 2016 • NASPA Assessment and Persistence

Conference, 2016

Chuck Masoka, Information Coordinator • Symplicity Conference, 2016

Parul Acharya, Assessment Analyst • NASPA Assessment and Persistence Conference, 2016

27

B. Recruitment and retention of highly qualified staff Dr. Stacy Ackerlind has served as the Director for AER since 2006 and as the Special Assistant to the Vice President for Student Affairs since 2012. Dr. Duszak has served as the Assistant Director for AER since 2015. Due to increased needs for data, the assessment analyst position was reinstated to focus on Enrollment Management data needs. A new position was provided to develop a process for and manage utilization data as well as manage the Career Services platform. This position is a joint report between AER and Career Services and titled, “Information Coordinator.” Both positions were filled after a national search.

C. Retirements, new hires, accomplishments In 2015, Chuck Masoka joined the AER team in a new position of “Information Coordinator” that is a shared position with Career Services to manage Symplicity. Dr. Parul Acharya was hired as an Assessment Analyst for Enrollment Management.

1. Awards: None received in 2015-2016.

2. Committees

Ackerlind, Stacy • Student Affairs Leadership Team

• NWCCU Accreditation Executive Team • NWCCU Accreditation Data Team (chair) • Academic Senate Committee for Campus Climate • University of Utah Retention and Completion Tasks Force • Graduating Student Survey Assessment Team (Chair) • NASPA Assessment, Evaluation and Research Knowledge

Community (National), Co-Chair Elect

Duszak, Elizabeth • Student Affairs Assessment Cooperative (Chair) • Academic Senate Committee for Campus Climate • NWCCU Accreditation Data Team • Graduating Student Survey Assessment Team • Student Affairs Professional Development Committee

Masoka, Chuck • Student Affairs Diversity Council

3. Presentations and publications

Duszak, E., & Ackerlind, S. J., (2016). Administering a Campus Climate Survey on Sexual Harassment

and Violence. Presentation at the NASPA Assessment and Persistence Conference, Portland, OR

4. Faculty appointments Ackerlind, Stacy Adjunct Assist Professor Educational Leadership and Policy