A s s e s s m e n t i n H i g h e r E d u c a t i o n : E ... · Assessment Considerations for Internship Programs in a Time of Social Justice Expectations ~Laura Spears How to Better

2019 Assessment Conference

Assessment in Higher Education: Enhancing Institutional Excellence

Emerson Alumni Hall

Institutional Assessment Office of the Provost

April 5, 2019

Welcome On behalf of the Academic Assessment Committee, I welcome you to UF’s third assessment conference, Assessment in Higher Education: Enhancing Institutional Excellence. Our presenters come from UF, Florida SUS member institutions, and from across the United States. Each will share ways that data and information collected through the assessment process can be used to enhance the excellence of our programs and institutions.

Our opening plenary is a workshop led by Bess de Farber and Laura Spears on developing collaborations for assessment. Following the plenary, you will choose from the excellent presentations offered in three concurrent sessions. We will close with an award ceremony and a panel discussion with four University of Florida students who will discuss their perceptions of excellence in assessment.

The Academic Assessment Committee is the joint Senate committee that oversees academic assessment processes at the University of Florida (members listed below). This committee not only reviews and approves Academic Assessment Plans (including Academic Learning Compacts, Student Learning Out-comes and Program Goals), but takes a leading role in developing and improving academic assessment processes at the institutional level.

Thank you for joining us today. Here is what we hope you will take away from this event:

- at least one good idea about assessment that you did not have before- at least one question or matter you would like to learn more about- at least one next step that you intend to take- the contact information for at least one colleague with whom you may be able to collaborate about assessment

Enjoy your day with us!

Sincerely, Timothy S. Brophy, Professor and Director of Institutional Assessment Chair, Academic Assessment Committee

Timothy S. Brophy

2

We are pleased to announce the launch of the new open-access, peer-reviewed Journal of Assessment in Higher Education. The journal is a collaborative publication of the George A. Smathers Libraries and the

University of Florida Office of Institutional Assessment.

A call for papers will be issued soon – we hope you will be interested in sharing your work.

3

Conference Program Breakfast and Conversation 8:00am-8:45am

Ballroom

Welcome ~Dr. Kent Fuchs, President, University of Florida ~Dr. Timothy S. Brophy, Professor and Director, Institutonal Assessment

9:00am-10:45am Ballroom

Keynote: Initiating Assessment Partnerships: CoLAB Planning Series® Workshops ~Laura I. Spears, Bess de Farber

8:45am-9:00am Ballroom

Ballroom 2

Ballroom 3

Warrington

Classroom

LUNCH (provided) 12:00PM-1:00PM Ballrooms

The Impact of an Assessment Certificate on Faculty Perception and Knowledge ~Katherine Perez, Jennifer Doherty-Restrepo, Eilyn Sanabria, Suzanne Lebin

Using Predictive Analytics in an Ethical, Meaningful Way ~Will Miller

Linking Program Assessment to Institutional Goals and Strategic Planning ~Patrice Lancey, Liz Grauerholz, Kristen Couper Schellhase, Cory Watkins

Student Feedback on Online Course Design: Going beyond traditional faculty evaluations to gather a more comprehensive view of the student experience and preferences ~Heather Maness and Allyson Haskell

Creating diagnostic assessments: Automated distractor generation with integrity ~Darryl Chamberlain Jr.

Ballroom 2

Concurrent Session 1 - 11:am-11:45am

Concurrent Session 2 - 1:15pm-2:00pm

4

Conference Program Working with Faculty to Improve Pedagogy and Student Success ~Will Miller

Ballroom 2

Warrington

Classroom

2:00pm-2:30pm

~A Sidi, MD

Ballroom 2

Ballroom 3

Warrington

Classroom

UFIT Exceptional Course Development Award

CLOSING SESSION: Student Panel: Excellence in Assessment: The Student Perspective ~Moderator: Timothy S. Brophy

3:30pm-3:45pm Ballroom

3:45pm-4:30pm

Creating Effective Skills Instruction & Assessment ~Taryn Marks & Christopher Vallandingham

Integrating General Education and Academic Program Assessment: A World Cafe Discussion ~Kyna Betancourt, Christopher Combie

Coffee and Snack Break

Concurrent Session 3 - 2:30pm-3:15pmAssessment Considerations for Internship Programs in a Time of Social Justice Expectations ~Laura Spears

How to Better Assess the Effectiveness and Impact of the Student Experience ~Maggie Walsh, Strategist and Adam Griff, Managing Director, brightspot strategy

Using Administrative Data to Support Program Review and Assessment ~Mariajosé Romero PhD

Simulation-based Assessment Identifies Longitudinal Changes in Cognitive Performance in the UF Anesthesiology Residency Training Program

Concurrent Session 2, continued

5

Emerson Alumni Hall Map

6

With Gratitude for Their Support Dr. Kent Fuchs, President Dr. Joseph Glover, Provost The Academic Assessment Committee Faculty, College, Student, and Guest Presenters Ann Greene, Executive Secretary, Office of the Provost Dr. Lily R. Lewis, Director of Postdoctoral Affairs and Academic Affairs Communications

Today’s sessions are being recorded. The videos and presenter materials will be posted on the Institutional Assessment Faculty Resources Page

as soon as they are available. https://assessment.aa.ufl.edu > Faculty Resources

Amarat Simonne, Co-Chair, College of Agricultural and Life Sciences Laura Spears, George A. Smathers Libraries Margaret Fields, College of Liberal Arts and Sciences Teresa Mutahi, College of Liberal Arts and Sciences Gail Childs, College of Dentistry Melissa L. Johnson, Honors Program Avner Sidi, College of Medicine Jinying Zhao, College of Public Health and Health Professions and College of Medicine

The Academic Assessment Committee

7

Keynote Speakers 9:00am-10:45am

Laura I. Spears, PhD. ([email protected]), is the Associate Assessment Librarian with the University of Florida Libraries. Her job responsibilities include examining a variety of library data that demonstrate library impact on the life of all types of library users Her research examines library value in funding advocacy and values assessment in academic libraries. Recent publications focused on overnight library use perceptions, academic research ethics and use of Appreciative Inquiry in CoLAB Workshops. Laura works closely with the Academic Assessment Committee to establish and expand the Smathers Libraries’ impact.

Bess de Farber has been the Grants Manager at the University of Florida Libraries since 2008, and previously served in the same position at the University of Arizona Libraries. She has provided grantsmanship instruction throughout the past 30 years for audiences including graduate-level library and information students, arts, museum, and social service agency professionals, and has led efforts to secure millions in grant funding for nonprofits and academic libraries. Her research interest is asset-based collaboration development. As a Certified Professional Facilitator through the International Association of Facilitators, she invented Collaborating with Strangers Workshops, under the umbrella of CoLAB Planning Series®, which are large group facilitated processes for individuals and organizations seeking new collaborative partnerships. This process has served more than 600 organizations and 3,000 individuals since 2002.

Title: Initiating Assessment Partnerships: CoLAB Planning Series® Workshops

Speakers: Laura I. Spears and Bess de Farber

Biography:

Biography:

Laura I. Spears

Bess de Farber

8

Location: Ballroom

Concurrent Session 1: 11:00am-11:45am

Predictive analytics are being utilized more and more frequently in higher education as we aim to determine ways we can better determine which students are likely to be successful on our campus. With holistic data becoming more readily available and advanced statistical techniques becoming more higher-education friendly, it’s clear that innovative uses of data are not merely some passing fad. Yet, for campus stakeholders, figuring out ways to start making use of data and conducting predictive analyses can be a daunting task. We will discuss what predictive can do to help campuses, how they are typically conducted, and perhaps most importantly the ethical considerations that must be acknowledged and considered prior to use. We will examine how results can be dispersed and best used by campus stakeholders to improve pedagogy, program design, and chances for students to succeed, Data governance and student privacy will also be discussed.

As Assistant Vice President, Campus Adoption, Will Miller, an unabashed data wonk, leverages data best practices to help campuses make strategic decisions. He joined the Campus Labs team in late 2016, after serving as a faculty member and senior administrator at Flagler College in Florida. There, as Executive Director of Institutional Analytics, Effectiveness, and Planning, he helped transform the campus-wide outcomes assessment process. He also served as Accreditation Liaison to the Commission on Colleges of the Southern Association of Colleges and Schools (SACSCOC). A prolific author and dynamic speaker, Will regularly presents at professional conferences for both higher education and political science.

Title: Using Predictive Analytics in an Ethical, Meaningful Way

Presenter: Will Miller

Abstract:

Biography:

Will Miller

9

Location: Ballroom 3


Dr. Katherine Perez is the Associate Director of Institutional Effectiveness at Florida International University. For the past 11 years, she has worked in this department and has successfully gone through two compliance reports submissions. As higher education professional, Katherine’s areas of interest include assessment best practices, educational pedagogy, statistics, educational research, and metric development/validation.

Title: The Impact of an Assessment Certificate on Faculty Perceptions and Knowledge

Presenters: Katherine Perez, Jennifer Doherty-Restrepo, Eilyn Sanabria, Suzanne Lebin

Katherine Perez

Biography:

Abstract: The Institutional Effectiveness team at Florida International University created a certificate program designed to instruct faculty on assessment best practices. Faculty learned how to develop student learning and program outcomes, curriculum maps, assessment methods, and assessment instruments. Additionally, the program covered data collection, best practices for analyzing and reporting results, developing improvement actions and following up on the implementation of these actions. The impact of this program was measured by: 1) pre/post survey focused on six dimensions 2) pre/post quizzes for each of the four modules of the certificate. Results indicate significant improvements on faculty perceptions of self-efficacy and support offered by the IE office, but no significant changes in perceptions of the value of assessment practices, willingness to participate in assessment, utilization of assessment data, and support offered by their departments or colleges. Significant learning gains were observed in the first three modules.

10


Concurrent Session 1: 11:00am-11:45am Biography: Ms. Eilyn Sanabria is the Senior Program Coordinator of Institutional Effectiveness at Florida International University. She is currently working on her PhD in Curriculum and Instruction with a focus in Reading Education. She is responsible for facilitating university-wide Institutional Effectiveness efforts and for quality assurance of assessment reports. Her research interests are pedagogical and assessment best practices in higher education, preservice teacher preparation, and primary reading instruction and assessment. Mrs. Suzanne Lebin is the Program Coordinator of Institutional Effectiveness at Florida International University. She is responsible for coordinating and facilitating Institutional Effectiveness efforts across the University. Mrs. Lebin earned her master's degree in Educational Psychology from Florida Atlantic University and her interests include improving educational quality, assessment in higher education, and learning and professional development. Dr. Jennifer Doherty-Restrepo is a Clinical Associate Professor of Athletic Training in the Nicole Wertheim College of Nursing and Health Sciences at Florida International University. As the Director of Accreditation in the Office of Academic Planning and Accountability, she is responsible for overseeing compliance with state, regional, and national accreditation standards at the program, college, and university level. Dr. Doherty-Restrepo has over eighteen years of experience in higher education as both an educator and administrator. Her areas of expertise include andragogy, simulation-based instruction, and professional/continuing education in athletic training. Jennifer serves SACSCOC as an Institutional Effectiveness Evaluator. Additionally, she serves the Commission on Accreditation of Athletic Training Education as an accreditation site visitor.

Ms. Eilyn Sanabria

Mrs. Suzanne Lebin

Dr. Jennifer Doherty-Restrepo

Biography:

Biography:

11


Integration of major institutional research-based planning and evaluation processes is a mechanism to focus all university constituents on implementing and evaluating strategic initiatives designed to improve institutional quality and effectiveness. While educational program assessment to foster evidence based improvements is strongly infused in the culture of many universities, drawing intentional connections between program assessment, which primarily focuses on student learning outcomes, and institutional strategic planning can be challenging for faculty. This session will highlight the assessment work of several diverse disciplines in a large public research institution that have articulated connections between their program’s student learning and program outcomes and elements of the university strategic plan. A panel of faculty will report their program assessment goals and measures. The benefits and challenges of explicitly linking outcomes or measures in program assessment to university planning will be highlighted. The panel will share the Institutional Effectiveness Assessment Rubrics used by this institution, which includes criteria related to linking planning processes, that provides a framework for communicating clear expectations, fostering mentoring, monitoring the assessment process and deepening its quality.

• “Linking program assessment to institutional goals and strategic planning: Why does it matter?” Dr. PatriceLancey, Assistant Vice President for Operational Excellence and Assessment Support.• “Assessing Student Diversity and Inclusiveness in the Social Sciences B.S. Program to Improve StudentLearning and Further Institutional Goals.” Dr. Liz Grauerholz, Professor of Sociology and Past-Director of SocialSciences.• “Using professional standards to create outcomes that increase student success and program prominence inan undergraduate Athletic Training program.” Dr. Kristen Schellhase, Program Director of the Athletic TrainingProgram.• “Identifying and Monitoring Program Assessment Markers: An Effort to Improve Retention and Time toGraduation.” Dr. Cory Watkins, Associate Professor of Criminal Justice and Undergraduate Director/InternshipCoordinator.

Title: Linking Program Assessment to Institutional Goals and Strategic Planning

Presenters: Patrice Lancey, Liz Grauerholz, Kristen Couper Schellhase, Cory Watkins

Abstract:

12

Location: Warrington


Patrice Lancey, Phd. Vice President, Operational Excellence and Assessment Support, University of Central Florida. Dr. Lancey earned her M.A. and Ph.D. in Clinical Psychology from Wayne State University, Detroit, Michigan, in 1979 and 1996 respectively. She joined the University of Central Florida in 2001 where she serves as Assistant Vice President of Operational Excellence and Assessment Support. Dr. Lancey directs the university wide Institutional Effectiveness Assessment process. She also designs statistical studies about student engagement, institutional conditions that enhance student learning outcomes, retention and graduation. She serves as the outside evaluator for externally funded grants, presents papers and workshops on assessment topics at conferences, serves as a consultant to other universities and as a proposal reviewer for the Association for Institutional Research. She regularly works with faculty to develop research methodology and student learning assessment.

Liz Grauerholz, PhD. Professor of Sociology and Past-Director of Social Sciences B.S. Program. Dr. Grauerholz is Professor of Sociology at University of Central Florida. She has extensive experience with program assessment from both administrative and faculty positions. She served for 3 years as Chair of the College of Sciences Divisional Review Committee that oversees program assessment for 41 programs within the College. In this role, Dr. Grauerholz worked directly with faculty and other groups to improve their programs and most importantly, student learning. In AY 2017-2018, Dr. Grauerholz served as Chair of the UCF University Assessment Committee, which oversees assessment efforts for all programs and units at UCF and was responsible for reviewing assessment results and plans for one of the divisions of Academic Affairs. As a faculty member, she served as Assessment Coordinator for the UCF’s Social Sciences major and in the past, as Assessment Coordinator for the Department of Sociology’s MA and PhD programs. She currently serves as a reviewer for the College of Sciences assessment programs.

Biography:

Biography:

Patrice Lancey

Liz Grauerholz

13


Kristen Couper Schellhase, EdD, LAT, ATC. is the Program Director for the Athletic Training Program at the University of Central Florida. She has lead the assessment initiatives for Athletic Training since becoming the Program Director in 2004. She also served as the Department Chair for the Department of Health Professions from 2010 to 2017 and served as the Assessment Division Review Committee Chair for the College of Health and Public Affairs in 2017-2018. Dr. Schellhase was a 2007 recipient of the NACADA Outstanding New Faculty Advisor Award, the 2005 College and University Athletic Trainer of the Year Award and a 2012 NACADA Outstanding Advising Certificate of Merit Winner. Dr. Schellhase has a Doctorate in Curriculum and Instruction from UCF, a Master of Education in Athletic Training from the University of Virginia, and a Bachelor’s in Exercise Science/Athletic Training from The Pennsylvania State University.

Cory Watkins, PhD. is an Associate Professor and the Undergraduate Program Director and Internship Coordinator in the Department of Criminal Justice at the University of Central Florida. Dr. Watkins has worked on Institutional Effectiveness assessments and committees at the department and college levels for the last six years. He has been involved in several federally funded law enforcement technology related projects and has worked with GIS based applications extensively for the last 18 years. He has published articles in the area of police effectiveness and police technological solutions in Policing: An International Journal of Police Strategies and Management, Police Practice and Research: An International Journal, Police Quarterly, and The American Journal of Criminal Justice.

Kristen Couper Schellhase

Biography:

Biography:

Cory Watkins

14


This session will cover the key findings from our surveys of students (n = 1,841) enrolled in newly developed online courses or courses running for the first time after significant revisions (n = 99) from Fall 2015 through Summer C 2018. We will focus on the instructional design preferences and expectations of these online students and the application of these findings to continuous course improvement goals. The primary topics will include student use of module objectives (it’s more than you think!), lecture recording preferences, course engagement (peer, instructor, and platforms), and expectation management. Additionally, we will use active learning techniques (facilitated discussion and think-pair-share) to explore instructional design best practices and the benefits of more in-depth surveys, such as the instrument we developed, for instructors and administrators. Survey design and methods will also be covered (ex: response rate issues, question evolution, etc).

Heather Maness, MS, PhD. has worked for the University of Florida in various capacities over the past 14 years, and has spent the past four years working as an instructional designer for UF Information Technology’s Center for Instructional Technology and Training Lately her work has focused on the transformation of courses adopting the flipped classroom model as well as consulting on assessment challenges at the course or programmatic level. In 2016, she completed her PhD, majoring in agricultural education and communication, and minoring in higher education administration. Her research focuses on stakeholder (employers and students) evaluation for curriculum development and revision.

Allyson Haskell, ME. has four years of instructional design experience, two years’ teaching experience, and holds a master’s of engineering in electrical engineering. She has worked at the University of Florida for 10 years in various positions, most recently as an instructional designer facilitating course launches in all disciplines and levels. She also teaches several workshops for teaching assistants focused on online teaching.

Title: Student Feedback on Online Course Design: Going Beyond Traditional Faculty Evaluations to Gather a More Comprehensive View of the Student Experience and Preferences

Presenters: Heather Maness and Allyson Haskell

Abstract:

Biography:

Biography:

Heather Maness

Allyson Haskell

15

Location: Classroom

Concurrent Session 2: 1:15pm-2:00pm

Assessment is a driving-force in any course – it informs students of what the instructor believes to be the most important content while also providing the instructor a measure of student understanding. Multiple-choice assessments are common in large-enrollment courses to efficiently assess students. However, two major barriers to quality multiple-choice assessment development exist: (1) creating plausible, alternative options and (2) protecting from students “working backwards” from options to the solution. First, I will illustrate how open-source technology was used in a College Algebra course to dynamically generate a question, solution, and plausible alternative options while simultaneously ensuring students could not work backwards. I will then solicit questions from the audience about how these assignments have been used in the course or could be used in other courses. Finally, time will be provided to allow audience members to consider how they could automate a question in their content field.

Darryl is a lecturer in the Mathematics department. He received his PhD in Mathematics, concentrating on Collegiate Mathematics Education, from Georgia State University in 2017. His research interests include: Diagnostic Assessment, Curriculum Development, Teaching with Technologies, and Argumentation and Reasoning.

Title: Creating Diagnostic Assessments: Automated Distractor Generation with Integrity

Presenter: Darryl Chamberlain Jr.

Abstract:

Biography: Darryl Chamberlain Jr.

16



When it comes to assessment data, institutional effectiveness and research administrators can do little without the contribution of faculty. Regardless of how well-designed a system might be or accreditors calling for more granular data, faculty own their classroom—if they choose not to track outcomes, we lose our holistic picture of student learning. As a result, it is essential that aim to maximize faculty buy-in to assessment practices. First, we must understand what the issue is. By building assessment around the ongoing work of faculty, continuing to emphasize teaching and learning as a research avenue, and creating opportunities to discuss assessment efforts, we can undoubtedly encourage faculty to check the box and fill out the form—but we may not help them to truly embrace the process. We need to work with faculty to show them how assessment can help improve their personal pedagogy, their program effectiveness, and ultimately student success.

Title: Working with Faculty to Improve Pedagogy and Student Success

Presenter: Will Miller

Abstract:

Will Miller

Biography: See Page 9

17



Students expect to graduate with practical skills that they can immediately use in their jobs. But it is difficult to create effective simulations that teach skills and provide a robust, adequate assessment of those skills. This interactive session, for anyone who wants to incorporate skill instruction into their curriculum, will first discuss the challenges of teaching and assessing skills. Then, through interactive lightning lessons, the session will explain best practices for assessing skills-based instruction, including how to identify skills that can be taught in the classroom; how to prioritize which skills to teach and assess; how to break down skills into their component parts to determine the most effective means of teaching those skills; and how to teach students to actively transfer the instruction they receive from theory into practice to maximize the skills that instructors are able to teach and assess.

Taryn Marks joined the University of Florida’s Levin College of Law’s Legal Information Center in 2014. She is currently Faculty Services Librarian & Professor of Legal Research. She teaches Legal Research, Advanced Legal Research, and Business Law Advanced Legal Research. She has published several articles, with topics ranging from shifting legal education in China to governmental documents in the digital age. Her book chapter “Using the Synergy of Millennials’ Activism and Librarians’ Radicalism to Create Libraries of the Future” was recently published in Millennial Leadership in Libraries. Marks earned both her J.D. degree and LL.M. in International Law from Duke University School of Law, and earned her Masters of Library and Information Science from the University of Washington.

Christopher Vallandingham is Head of Collections and Professor of Legal Research in the University of Florida’s Levin College of Law’s Legal Information Center. He has worked at UF Law since 2000, and has expertise in military law, intelligence law, and Florida territorial law. Since 2010, Vallandingham has served as a reserve officer in the United States Army Judge Advocate General’s Corps. His current rank is Major and he serves as a legal advisor to Task Force 76. He previously served as an Arabic linguist in U.S. Army intelligence units. He was the co-founder and president of the International Intelligence Ethics Association, the co-founder of the International Journal of Intelligence Ethics and he briefly served as the publication’s managing editor. He earned his J.D. degree from UF Law.

Title: Creating Effective Skills Instruction & Assessment

Presenters: Taryn Marks and Christopher Vallandingham

Abstract:

Biography:

Biography:

Taryn Marks

Christopher Vallandingham

18



Kyna Betancourt, Ph.D. is Assistant Dean of General Education, Assessment, and Communication in Undergraduate Studies at the University of South Florida. She was the Chair of the General Education Council during the curriculum enhancement and now provides guidance to the council in its function of overseeing the Enhanced General Education. Dr. Betancourt also leads the general education assessment overhaul and works with USF colleagues to integrate degree program and general education assessment. She is the Director of the Bachelor’s in General Studies and Applied Science degree programs and is an adjunct instructor in the Department of Communication Sciences and Disorders.

Christopher Combie, Ph.D. is Assistant Director of Assessment, Institutional Effectiveness. He oversees enhanced assessment initiatives to include overarching assessment, documenting evidence of using assessment results, and assessment accountability and transparency. Dr. Combie also oversees 800 plans and reports for academic and certificate program assessment, academic and student support services, and institutes and centers. He serves as an adjunct in the department of Leadership, Counseling, Adult, Career, and Higher Education. This is his ninth year at USF, having spent the past four years in the Morsani College of Medicine overseeing PhD and Postdoctoral Programs in the Biomedical Sciences.

A strong culture of assessment is one in which faculty are motivated to complete authentic assessment as a way of enhancing student learning. However, when assessment is not connected to what’s being taught in the classroom and results are not shared with faculty, buy-in and motivation are lost. An integrated assessment plan would be one where faculty are completing one authentic assessment and accessing the data for multiple course and program enhancements. At the University of South Florida, we are working to implement such an assessment plan based on the general education learning outcomes that can also apply to program assessment. This interactive session is meant to engage anyone involved in general education and program assessment in a meaningful conversation around how to integrate these two initiatives. By finding common student learning outcomes, diverse programs can be assessed efficiently and meaningfully.

Title: Integrating General Education and Academic Program Assessment: A World Cafe Discussion Presenters: Kyna Betancourt and Christopher Combie

Abstract:

Biography:

Biography:

Kyna Betancourt

Christopher Combie

19

Location: Classroom

Concurrent Session 3: 2:30pm-3:15pm Presenter: Laura Spears

Title: Assessment Considerations for Internship Programs in a Time of Social Justice Expectations

Biography: See Page 8

Laura Spears

Evolving academic and research environments require the inclusion of professional expertise and abilities from multiple disciplines. Concurrently, the declining market for tenure-track academic positions and increasing competition for other professional positions are challenging to both graduate students, seeking career opportunities aligned with their educational experience, and to graduate programs, seeking to place their graduates in fulfilling careers. In this environment, students need exposure to careers to expand their professional horizons and opportunities to gain marketable work experience. This presentation and facilitated discussion will review change management processes for implementing, with ongoing evaluation, of the Libraries Graduate Internship Program, including evaluative processes supporting the Libraries, librarians as internship directors, collaborating faculty members in other colleges, and graduate student interns. The session will conclude with a facilitated discussion on graduate student internships on campus, payment rates, additional compensatory factors, learning outcomes, broader impacts for careers, and broader impacts for collaborative participants.

Abstract:

20



Facing competitive pressures as well as student’s increasingly higher expectations and complex needs, colleges and universities need to better assess the effectiveness and impact of their student experience. In seeking to improve the student experience, many institutions have undertaken large-scale initiatives without first assessing it in a way that can effectively inform their planning process.

This interactive session will first review current methods of defining and assessing the student experience.. Next, the session will share the development of a new tool, Student Experience Index, to identify opportunities to improve the student experience and how one campus used it in planning a new campus experience to support student success. We will engage the attendees in a discussion of their challenges in assessing the student experience and how to use the Student Experience Index to diagnose and improve their improve the student experience.

Title: How to Better Assess the Effectiveness and Impact of The Student Experience

Presenters: Adam Griff and Margaret Walsh

Adam Griff is a director at brightspot strategy, a consultancy focused on reimagining the higher education experience. He has directed projects for leading colleges and universities including University of Miami, University of Virginia, Florida International University, MIT, and UC Berkeley. He is a frequent speaker on the future of student services and the changing campus experience including SCUP, Educause, and Tradelines.

Margaret Walsh is a strategist at brightspot strategy. With her background in ethnographic research in cross-cultural digital design contexts, she has been conducting research the evolving definition of “student experience” in order to address the 21st Century student’s needs.

Abstract:

Adam Griff

Margaret Walsh

Biography:

Biography:

21



This presentation will discuss how I integrate, analyze and report on administrative data through Excel macros to support program review and assessment. Administrative data allows us to contextualize the assessment of student learning and to address questions of institutional effectiveness. There are well documented challenges in the use of administrative data for research-based evaluation, hence considerable cleaning and processing are needed before they are ready for use. Wide variations in data, information and technological literacy among administration, faculty and staff call for simple and intuitive reporting systems requiring a short learning curve. Excel macros is an affordable and cost-effective tool to process administrative data when we do not have optimal conditions regarding technology, staff and funding. I will argue that integrating and analyzing administrative data provide a more global perspective on programs which can uncover important patterns potentially affecting student learning.

Dr. Mariajosé Romero is a senior researcher with strong multidisciplinary training and experience in research-based program evaluation and assessment in higher education. She has extensive experience in quantitative and qualitative research-based evaluations of educational programs, school-based interventions, and education and arts and sciences programs in the U.S. and abroad. Dr. Romero has been involved exclusively in institutional research, assessment and accreditation in higher education for the past seven years.

Title: Using Administrative Data to Support Program Review and Assessment

Presenter: Mariajosé Romero

Abstract:

Biography:

Mariajosé Romero

22


Concurrent Session 3: 2:30pm-3:15pm T itle: Simulation-based Assessment Identifies Longitudinal Changes in Cognitive Performance in the UF Anesthesiology Residency Training Program

Avner Sidi is an Anesthesiologist and has practiced Anesthesia for more than 30 years both in Israel and in the University of Florida College of Medicine. In addition to his clinical activity, his passion and main research interest in the last decade has been to explore and understand teaching and assessment especially as it relates to non-technical skills. His teaching activity in our Anesthesiology department includes working with the residents to improve their performance in simulation environments of the operating room (OR), trauma, and resuscitation.

Deficiencies in anesthesiology resident cognitive performance have previously been identified across three clinical domains (operating room, trauma, and cardiac resuscitation). Our main objectives were to evaluate higher-order, cognitive performance and to demonstrate that simulation can effectively serve as an assessment of cognitive skills and can help detect “higher-order” deficiencies, which are not as well identified through more traditional assessment tools. We hypothesized that simulation can identify longitudinal changes in cognitive skills and that cognitive performance deficiencies can then be remediated over time.

In this session we describe observed improvements in non-technical or “higher-order” deficiencies and cognitive performance skills in an anesthesia residency cohort for a 1-year time interval. Individual basic knowledge and cognition performance in simulation-based scenarios were assessed in 48 residents (PGY-3 and 4) using a 20- to 27-item scenario-specific checklist. Identical or similar scenarios were annually repeated by the same residents 18times. For every scenario and item, we calculated: group error scenario rate (frequency) and individual (resident)item success. Grouped individuals’ success rates are presented as mean ± SD and group error rates arepresented as proportions. For all analyses, alpha was designated as 0.05.

Presenter: Avner Sidi

Abstract:

Avner Sidi

Biography:

23

Location: Classroom

Student Panel – 3:45pm-4:30pm

Val Raymond is a fourth year Economics major from Port-au-Prince, Haiti. Val will be graduating from UF in May and returning to UF in August 2019 to begin a Master of Arts in Economics degree. After completing his Masters degree, Val hopes to pursue work in social and economic public policy. Cindy Jara is a third year studying public relations and criminology. Born in Colombia and raised in Central Florida, Cindy is a first-generation student who loves social media as much as social justice. She found her calling in helping underrepresented communities through inclusive advocating and hopes to attend law school to further advocate for underrepresented communities. Ryan Walsh is a fourth year Gator pursuing a degree in International Studies with a concentration on the Middle East. He is a first-generation college student from Daytona Beach, Florida. After graduation, Ryan hopes to begin a career in foreign policy advocating for human rights, conflict prevention, and international development. Sianne Powe is a senior studying Agricultural Education and Communication. Upon graduation, she plans to combat educational inequity as a middle school teacher with Teach for America in Houston, TX. She will also be working toward a Masters degree in education before pursuing a long-term career in environmental education through nonprofits.

Val Raymond Cindy Jara Ryan Walsh Sianne Powe

Guest Panelists: Val Raymond, Cindy Jara, Ryan Walsh, and Sianne Powe

Biographies:

24

Excellence in Assessment: The Student Perspective

Location: Ballroom

UFIT Exceptional Course Development Award

The UFIT Exceptional Course Development Award recognizes instructors who have made an outstanding contribution to student learning through intensive effort on course design and implementation. The winner and runner-up for the 2017-2018 academic year were chosen from among the 99 courses that received instructional design support from the UFIT Center for Instructional Technology and Training (CITT). Dr. Shannon Dunn, Interim Manager of CITT, will present the award to Dr. Clarence Gravlee (Associate Professor in Anthropology) and Elizabeth Ibarrola (Doctoral Candidate in Anthropology) for co-designing ANT3451 Race and Racism, as an online course. Together they created a student-focused learning community, providing strong instructor presence and facilitating critical thinking about the very challenging topic of race and racism. The award committee was also impressed by HUM4930/IDH3931 Music and Health and wishes to additionally recognize Ferol Carytsas (Lecturer in the Center for Arts in Medicine) as the runner-up. She received powerful student praise for the active learning strategies in her hybrid course design and revised many of her assessments to infuse more best practices, such as scaffolding and peer review.

Recipients: Dr. Clarence Gravlee, Elizabeth Ibarrola, Ferol Carytsas

25

Location: Ballroom

Timothy S. Brophy

Timothy S. Brophy is Director of Institutional Assessment at the University of Florida and Professor of Music Education. He holds a PhD in Music Education from the University of Kentucky, a Master of Music degree from the University of Memphis, and a Bachelor of Music Education degree from the Cincinnati College-Conservatory of Music. Prior to his appointment as Director, he was Assistant Dean for Research, Technology, and Administrative Affairs in the College of the Arts and a member of the music education faculty in the School of Music. He joined the University of Florida faculty in 2000. Dr. Brophy holds national and international leadership roles in assessment, has extensive experience with test development and data collection processes for standards-based assessments, large-scale assessment structures and processes, policy, and state and national assessment issues. At the University of Florida, Dr. Brophy is responsible for all matters related to the university’s accreditation with the Southern Association of Colleges and Schools Commission on Colleges (SACSCOC), and for establishing, maintaining, and refining the university’s institutional effectiveness and assessment processes. He advises the Provost of any recommended policy or procedural modifications related to assessment or institutional effectiveness, maintains communication with the university community regarding accreditation, assessment, and institutional effectiveness matters, and serves as Chair of the Academic Assessment Committee. He served on the SACSCOC Board of Trustees from 2011-2017 and is the University of Florida’s SACSCOC liaison. He currently serves as chair for SACSCOC on-site and off-site review committees.

Timothy S. Brophy

26

Professor and Director of Institutional Assessment

Nick Tharp: Conference Assistant I am a graduate student at the University of Florida who is enrolled in the Master of Science in Management program at the Warrington College of Business and in the graduate certification program in social media from the College of Journalism and Communication. I received a Bachelor’s degree in Communication with a focus in Public Relations from the University of North Florida. I plan to start an MBA program in marketing in 2020 after earning my current Master in Management. With a wide array of work history including various internships at the Florida Senate, Brooksource (IT recruiting), and various leadership roles at university campuses, I have been able to engage in numerous professional growth opportunities. My role as the 3rd Annual University of Florida Assessment Conference assistant and Smathers Graduate Intern has provided additional management, marketing, and online materials development experiences at the University of Florida. I have worked on the development of an updated open source journal, created a journal logo, complied and formatted the conference program book, and other key materials during the course of my internship. I am grateful for the opportunity to collaborate and learn from numerous faculty members who have generously shared their knowledge and would not trade this unique experience for anything. I believe that internships enhance one’s education through real world experiences that challenge and develop a student into a stronger candidate for his or her future. I am forever grateful to be a part of planning, developing, and the implementation of today’s conference.

Nick Tharp

27

Documents

A s s e s s m e n t i n H i g h e r E d u c a t i o n : E ... · Assessment Considerations for Internship Programs in a Time of Social Justice Expectations ~Laura Spears How to Better