Upload
others
View
2
Download
1
Embed Size (px)
Citation preview
Who Owns Student Success? A Case for Institution-wide
Engagement
Steve Robbins, Director, Research Innovation , Center for Academic and Workforce Readiness and Success, Educational Testing Service, Princeton, NJ
Kevin Li, Dean of Instruction, Wilbur Wright College, Chicago, IL
Copyright © 2013 by Educational Testing Service. All rights reserved.
2 Copyright © 2013 by Educational Testing Service. All rights reserved.
Consultant in higher education and Executive Director of The Leadership Board for CIO's in higher education. Dr. Zastrocky examines and analyzes emerging higher education IT trends and assesses the implications of those trends on educational institutions globally. Dr. Zastrocky has served as Managing Vice President, Education Research Leader and Research Director for Academic Strategies at Gartner Inc., a faculty member at several universities, assistant dean, chief information officer, vice president of CAUSE, and vice president of the Kaludis Consulting Group.
Dr. Michael Zastrocky
Moderator
3 Copyright © 2013 by Educational Testing Service. All rights reserved.
Steven Robbins
Speaker
Steven Robbins is Director of Research Innovation in the Center for Academic and Workforce Readiness and Success at Educational Testing Service in Princeton, N.J. He received his Ph.D. degree in counseling psychology and his M.S. degree in rehabilitation counseling — both from the University of Utah. He received his B.A. degree in psychology from the University of Illinois. Before joining ETS, Robbins was Vice President for Research at ACT, Inc.
4 Copyright © 2013 by Educational Testing Service. All rights reserved.
Kevin Li holds a Master of Arts degree in economics and is a Ph.D. candidate at the University of Illinois at Chicago in the field of educational psychology. He has served in various roles in the City Colleges of Chicago for the past fifteen years. Since 2009, he has served as Dean of Instruction at Wilbur Wright College, one of the City Colleges of Chicago. Li is responsible for overseeing and coordinating the college’s accreditation process.
Kevin Li
Speaker
• 50 percent of community college
students need remedial – or below
college level – courses
• 20 percent of 4-year college students
also start in remediation
• 40 percent of these students –
frustrated by their low placement –
don’t even bother to enroll in classes
But many are unprepared
5 Copyright © 2013 by Educational Testing Service. All rights reserved.
More than 3.4 million students begin college each year.
• There is convincing evidence that
many college students are not
prepared for college and don’t have
the behavioral skills needed for
success.
• “Preparedness” encompasses a
holistic look at both cognitive and
noncognitive skills.
First-year college students are often confused and overwhelmed by the complexity of navigating their college experience
6 Copyright © 2013 by Educational Testing Service. All rights reserved.
And the path to college-level work has detours.
7 Copyright © 2013 by Educational Testing Service. All rights reserved.
The Challenges
• Students are not able to complete credit bearing general education courses in a timely fashion
• Even incremental success in improving student retention a success; 1% improvement means more students succeed
• The swirl (students coming and leaving) reflects the increasing complexity of student goals and barriers to success
• Institutions must reach across natural silos and structures to effectively drive student success
8 Copyright © 2013 by Educational Testing Service. All rights reserved.
The Solution
How do we . . .
• Strategically accelerate students into their credit bearing courses?
• Consider individual differences in background, skills, abilities, and experiences to promote success?
• Develop and target sound and practical interventions?
• Best disseminate that information to students, faculty, staff, and administrators?
9 Copyright © 2013 by Educational Testing Service. All rights reserved.
The Solution: From the “so what” to the “now what”…
Holistic Assessment
Data Integration & Use Across Silos
Student Commitment & Classroom Success
10 Copyright © 2013 by Educational Testing Service. All rights reserved.
Adding a Noncognitive Assessment Allows for a Holistic View
• Respects the whole student (examines cognitive, noncognitive, behavioral factors)
• Acknowledges the unique individuality of students
• Focuses on factors educators can control and empower students to modify: motivation, social connectedness, as opposed to socio-economic, situational factors
• A common language, equitable model, for early warning, advising, and acceleration decisions
11 Copyright © 2013 by Educational Testing Service. All rights reserved.
Domain Facet Definition Example Items
Academic Skills Tools and strategies for academic success
Organization Strategies for organizing work and time. I make a schedule for getting my school work done.
Meeting Class Expectations
Doing what’s expected to meet the requirements of your course including assignments and in-class behaviors.
I attend almost all of my classes.
Commitment Active pursuit toward an academic goal
Commitment to College Goals
Perceived value and determination to succeed in and complete college.
One of my life goals is to graduate college.
Institutional Commitment
Attachment to and positive evaluations of the school.
This is the right school for me.
Self-Management Reactions to academic and daily stress
Sensitivity to Stress
Tendency to feel frustrated, discouraged, or upset when under pressure or burdened by demands.
I get stressed out easily when things don't go my way.
Academic Self-Efficacy
Belief in one’s ability to perform and achieve in an academic setting.
I'm confident that I will succeed in my courses this semester.
Test Anxiety General reactions to test-taking experiences, including negative thoughts and feelings (e.g., worry, dread).
When taking a test, I think about what happens if I don't do well.
Social Support Connecting with people and students resources for success
Connectedness A general sense of belonging and engagement.
I feel connected to my peers.
Institutional Support
Attitudes about and tendency to seek help from established resources.
If I don't understand something in class, I ask the instructor for help.
Barriers to Success
Financial pressures, family responsibilities, conflicting work schedules, and limited institutional knowledge.
Family pressures make it hard for me to commit to school.
Risk Indices: Separate indices for both classroom and enrollment success. Based on background, cognitive , and psycho-social information and supported by statistical relationships with success.
Domain Scores: Four general areas of student strengths and weaknesses. Scores are presented normatively.
Feedback: Determine by more specific “facet” scores (see next page).
Action Plans: Suggested interaction with programs and services on campus.
Background Information: Communicate key student information from both SuccessNavigator and SIS to faculty/advisor.
Academic Success: Classroom success broken into ability to meet expectations and organizational skills.
Self-management: Ability to handle stress, testing anxiety and academic self-efficacy.
.
Social Support: Tendency to seek help, attitude toward barriers to success and ability to relate to peers in school.
Commitment: Measures both commitment to college and commitment to the specific institution.
Biographical Background Information: Student-reported background information for use in context of advising.
Education plans: Helps advisor understand student’s commitment to college, goals.
Education Background Information: Communicates key student information about high school performance.
Influences : Understand outside influences that may affect student performance, yet go unmentioned.
15 Copyright © 2013 by Educational Testing Service. All rights reserved.
Early Intervention is key
Three basic implications: • Educators have a profound influence on students
– Faculty can motivate and inspire students
– Educators can also shape and foster their intentionality and academic determination
• Observable behaviors are indicative of internal states
– E.g., Motivation, engagement
• Educators have the opportunity and capability to guide students towards desirable classroom/ academic behaviors
16 Copyright © 2013 by Educational Testing Service. All rights reserved.
Data Integration and Effective Coordination
Rethinking institutional enterprise systems:
• A data warehouse that populates by student rather than transaction
• Ability to connect student information across critical decision points, whether course scheduling and placement, advising, classroom monitoring, or semester and yearly status
• A vehicle for communicating across groups: testing, faculty, student, advising
• Business Intelligence capability to support dashboards and reporting
17 Copyright © 2013 by Educational Testing Service. All rights reserved.
TACTICAL
Holistic Placement Decision Support Systems
SuccessNavigator
Profile
Student Education
Plan
Student Bio,
Registration
Student Active
Course Information
Success Navigator
GradesFirst PeopleSoft Blackboard
Advisor Feedback
Report
Accelerated Students # of Students
% of Students
Non-Accelerated Students # of Students
% age of Students
Non-SuccessNavigator – College Advising Interventions
253 79% Non-Accelerated College Advising Interventions
10,775 85%
INFORMA- TIONAL
STRATEGIC
Data Warehouse Reports
Reports Business
Intelligence (Zogotech)
18 Copyright © 2013 by Educational Testing Service. All rights reserved.
What we have learned at CCC
• Evidence-based solutions
• ETS/CCC Collaboration Objectives
• Lessons Learned and Next Steps
19
Noncognitive Assessment Experience at Wright College
semester
Note. Results from Wright College (Spring 2009).
n = 453; All paths are statistically significant.
Copyright © 2013 by Educational Testing Service. All rights reserved.
20
High Medium Low
Math Readiness
High 96% 91% --
Medium 94% 77% 82%
Low 86% 61% 38%
n=411
Developmental Education Class Success Rates Based on Math Readiness and In-Course Behavior Levels
Success rate not
calculated due to
insufficient sample size
Effort level was measured as whether a student 1) actively participated in group work 2) actively participated in lecture 3) attended class 4) completed homework
Success rates for students with low readiness but high effort more
than twice as high as for low effort, low readiness
Future area of focus
Research shows that behavioral monitoring and intervention are key to student success
Copyright © 2013 by Educational Testing Service. All rights reserved.
ETS/CCC Collaboration Objectives
• Combine COMPASS and Psychosocial Measure to accelerate student progress and college completion
– Students placed within the upper-band with high psychosocial scores can enroll in the next level course
• Allow more individualized, actionable plans for students, tied to curricular goals
• Noncognitive survey and in-class behavioral ratings will be used to trigger Early Alert and intervention strategies
• Create supplementary services tied to Early Alert and to behavioral monitoring
21
Copyright © 2013 by Educational Testing Service. All rights reserved.
22 Copyright © 2013 by Educational Testing Service. All rights reserved.
Resource Library
• Also available to advisors, the Resource Library will include such topics as:
– Guidelines for working with students
– Sample advising scripts
– Helping a student choose a planner/calendar
– Goal setting worksheets
– Costs and benefits of a college degree
– Conflict resolution tips
– Financial/budget planner for students
– Communication skills quick reference
– Anxiety symptoms sheet
23 Copyright © 2013 by Educational Testing Service. All rights reserved.
Comprehensive Early Interventions
• Serve to model and coach at-risk students toward a set of desirable behaviors.
• Findings from Li et al. (2013 or in press) indicate that low-level developmental education math students can increase the likelihood of passing the course by 30% if they exhibit desirable in-class behaviors:
Active participation in group work
Actively engaged during group work Helps other students with assignments Does his/her fair share of the work
Active participation in lecture Displays alertness Attentive during class Asks and answers questions
Attendance Attends class Stays for whole period
Completion of homework assignments
Completes assignments thoroughly Submits assignments on time
24 Copyright © 2013 by Educational Testing Service. All rights reserved.
Recommendations: The 3 Pillars of Success
• Place students more accurately using multiple measures, including noncognitive skills for possible acceleration
• Smarter, strategic interventions: coordinate efforts to support the whole student
• Institutional commitment: Faculty, advisors, and staff collaborate in early alert program
• Generate fine-grained reporting to empower frontline workers and drive further research and program development
Holistic Assessment
Instructional and Advising Supports
Academic Readiness
25 Copyright © 2013 by Educational Testing Service. All rights reserved.
Next steps in Chicago
• CCC is balancing faculty, staff and administration perspectives to achieve buy-in and reach effective solutions to emerging problems.
• Coordination across instruction, advising and other support is critical. Classroom monitoring flags early alert for proactive advising.
• The need for integration of data that enhances process and resource planning.
26 Copyright © 2013 by Educational Testing Service. All rights reserved.
Leadership and Change: Final Thoughts and Perspective
• Who “owns” this change agenda?
• Can this system be replicated at other institutions?
• Likely challenges and issues to address.
For more information:
Web: www.ets.org/SuccessNavigator/chronicle
Phone: 1-800-745-0269
Email: [email protected]
Questions?
27 Copyright © 2013 by Educational Testing Service. All rights reserved.
28 Copyright © 2013 by Educational Testing Service. All rights reserved.
Useful Background References
Allen, J., & Robbins, S. (2010). Effects of interest-major congruence, motivation, and academic performance on timely degree attainment. Journal of Counseling Psychology, 57, 23-35.
Allen, J., Robbins, S., Casillas, A., & Oh, I.-S. (2008). Third-year college retention and transfer: effects of academic performance, motivation, and social connectedness. Research in Higher Education, 49(7), 647-664.
American Association of Community Colleges (AACC). (2009). Community college fast facts. Retrieved July 10, 2009 from http://www2.aacc.nche.edu/research/index.htm
Belfield, C. R., & Crosta, P. M. (2012). Predicting success in college: The importance of placement tests and High school transcripts. CCRC Working Paper No. 42. New York: Community College Research Center, Teachers College, Columbia University.
Berkner, L., & Choy, S. (2008). Descriptive summary of 2003-04 beginning postsecondary students: Three years later. Washington, DC: U.S. Department of Education, Institute of Education Sciences.
Bragg, D. D. (2001). Community college access, mission and outcomes: Considering intriguing intersections and challenges. Peabody Journal of Education. 76(1), 93-116.
Community College Research Center. (2002a). Accountability and learning outcomes in community colleges. Retrieved July 7, 2008, from http://ccrc.tc.columbia.edu/Seminar.asp?uid=8
Community College Research Center. (2002b). Accountability and learning outcomes in community colleges. Retrieved July 7, 2008, 2008, from http://ccrc.tc.columbia.edu/Seminar.asp?uid=8
29 Copyright © 2013 by Educational Testing Service. All rights reserved.
References (cont.)
Habley, W., Bloom, J, & Robbins, S. (2012). Increasing persistence: Research-based strategies for college student success. San Francisco: Jossey-Bass
Habley, W., & McClanahan, R. (2004). What works in student retention – All survey colleges. Iowa City, IA: ACT, Inc.
Harris, S. (1991) Can’t you guys read? Cartoons on academia. New Brunswick, NJ: Rutgers University Press.
Herzog, K., & Miller, M. P. (Eds.). (1985). Scholarship: More great cartoons from the Kappan. Bloomington, IN: Phi Beta Kappa International Inc.
Le, H., Casillas, A., Robbins, S., & Langley, R. (2005). Motivational and skills, social, and self-management predictors of college outcomes: Constructing the Student Readiness Inventory. Educational and Psychological Measurement, 65, 482-508.
Li, K., & Cordes, C. (2010). Individualizing the Delivery of Math Developmental Education. Talk presented at the 34th Annual National Association of Developmental Education Conference, Columbus, OH.
Li, K., Zelenka, R., Buonaguidi, L., Beckman, R., Casillas, A., Allen, F., Crouse, J., Hanson, M. & Robbins, S. (In press). Modeling Effects of Math Readiness and Student Behavior on Foundational Skills Course Success. Journal of Developmental Education.
Lotkowski, V., Robbins, S., & Noeth, R. (2004). The role of academic and non-academic factors in improving college retention. Retrieved from ACT, Inc. website: www.act.org/research/policy/index.html.
30 Copyright © 2013 by Educational Testing Service. All rights reserved.
References (cont.)
Markle, R., & Robbins, S., (2013). Breaking the developmental education logjam: A holistic approach to acceleration. Princeton, NJ: Educational Testing Service.
Markle, R., & Robbins, S., (2013). A holistic view of course placement fecisions – Avoiding the HS GPA trap. Princeton, NJ: Educational Testing Service.
McCabe, R. H. (2000). No one to waste. Washington, DC: Community College Press.
NCES (2004). Remedial education at degree-granting postsecondary institutions in fall 2000 (NCES 2004-010).
NISOD (2012, March). Assessing college-readiness assessments. Webinar delivered at the annual NISOD Conference.
Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: A third decade of research. San Francisco: Jossey-Bass.
Peterson, C. H., Casillas, A., & Robbins, S. B. (2006). The Student Readiness Inventory and the Big Five: Examining social desirability and college academic performance. Personality and Individual Difference, 41, 663-673.
Porchea, S., Allen, J., Robbins, S., & Phelps, R. (in press). Predictors of long-term enrollment and degree outcomes for community college students: Integrating academic, psychosocial, socio-demographic, and situational factors. Journal of Higher Education.
President’s Commission on Higher Education. (1947). Organizing higher education. Washington, DC: Author.
31 Copyright © 2013 by Educational Testing Service. All rights reserved.
References (cont.)
Robbins, S. B., Allen, J. Casillas, A., Akamigbo, A., Saltonstall, M., Campbell, R., Mahoney, E., & Gore, P.A. (2009). Associations of resource and service utilization, risk level, and college outcomes. Research in Higher Education, 50(1), 101-118.
Robbins, S., Allen, J., Casillas, A., Peterson, C., & Le, H. (2006). Unraveling the differential effects of motivational and skills, social, and self-management measures from traditional predictors of college outcomes. Journal of Educational Psychology, 98, 598-616.
Robbins, S. B., Lauver, K., Le, H., David, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261-288.
Robbins, S., Oh, I., Le, H., & Button, C. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta-analytic path analyses. Journal of Applied Psychology, 94, 1163-1184.
Scott-Clayton, J. (2012). Do High-Stakes Placement Exams Predict College Success? Predicting Success in College: The Importance of Placement Tests and High School Transcripts. CCRC Working Paper No. 41. New York: Community College Research Center, Teachers College, Columbia University.
Swail, W. S. (2004, January 23). Legislation to improve graduation rates could have the opposite effect. The Chronicle of Higher Education, B16.