Upload
feng
View
37
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Transformative Assessment. Joan K. Lippincott Coalition for Networked Information Gary Brown Washington State University. Transformative Assessment at Washington State University. Gary Brown, Director The Center for Teaching, Learning, & Technology [email protected]. - PowerPoint PPT Presentation
Citation preview
Transformative AssessmentTransformative Assessment
Joan K. LippincottCoalition for Networked Information
Gary BrownWashington State University
Transformative AssessmentTransformative Assessmentat
Washington State UniversityWashington State University
Gary Brown, DirectorThe Center for Teaching, Learning, & TechnologyThe Center for Teaching, Learning, & [email protected]
Transformative Assessment PracticesTransformative Assessment Practiceshttp://www.educause.edu/TransformativeAssessment/2599
• A program initiated by:– EDUCAUSE/National Learning Infrastructure Initiative
(NLII)– Coalition for Networked Information (CNI)– The Teaching, Learning, & Technology Group (TLT
Group)– The Center for Teaching, Learning, & Technology
at Washington State University• Elicit new ideas about assessment• Help institutions put these ideas into practice• Map assessment to assessment
Dimensions of Transformation
1. Purpose of Assessment• Outcomes not coverage (Learning not Teaching)
• Alignment
2. Data• Learning & Learning Skills
3. Application• Systematic
• Integrative
• Closes the loop
4. Dissemination• Embrace new collaborations & new constituencies
AdministrativeAdministrative
ProgressiveProgressive
TransformativeTransformative
The purposeful gathering and use of data and data analysis to ensure that the application of findings and the dissemination of results will help align, substantially change and enrich the student learning experience.
Transformative Assessment is…
What is Transformative Assessment?
Not TransformativeNot Transformative Transformative
4. Student Evaluations Learning Process Evaluations
5. Tests Outcomes & Collective Rating
3. Selectivity Measures Equity Measures
2. Old Accreditation New Accreditation
Trans-course Collaborations1. Technology
TechnologyTechnology
Enrollments in WSU Online Learning SpacesEnrollments in WSU Online Learning Spaces
0
10000
20000
30000
40000
50000
60000
70000
1998 1999 2000 2001 2002 2003 2004
Pedagogy & Technology
Good to Great (Collins, 2001,Harper Business)
• “Technology and technology-driven change has virtually nothing to do with igniting a transformation from good to great.”
• “Yet, paradoxically, great organizations are pioneers in the application of carefully selected technologies.”
Transformative…Transformative… Trans-Course CollaborationTrans-Course Collaboration
(using technology)(using technology)
Online Decision ScienceOnline Decision Science
• Saved a dairy plant $250,000• Saved the plant• Received $95,000 from company executives
to complete a study on call-center efficiency.– And incentives from the company resulted in
more employees enrolling in the course. • Managed pantry inventory for a house
of teens
In A Human Development ClassIn A Human Development Class• “This information [researched &
reported by students] was better than text books that are obsolete as soon as they are printed.”
• “The students in this course provided up to date pertinent information about current issues and how to affect change.”
• “The next group of students who take this class will write their own text book.”
NOTNOT
Transformative…Transformative… Old AccreditationOld Accreditation
The Good Old Days
My Class Your Class His Class Her Class Another Class
1st Year Content
2nd Year Content
3rd Year Content
4th Year Content
Capstone Content
The Good Old Days
My Class Your Class His Class Her Class Another Class
1st Year Content
2nd Year Content
3rd Year Content
4th Year Content
Capstone Content
Transformative…Transformative… Accreditation Accreditation
Focus onFocus on
Closing the LoopClosing the Loop
Outcomes—A Science ProgramOutcomes—A Science Program
• Faculty assessed competency of 60 papers collected from 11 different courses of their program.
• Maintained inter-rater reliability of 75%
Why Worry About Consensus?Why Worry About Consensus?
• Accreditation– High standards are not the same as
standardization…• Legal
– Faculty content expertise trumps faculty exam design expertise.
• Curricular coherence– No single course will meet all
program/institutional outcomes.
0
1
2
3
4
5
6
Graduate Students Upper Division Lower Division
A Science Program OutcomesA Science Program Outcomes Fall PilotFall Pilot
2003
Professional Competency
Mastery
Developing
Emerging
Crisis and Opportunity
Closing the Loop
• Faculty agreed:–Assignments need to be revised to align
with the shared OUTCOMES.–Jointly create & embed assignments for
target points in curricula…– Include accreditors, professional
advisory board partners, and upper-level students in the process• Use an ePortfolio….
Round OneRound One(Of a Continuous Process)
“Working together to design assignments is unusual because faculty are accustomed to working independently. However, collaboration should enable improvement of individual assignments, thus the entire curriculum shall benefit.”
—Stephanie Clark, FSHN
NOT NOT
Transformative…Transformative… Selectivity MeasuresSelectivity Measures
An Inland Northwest Land GrantGOAL #1 Offer the Best Undergraduate Experience in a
Research UniversitySub goal 1
Attract, recruit, and retain a diverse high quality student body.
Required actions Implement recruitment and admissions strategies that reach and serve high ability students from high schools and community colleges.
What DoesWhat Does
Institutional Selectivity Institutional Selectivity Tell Us AboutTell Us About Educational Quality? Educational Quality? “National magazines that purport to
identify the nation’s ‘best’ colleges are essentially ranking institutions by their selectivity, not by the likelihood of their exposing students to the most effective educational practices.”
“Given the challenges facing the nation and the higher education system, it’s time for some straight talk about the deleterious grip that selectivity has had on our perceptions of what constitutes collegiate quality. It’s bad public policy and educationally indefensible.”
“It would be much more productive to focus on developing indicators that more accurately represent what happens to students during college and to make this information available in a responsible way so that prospective students, policymakers, and institutional leaders can use it to make decisions that can improve student learning.”
George D. Kuh and Ernest T. Pascarella (Change, September/October 2004 pp.53-58)
The Center for Urban EducationEstela Mara Bensimon, Director, CUE
Equity Frame Color-conscious, awareness of racialized practices, personal and collective responsibility/accountability for outcomes, focus on outcomes
Diversity Frame Visibility of racial and ethnic diversity, diversity as an institutional property, cross-racial relationships, color blind, focus on access
Deficit Frame Underperformance is person-centered & attributed to membership in a particular group; “at risk,” lack of motivation, lack of socialization, missionary attitude
Mental ModelsDifferent Questions
Diversity Questions Equity Questions
In what ways can we attract more students of color to our campus?
In what ways can we improve the educational outcomes of historically underrepresented students?
In what ways can we increase students’ understanding of cultural, racial, and ethnic differences?
In what ways can we—as faculty members, deans, counselors—be more accountable for the outcomes of historically underrepresented students?
NOT…NOT… Traditional Student EvaluationsTraditional Student Evaluations
Traditional Student EvaluationsAdministrative Progressive Transformative
Purpose
Data
Application
Dissemination
Rank Instructors
Student Satisfaction
RankInstructors
Between Chair & Faculty
The GOLDGOLD StandardParticipants Who Used Survey Data to Inform Change….
257
106
5516 2
0
100
200
300
Invited faculty Faculty who tookinstructor survey
Faculty w/ studentresponses
Faculty w/ >50%student response
rate
Faculty who usedthe data
Transformative…Transformative… Learning Focused Evaluations
Student Profile
CourseQuestions
Learning Environment
Critical Engagement
Relevance
College Questions
Department Questions
The TLT Group BeTA Project:The TLT Group BeTA Project:
Student Evaluations
Peer Review
Faculty DevelopmentSeminar Series
Validation
Outcomes Development(Rubrics)
Student Evaluations
Validation
Peer Reviews&
Focus Groups
0123456
Your Goals Your Results
Reporting Results
Evaluation as Community Practice
Administrative Progressive Transformative
Purpose
Data
Application
Dissemination
Community of Evidence
DeepenOutcomes
Public Seminars
Trans-Institutional& Community
SUNSHINE
LOGO
Bringing it TogetherBringing it Together
• Goal, Assessment, Assignment & Curricula Alignment
• Collective Community Engagement:– Activities, projects are co-created– Performance criteria are shared with students
• Technology in Community Context – ePortfolio– Blog-based Reflection
NOT…NOT… TestsTests
Transformative…Transformative… Outcomes Outcomes
& & Collective JudgmentCollective Judgment
Assignment Assessment & the Echo EffectAssignment Assessment & the Echo Effect
0
1
2
3
4
5
6
Faculty Assignment Student Critical Thinking
Mastery
Developing
Emerging
From the Messy NSSEGeorge Kuh
• “Students typically don’t exceed their own academic expectations.”
• But “Students will go beyond what they think they can do under certain conditions, especially when teachers expect, challenge, and support them.”
• “Are we willing to make the effort that such practices demand of us?”
Final ThoughtFinal Thought
Transformation is not an event…
IT’s a Lifestyle!
Ten Lessons from TAPTen Lessons from TAP
Joan K. Lippincott
Coalition for Networked Information
TAP ProgramTAP Program
• Focus Sessions
• READI Branch
• Rubric
• Case studies
• Report
What Did We Learn?What Did We Learn?
• Not many institutions are implementing transformative assessment programs
• Most institutions understand the need for better assessment practices
• Higher education is subject to pressures for accountability
1: Why Assess?1: Why Assess?
• Improve teaching and learning• Provide evidence of the value of the investment
in technology for teaching and learning– Produce students who can contribute to the
knowledge economy– Contribute to community development– Extend benefits of higher education to more sectors of
the society
2: Understand What 2: Understand What You Want to AssessYou Want to Assess
• Locate and review institutional documents
• Choose large issues that have importance for your institution
• Align the assessment goals and objectives with institutional goals and objectives
Institutional GoalsInstitutional Goals
Align Goals and AssessmentAlign Goals and Assessment
3: Work Towards an Institutional 3: Work Towards an Institutional Culture of AssessmentCulture of Assessment
• Assessment is embedded in major institutional processes such as strategic planning, budgeting, curriculum review
• Studies inform others’ work across the institution
• Incentives are available
4: Seek Support of Top Administrators
• Need for vision and mission related to teaching, learning, and technology
• Need for resources
• Need for action based on assessment findings
5: Work with Assessment Experts5: Work with Assessment Experts
• Campus– Institutional Research or Planning– Undergraduate Education– Center for Teaching and Learning
• Consultants– Flashlight Program, TLT Group
6: Assemble Stakeholders6: Assemble Stakeholders
• Academic administrators
• Faculty senate
• Center for Teaching Excellence
• IT/Library/Instructional Technology
• Assessment Experts
• “Risk-takers and consensus-seekers” (Carmean)
7: Consider Qualitative and 7: Consider Qualitative and Quantitative MeasuresQuantitative Measures
• “Triangulation”
• The power of stories
“This summer, I worked for a biomedical engineeringCompany… and I used what I learned from the class… I had a problem to solve, which required an immense amount of background research, which had to be organized in an effective manner. I knew exactly how to acquire reputable resources, document them, and put them together in a literature review. I used every aspect of the Library Studies class, and my employers were very impressed with my ability…”
Student quoted in Snavely and Wright, 2003
8: Take Advantage of Others’ 8: Take Advantage of Others’ Tools and ExperiencesTools and Experiences
• National Survey of Student Engagement (NSSE)
• AAHE Assessment Forum
• TLT Group Flashlight Project
• ACRL Information Literacy Standards
9: Be Persistent9: Be Persistent
“Transformative assessment practice lies in ruthless inquiry and a determined commitment of resources.”
Colleen Carmean, “Assessment Boot Camp,” 2003
10: Use What You Have Learned 10: Use What You Have Learned to Improve Your Institutionto Improve Your Institution
• Use findings to systematically inform and reshape teaching and learning practice and to promote a culture of evidence
NLII Key Theme:NLII Key Theme:Alignment & AssessmentAlignment & Assessment
http://www.educause.edu/Alignment/934