View
215
Download
0
Category
Tags:
Preview:
Citation preview
WASC Assessment Leadership Academy
Oakland, California
August 1, 2011
Presentation by
Trudy W. BantaProfessor of Higher Education
andSenior Advisor to the Chancellor forAcademic Planning and Evaluation
Indiana University-Purdue University Indianapolis355 N. Lansing St., AO 140
Indianapolis, Indiana 46202-2896tbanta@ iupui.edu
http://www.planning.iupui.edu
Outline
• The Great Testing Debate
• Alternatives to standardized tests of generic skills
© TWBANTA-IUPUI
© TWBANTA-IUPUI
Group Assessment Has Failed toDemonstrate Institutional Accountability
• Focus on improvement at unit level
• Rare aggregation of data centrally
• Too few faculty involved
• HE scholars focused on K-12 assessment
© TWBANTA-IUPUI
2006Commission on the Future of
Higher Education
We need a simple way to compare institutions
The results of student learning assessment, including value added measurements (showing skill improvement over time) should be . . . reported in the aggregate publicly.
My History
Educational psychology
Program evaluation & measurement
Performance funding in Tennessee
1990 USDOE effort to build a national
test
1992 Initiated evidence-based culture
at IUPUI
© TWBANTA-IUPUI
2007
Voluntary System of Accountability
~ Assessment of Learning ~
defined as
critical thinking, written communication, analytic reasoning
© TWBANTA-IUPUI
VSA Recommendations(over my objections)
Collegiate Assessment of Academic
Proficiency (CAAP)
Measuring Academic Proficiency &
Progress (MAPP)
Collegiate Learning Assessment (CLA)
(College BASE)
© TWBANTA-IUPUI
© TWBANTA-IUPUI
TN = Most Prescriptive(5.45% of Budget for Instruction)
1. Accredit all accreditable programs (25)
2. Test all seniors in general education (25)
3. Test seniors in 20% of majors (20)
4. Give an alumni survey (15)
5. Demonstrate use of data to improve (15)
___
100
© TWBANTA-IUPUI
At the University of Tennessee
CAAP
Academic Profile (now MAPP)
COMP (like CLA and withdrawn by 1990)
College BASE
© TWBANTA-IUPUI
In TN We Learned
1) No test measured 30% of gen ed skills
2) Tests of generic skills measure primarily prior learning
3) Reliability of value added = .1
4) Test scores give few clues to guide improvement actions
An Inconvenient Truth
.88 = correlation of VSA – recommended
test scores with SAT/ACT scores
thus
77% of the variance in institutions’ scores
Is due to students’ prior learning
© TWBANTA-IUPUI
© TWBANTA-IUPUI
How Much of the Variance in Senior Scores is Due to College Effects?
• Student motivation to attend that institution (mission differences)
• Student mix based on • age, gender • socioeconomic status • race/ethnicity • transfer status • college major
© TWBANTA-IUPUI
How Much of the Variance in Senior Scores is Due to College Effects?
(continued)• Student motivation to do well• Sampling error• Measurement error• Test anxiety College effects
______
23 %
Threats to Conclusions Based on Test Scores
1. Measurement error
2. Sampling error
3. Different tests yield different results
4. Different ways of presenting results
5. Test bias
6. Pressure to raise scores- Daniel Koretz
“Measuring Up”
Harvard U. Press - 2008
© TWBANTA-IUPUI
Using NSSE
95% variance in responses
is WITHIN Institutions
- Pike, Kuh, McCormick, Ethington & Smart (2011)
© TWBANTA-IUPUI
© TWBANTA-IUPUI
Student Motivation
• Samples of students are being tested
• Extrinsic motivators (cash, prizes) are used
We have learned:
• Only a requirement and intrinsic motivation will bring seniors in to do their best
Recent University of Texas Experience
30 – 40% of seniors at flagships earn
highest CLA score (ceiling effect)
flagship campuses have lowest value
added scores
© TWBANTA-IUPUI
Concerns About Value Added
• Student attrition• Proportion of transfer students• Different methods of calculating • Unreliability• Confounding effects of maturation
© TWBANTA-IUPUI
Word from Measurement Experts
Given the complexity of
educational settings, we may never be satisfied that value added models can be used to appropriately partition the causal effects of teacher, school, and student on measured changes in standardized test scores.
- Henry Braun & Howard Wainer
Handbook of Statistics, Vol. 26: Psychometrics
Elsevier 2007
© TWBANTA-IUPUI
Employing currently available standardized tests of generic
skills to compare the quality
of institutions is not a valid use of those tests.
© TWBANTA-IUPUI
© TWBANTA-IUPUI
Consequences of ScoresBelow Expectations?
Public criticism Loss of students Reduced funding Corrective actions imposed Prep courses for students Profits for coaching consultants
2009 NILOA SurveyCampus-wide Approaches to Assessment
1. A national survey (76%)
2. Standardized test of general skills (39%)
3. Portfolios, specialized tests, external judges
© TBANTA-IUPUI
OECD’s AHELOfor
HEIs from 15 countries
1. Generic skills (CLA)
2. Disciplines (Engineering and Economics)
3. Value added
4. Contextual information indicators
© TWBANTA-IUPUI
Designing Effective Assessment:
Principles & Profiles of Good Practice
Trudy W. Banta
Elizabeth A. Jones
Karen E. Black
Jossey-Bass (Wiley) 2009
© TWBANTA-IUPUI
Profiles
Invited over 1000
Received 146
Selected 49 for use in full
Categorized all 146 and published
Web sites
© TWBANTA-IUPUI
Outline for Profiles
Background and Purpose
Methods over ? Years
Resources Required
Findings
Use of Findings
Impact of Using Findings
Success Factors
Web sites© TWBANTA-IUPUI
Standardized tests
of generic skills
(e.g., writing, critical thinking)
used by just 8%
always supplemented
© TBANTA-IUPUI
Advantagesof standardized tests of generic skills
promise of increased reliability & validity
norms for comparison
© TWBANTA-IUPUI
Limitationsof standardized tests of generic skills
cannot cover all a student knows
narrow coverage, need to supplement
difficult to motivate students to take
them!
© TWBANTA-IUPUI
Better Ways to Demonstrate Accountability
Performance Indicators
1. Access (to promote social mobility)
2. Engaging student experience
3. Workforce development
4. Economic development
5. Civic contribution of students, faculty, staff, graduates
© TWBANTA-IUPUI
Effects of Education on SocialMobility
Relate data on
high school courses and completion
college courses and completion
to
Career placements & earnings
private and public employment
military enlistments
incarcerations
- Florida DOE
© TWBANTA-IUPUI
Australian DOE1. Increase participation of individuals
from low SES backgrounds
as undergraduates
as graduate students
2. Improve engagement & satisfaction
survey of student engagement
first year retention
satisfaction of completers
© TWBANTA-IUPUI
Workforce Development
• % graduates placed in field• % graduates placed locally• Degree, certificate, CE programs
aligned with regional priorities• Internships/coop programs• Increased earnings of graduates
- APLU
© TWBANTA-IUPUI
Economic Development• Creation of intellectual property• Patent and license awards• # start-up companies• Sponsored research $
% NSF, NIH• Use of academic facilities by industry• Alignment of assets to support regional
economic clusters
- APLU© TWBANTA-IUPUI
Civic Contribution Internships supported by institution Volunteer hours for students, faculty,
staff Service learning placements $ contributed to United Way et al. Pro bono legal & health services No-cost presentations to community
groups
© TWBANTA-IUPUI
Land-Grant Extension1. Useful information for
agricultural producers
small business owners
rural families
2. After-school STEM programs
3. Driver-ed for teen traffic offenders
4. Green jobs in energy fields
5. Sustainable agriculture practices
© TWBANTA-IUPUI
National Governors AssociationCenter for Best Practices
A simple graduation rate
penalizes colleges serving low
SES students
may discourage open enrollment
may lead to lower graduation
standards
© TWBANTA-IUPUI
National Governors AssociationCenter for Best Practices
Track intermediate student milestones
Successful completion of remedial
and core courses
Advancement from remedial to credit
courses
Transfer from 2-to 4-year college
Credential attainment
© TWBANTA-IUPUI
Oregon Community Collegestrack:
% needing remedial courses % completing remedial or ESL courses # credits earned each year toward
degree or certificate Semester-to-semester and fall-to-fall
persistence
- Inside HE 10/12/09
© TWBANTA-IUPUI
Peter Ewell:
“ . . . shift state funding formulas so that colleges receive money based on how many students are still enrolled by the end of the academic term rather than at the beginning.”
- Inside HE 11/18/09
© TWBANTA-IUPUI
If We Must Measure LearningLet’s Use:
1. Standardized tests in major fieldslicensure and certification tests
ETS Major Field Tests
2. Internship performance
3. Senior projects
4. Study abroad performance
5. Electronic portfolios
6. External examiners
© TWBANTA-IUPUI
© TWBANTA-IUPUI
Western Governors Universityoffering
Competence-based on-line degrees
1) Define the domain of knowledge/skill2) Develop objective test items3) Design performance tasks4) Evaluate performance5) Use findings to improve curriculum,
instruction, student services6) Continuously improve assessment quality
and validity
2009 NILOA SurveyProgram Level Approaches
1. Portfolios (80% in at least 1 area)
2. Performance assessments
3. Rubrics
4. External judges
5. Student interviews
6. Employer surveys
© TBANTA-IUPUI
© TWBANTA-IUPUI
Student Electronic Portfolio• Students take responsibility for
demonstrating core skills• Unique individual skills and
achievements can be emphasized• Multi-media opportunities extend
possibilities• Metacognitive thinking is enhanced
through reflection on contents- Sharon J. Hamilton
IUPUI
VALUE Rubrics
• Critical thinking• Written communication• Oral communication• Information literacy• Teamwork• Intercultural knowledge• Ethical reasoning
© TWBANTA-IUPUI
Accountability Report85% achieve Outstanding ratings in
writing as defined . . .
78% are Outstanding in applying knowledge and skills in internships
75% are Outstanding in delivering an oral presentation
© TWBANTA-IUPUI
For External Credibility
Collaborate on rubrics
Use employers as examiners
Conduct process audits
© TWBANTA-IUPUI
E-Port Challenges
• Reliability of rubrics• Student motivation if used for
assessment
(Barrett, 2009)
• Differences in topics for products to be evaluated
(Sekolsky & Wentland, 2010)
© TWBANTA-IUPUI
Obstacles to UsingPerformance-Based Measures
• Defining domains and constructs• Obtaining agreement on what to
measure and definitions• Defining reliability and validity• Creating good measures
- Tom Zane
WGU© TWBANTA-IUPUI
Will it take 80 years . . . ?
3 Promising Alternatives
E portfolios
Rubrics
Assessment communities
- Banta, Griffin, Flateby, Kahn
NILOA Paper #2 (2009)
© TWBANTA-IUPUI
PART 2
Building An Evidence-Based Culture
Evidence at several levels
An institutional example
© TWBANTA-IUPUI
© TWBANTA-IUPUI
Organizational Levels for Assessment
National
Regional
State
Campus
College
Discipline
Classroom
Student
Evidence at the Classroom Level
• Background Knowledge Probe
• Minute paper in class
• Just-in-time teaching on line
© TWBANTA-IUPUI
© TWBANTA-IUPUI
Background Knowledge Probe(Pre-Test – Indirect Measure)
1. ARCHAEOLOGYA. Have never heard of thisB. Have heard of it, but don’t really know what it meansC. Have some idea what it means, but
not too clearD. Have a clear idea what this means
and can explain it- Classroom Assessment Angelo and Cross
Primary Trait Scoring
Assigns scores to attributes (traits) of a task
STEPS Identify traits necessary for success in
assignment Compose scale or rubric giving clear
definition to each point Grade using the rubric
Assessment of Group Interaction
The Student Participant:• Listened to others • Actively contributed to discussion• Challenged others effectively• Was willing to alter own opinion• Effectively explained concepts/insights• Summarized/proposed solutions
5=Consistently excellent
3=Generally satisfactory
1=Inconsistent and/or inappropriate
Evidence at the Program Level
• Individual and team projects• Research papers• Internships• Electronic portfolios• Peer review
© TWBANTA-IUPUI
Assessment in Sociology and Anthropology
Focus groups of graduating students Given a scenario appropriate to the discipline, a
faculty facilitator asks questions related to outcomes faculty have identified in 3 areas: concepts, theory, methods.
2 faculty observers use 0-3 scale to rate each student on each question
GROUP scores are discussed by all faculty Murphy & Goreham
North Dakota State University
© TWBANTA-IUPUI
Internships
Evaluated against specific criteria by
• Students
• Faculty
• Field-based supervisors
Elements of Program Review
Self Study
Review by Respected Peers
Recommendations
Follow-up
© TWBANTA-IUPUI
Evidence at the Institutional Level
• Learning outcomes• Questionnaires, interviews, focus groups• Productivity measures• Cost analyses• Management ratios• Program evaluation• Peer review• Accreditation
© TWBANTA-IUPUI
© TWBANTA-IUPUI
Outcomes Assessment Requires Collaboration
In setting expected program outcomes
In developing sequence of learning experiences (curriculum)
In choosing measures In interpreting assessment findings In making responsive improvements
Faculty and Staff Development
Focus faculty and student affairs professionals on improving learning in and outside class
Attend conferences together Study literature on student learning Provide workshops on teaching and learning Provide resources (e.g., grants, summer
salary, release time)
© TWBANTA-IUPUI
Some Evaluative Questions If we undertake a new approach:
Is instruction more effective? Are students learning more?
Are students more satisfied? Are faculty more satisfied?
Do outcomes justify costs?
© TWBANTA-IUPUI
Campus Interest in Assessment
WHAT WORKS in….
increasing student retention? general education? use of technology in instruction? curriculum in the major?
© TWBANTA-IUPUI
Good assessment is good research . . .
An important question An approach to answer the
question Data collection Analysis Report
-Gary R. Pike (2000)
© TWBANTA-IUPUI
Involve Students1. Set learning expectations in recruiting
2. Communicate learning outcomes in orientation
3. Involve student leaders in promoting learning
4. Involve students in evaluating courses/curricula
5. Let students know their recommendations are used.
Student Advisory Council at Montevallo
A way to provide continuous student assessment
Student Recommendations1 Develop a statement of expected
ethical behaviors for students2 Add a second research course with lab3 Increase comparative psychology4 Add terminals for statistics lab5 Increase opportunities for research,
writing, and speaking
Involving EmployersCombination of survey and focus groups
for employers of business graduates Identified skills, knowledge, personality attributes
sought by employers Encouraged faculty to make curriculum changes Motivated student to develop needed skills Strengthened ties among faculty, students, employers
- Kretovics & McCambridge
Colorado State University
Colorado State UniversityCollege of Business
Curriculum changes based on employer suggestions:
1 credit added to Business Communications for team training and more presentations
Ethics & social responsibility now discussed in intro courses
New Intro to Business course emphasizing career decision-making
More teamwork, oral & written communication, problem-solving in Management survey courses
- Kretovics & McCambridge
PLANNING
1. Campus mission, goals
2. Unit goals aligned
3. Programs based on assessable goals with PIs
4. Annual reports on the Web© TWBANTA-IUPUI
Outline for Annual Reports
IUPUI Theme
Unit Goal
Objective
Actions Taken
Actions Planned
Evidence of Progress
© TWBANTA-IUPUI
Evaluation Services1. Assessment of learning
2. Surveys
3. Program reviews
4. Performance indicators
5. Program cost analysis
6. Web-based evaluation tools
7. Program evaluation/action research
8. Accreditation© TWBANTA-IUPUI
Surveys
1. Enrolled Students Our own NSSE
2. Graduates3. Employers4. Stop outs5. Faculty6. Staff
© TWBANTA-IUPUI
Information Gateway
http://reports.iupui.edu/gateway/
Information about
Students
Faculty
Staff
Alumni
Finances
© TWBANTA-IUPUI
Since 1993Campus-wide surveys
have stimulated changes in
Curricula Advising Increased writing practice Increased attention to first-year
experiences Placement of graduates
© TWBANTA-IUPUI
Goal and Objectives for Student Learning
Enhance undergraduate student learning
and success
1. Strengthen generic skills
2. Provide honors programming
3. Offer learning communities
4. Strengthen advising
5. Provide tutoring and mentoring
© TWBANTA-IUPUI
Employ Multiple Methods
1) Direct
Projects, papers, tests, observations
2) Indirect
Questionnaires, interviews, focus groups
Unobtrusive measures
Syllabi, transcripts
© TWBANTA-IUPUI
© TWBANTA-IUPUI
Student Learning OrientedCourse Evaluation
1. Learners held high expectations for one another
2. Learners interacted frequently with others
3. Learners participated in learning teams4. Learners respected diverse talents and
ways of learning
-CournoyerAdvances in Social
Work – Fall 2001
Since 1994Assessment of Learning
has stimulated changes in
Student support programsCurriculumMethods of instructionInternshipsMethods of assessment
© TWBANTA-IUPUI
What is ABC? ABC is a costing methodology based upon the fact that different activities and products consume different proportions of resources
Resources
Product A
Product B
Product C
Activities
Activities
Activities
© TWBANTA-IUPUI
Some tasks within instruction curriculum planning course design class preparation class instruction assessment course evaluation
© TWBANTA-IUPUI
What Is ABC?Traditional vs. ABC
Traditional Accounting Perspective Salary & wages 1,350,000 Benefits 495,000 Travel 45,000 Facilities 220,000 Supplies 90,000
Total $2,200,000
Activity-Based Perspective Teach courses 940,000 Perform research 430,000 Provide service 250,000 Administer programs
350,000
Provide tech support
230,000
Total $2,200,000
© TWBANTA-IUPUI
Some Applications of Economic Model
1. Estimate costs of administrative services as compared to cost of outsourcing2. Determine fees for various programs3. Restructure processes to expedite work flow and minimize costs
© TWBANTA-IUPUI
Since 1992Activity-based Costing has stimulated changes in
Planning Budgeting Assessment
© TWBANTA-IUPUI
Elements of Program Review
Self Study
Review by Respected Peers
Recommendations
Follow-up
© TWBANTA-IUPUI
Goals of Program Review at IUPUI
To improve student learning To assess and improve program
quality To increase cross-disciplinary
collaboration To enhance community connections To reinforce importance of aligning unit
and campus planning
© TWBANTA-IUPUI
Following a Program Review
1. Program receives reviewers’ report
2. Faculty meet to consider findings
3. Faculty respond in writing
4. Program chair, dean, provost meet to consider written response
5. Improvements are implemented
© TWBANTA-IUPUI
Since 1995 Program Reviews have stimulated changes in
•Planning for the future•Research emphases•Faculty hiring priorities•Advisory councils•Cross-disciplinary collaboration
© TWBANTA-IUPUI
© TWBANTA-IUPUI
THE TEAM
1. Chancellor, Provost
2. IMIR
3. Program Review & Assessment Committee (PRAC)
4. Faculty Development
© TWBANTA-IUPUI
Open sharing of information and evidence-based decision-making
• Financial and satisfaction data for units
• Annual planning/budgeting hearings
• Performance indicators derived from unit
reports over time
• Campus performance report for
community
Program Review & Assessment Committee
� 2 reps from each school� 2 librarians� Other units
Student Life
Faculty Development
Internship coordinator
© TWBANTA-IUPUI
© TWBANTA-IUPUI
Program Review and Assessment Committee
• Provides a forum for exchange of information about assessment
• Oversees program review
• Suggests/provides faculty development
• Develops annual reports
© TWBANTA-IUPUI
Characterizing the Culture
▪ Appointment of Assessment Specialists• Faculty Development
• Library• Student Life• Service Learning• Enrollment Services• University College
▪ Appointment of Associate Deans for Assessment
© TWBANTA-IUPUI
Characterizing the Culture
New initiatives require assessment
University College
student support programs
Distance learning
New academic programs
© TWBANTA-IUPUI
Characterizing the Culture
Promotion & Tenure Guidelines
Faculty/Staff Development Grants
Awards
© TWBANTA-IUPUI
Build Assessment into Valued Processes
1. Assessment of learning
2. Curriculum review and revision
3. Survey research
4. Program review
5. Scholarship of Teaching & Learning
6. Evaluation of initiatives
7. Faculty development
8. Promotion & tenure
9. Rewards and recognition
Profiles
Invited over 1000
Received 146
Selected 49 for use in full
Categorized all 146 and published
Web sites
© TBANTA-IUPUI
Outline for Profiles
Background and Purpose
Methods over ? Years
Resources Required
Findings
Use of Findings
Impact of Using Findings
Success Factors
Web sites© TBANTA-IUPUI
~ Organization ~of
Principles & ProfilesPlanning
Implementing
Improving & Sustaining
- Building a Scholarship of Assessment
Banta & Associates
Jossey-Bass 2002
© TBANTA-IUPUI
Planning Principles
1. Engaging stakeholders
2. Connecting assessment to valued goals & processes
3. Creating a written plan
4. Timing assessment
5. Building a culture based on evidence
© TWBANTA-IUPUI
Planning Profiles
Brigham Young University
Campus Wiki for degree learning outcomes
USMA at West Point
Interdisciplinary teams assess 10
mission-related goals for learners
Kennesaw State University
2008 CHEA Award for linking assessment
with planning, program review, faculty
development
© TWBANTA-IUPUI
USMA @ West Point6 Developmental Domains
1. Intellectual
2. Physical
3. Military
4. Social
5. Moral-ethical
6. Human spirit
© TBANTA-IUPUI
USMA @ West PointIntellectual Domain
10 Goals (write, speak, think; engineering, math, info tech)
A. Stated learner outcomes
1. Standards
a. Rubrics developed by faculty
© TBANTA-IUPUI
USMA @ West PointInterdisciplinary Goal Teams
use
• Curriculum-embedded direct measures of learning
• Student surveys (fr., sr.)• Graduate survey (3 years after)• Employer surveys• Employer focus groups
© TBANTA-IUPUI
USMA @ West PointUse of Assessment Findings
Review of core curriculum
Changes in warranted areas:
History
English
Engineering
Information Technology
© TBANTA-IUPUI
Implementation Principles
1. Providing leadership
2. Creating faculty/staff development
3. Placing responsibility with unit
4. Using multiple methods
5. Communicating findings
© TWBANTA-IUPUI
Implementation Profiles
• California State University, Sacramento• Strong leadership, multiple methods
• Texas Christian University• Faculty learning communities
• Tompkins Cortland Community College• Capstone rubrics
© KBLACK-IUPUI
Cal State-Sacramento (1)
Sources of Motivation for Assessment
1. New VP for Student Affairs
2. Reaccreditation looming
3. Enrollment & budget challenges
4. Pledge to become more data-driven and focused on student learning
© TWBANTA-IUPUI
Cal State-Sacramento (2)
1. Align department & division missions
2. Develop SMART goals, 1 for student learning
Specific
Measurable
Aggressive, yet attainable
Results-oriented
Timely
© TWBANTA-IUPUI
Cal State-Sacramento (3)
MeasuresPre-post MC tests on policies, resources
Essays with rubrics (reinstatement)
Portfolios
Observation of skills
(Leadership, RA reports on scenarios,
role-playing)
© TWBANTA-IUPUI
Cal State-Sacramento (4)
Findings1. Some SLOs met
2. Some SLOs not met
3. Some measures not effective
4. Too few participants to assess
5. Too many participants to assess
effectively
© TWBANTA-IUPUI
Cal State-Sacramento (5)
Use of Findings1. Better training for RAs in reporting
2. Better training for peer mentors in orientation (emphasizing policies)
3. More time to discuss films
4. Better PowerPoint presentations
5. Increase participation in counseling
6. Redesign vague test items
© TWBANTA-IUPUI
Implementation Profiles (Continued)
Pennsylvania State UniversityPULSE Survey
Moravian College
Using technology for curriculum maps
Alverno CollegePortfolios in Teacher Education
Northeastern Illinois UniversityMultiple methods including national
standardized test
© KBLACK-IUPUI
A Look At The Profiles
• Leadership 18%• Faculty and Staff Development 18%• Responsibility at Unit Level 33%• Methods
• Rubrics 37%• Surveys 33%• Electronic/Technology 20%• Portfolios 14%• National Standardized Tests 8%
© KBLACK-IUPUI
Improving/Sustaining Principles
1. Providing credible evidence of learning to multiple stakeholders
2. Reviewing assessment reports
3. Ensuring use of results
4. Evaluating the assessment process
© EJONES-WVU
Improving/Sustaining Profiles
San Jose State University
Specialists in each college, awards,
learning outcomes in 5-year plans
Hocking Technical College
Annual assessment work day
Colorado State University
Integration of learning outcomes in
on-line template for program reviews
© TWBANTA-IUPUI
Sustaining Professional Development: Faculty Learning Communities
Texas Christian University
--Six areas in general education
1. religious traditions
2. historical traditions
3. literary traditions
4. global awareness
5. cultural awareness
6. social values
7. citizenship
© EJONES-WVU
Sustaining Professional Development: Faculty Learning Communities
Texas Christian University
--Created faculty learning communities to address the following:
a. identify and create assessment strategies
b. share results of assessment processes
c. discuss results to enhance teaching and learning experiences
© EJONES-WVU
Required Resources To Implement and Sustain Assessment
1. Faculty release time
2. Stipends for faculty leaders
3. Assessment committee
4. New full-time assessment position created
5. External consultants
© EJONES-WVU
Required Resources To Implement and Sustain Assessment
6. Financial resources to pay for tests and purchase surveys
7. Administrative support
8. Professional development
9. Technology
© EJONES-WVU
Some Big Ideas • Influence of accreditation is strong• Engaging faculty may require extra pay• Standardized tests of generic skills are
not used alone• Linking assessment with planning and
program review works• Impact is not measured in learning
gains
© TWBANTA-IUPUI
© TWBANTA-IUPUI
Group Assessment Has Failed toDemonstrate Institutional Accountability
• Focus on improvement at unit level
• Rare aggregation of data centrally
• Too few faculty involved
• Involved faculty return to discipline
• HE scholars focused on K-12 assessment
Impact of Using Findings
More attention to:
improving assessment tools
need to do assessment
participating in faculty
development
using assessment findings
© TWBANTA-IUPUI
Where Learning Has Improved
Alverno College – Milwaukee, WI
Truman State University – Kirksville, MO
© TWBANTA-IUPUI
Computer-Based Testingat
James Madison University
• Information Literacy
• Scientific Reasoning
• Quantitative Reasoning
University of South Florida
Cognitive Level & Quality of Writing Assessment
(CLAQWA)
Rubric of 16 traits X 5 (Bloom’s) levels
Used by peers and teachers
Improves writing and thinking
© TWBANTA-IUPUI
San Diego State University
In portfolios, masters & doctoral students
reflect on
curricular & co-curricular learning
program learning outcomes
Oral presentations of synthesized learning
Evaluated by faculty, external professionals
Synthesized learning has improved
© TWBANTA-IUPUI
North Carolina State University
DEAL Model for Critical Reflection(Description, Examination, Articulation of Learning)
Rubric levels based on Bloom’s Taxonomy
Improves higher order reasoning
and critical thinking skills
© TWBANTA-IUPUI
Building A Scholarship of Assessment
• National Institute for Learning Outcomes
Assessment (NILOA)• AAC&U’s VALUE Project• Teagle’s Wabash Study and Assessment
Scholars• Lumina’s big Goal and Degree
Qualifications Framework• New Leadership Alliance for Student
Learning & Accountability
© TWBANTA-IUPUI
ASSESSMENT UPDATE
Bi-monthly
Published by Jossey-Bass
Since 1989
Articles up to 2000 words
4 Columns
Book Reviews
© TWBANTA-IUPUI
© TWBANTA-IUPUI
Scholarship Reconsidered
• Four kinds of scholarship Discovery Integration Application Teaching
-Boyer (1990)
© TWBANTA-IUPUI
SoTLdiffers from
the scholarship of discovery
in its focus on the classroom
L.Shulman (2004)
© TWBANTA-IUPUI
SoTL Approach to Classroom Research 1. Articulate learning goals. 2. Formulate a question about the learning
situation based on the goals. 3. Design a way to collect data. 4. Teach to the goals. 5. Assess the student learning toward the
goal. 6. Analyze the feedback. 7. Reflect on the results for future teaching
decisions. 8. Share the results.
© TWBANTA-IUPUI
SoTL involves
1. Systematic investigation of a research question
2. Study of related literature3. Going public with findings4. Critical review by peers5. Use of research as foundation for
further work
H. Timberg (2007)
© TWBANTA-IUPUI
Types of SoTL QuestionsThe context of teaching: institutional
factors, physical facilities, organizational support
Example: Is it more effective to teach this class in one-hour or two-hour sessions?
Example: How does sitting around tables rather than sitting in rows affect learning?
© TWBANTA-IUPUI
Sound Familiar? The Scholarship of Assessment and
Scholarship of Teaching and Learning are integrally related:
Can address topics besides learning (civic engagement)
Study of learning issues in
actual settings
based on evidence
resulting in public
sharing
Can in-clude con-ceptual non-empirical questions or issues
SoASoTL
© TWBANTA-IUPUI
Building a Scholarship of Assessment
- Banta & Associates
Jossey-Bass Publishers
April 2002
© TWBANTA-IUPUI
Scholarly Assessment Involves
o selecting/creating assessment methods
o trying the methodso reflecting on strengths/weaknesseso modifying the methods or trying
new oneso improving assessment continuously
© TWBANTA-IUPUI
Scholarly Assessment
Conduct syllabus analysis
- Is critical thinking emphasized?
Develop student guide to assessment
- Do students understand
why and how they are assessed?
© TWBANTA-IUPUI
The Scholarship of Assessment Involves
basing assessment studies on relevant theory/practice
gathering evidence
developing a summary of findings
sharing findings with the assessment community
© TWBANTA-IUPUI
Scholarship of Assessment
Compare two teaching methods
- Is technology-enhanced instruction
more effective?
Validate a measure of student civility
- Do interventions increase civility?
Barriers to Scholarship in Assessment
• Campus coordinators are trained in other disciplines
• Scholars in relevant fields don’t do outcomes assessment
• Assessment scholarship is not rewarded
• Campus coordinators return to their own disciplines
• Few graduate programs prepare assessors
© TWBANTA-IUPUI
Some Research Traditions Underlying Assessment
Program evaluationOrganizational change and
developmentCognitive psychologyStudent developmentMeasurement Informatics
© TWBANTA-IUPUI
Assessment MethodsImprove instruments to measure
content knowledge at more
complex levels
affective development
effects of educational interventions
changes in learning over time
© TWBANTA-IUPUI
Assessment Methods
• How can we use technology in assessment more effectively?
• How can we demonstrate the validity of locally developed instruments?
• How can faculty make consensual judgments about the quality of student performance?
• How can student feedback be designed to help faculty improve their teaching?
© TWBANTA-IUPUI
Organizational Behavior & Development
How can assessment be combined with other systemic changes to improve teaching & learning?
What patterns of organizational behavior promote and sustain assessment?
What methods of providing and managing assessment information are most effective?
Which public policy initiatives are most effective in promoting improvement on campuses?
© TWBANTA-IUPUI
Shared Reflective Practice
Conduct meta-evaluations of approaches to assessment
Determine what works best within disciplines
Develop consortia of institutions to provide forums for reflection
© TWBANTA-IUPUI
Engaging Faculty
Introduce assessment as research
Connect assessment with the scholarship of teaching
Support learning about assessment through faculty development
© TWBANTA-IUPUI
Targets for Research on Engaging Faculty
How can we determine the interests and commitments of stakeholders?
How should we educate stakeholders for choosing methods?
How can we reduce costs and maximize assessment’s benefits?
What ethical principles should guide our work?
Derived from Michael Quinn Patton’sUtilization – Focused Evaluation (1997)
© TWBANTA-IUPUI
Recommended