Wssu session 2

MEASURING STUDENT LEARNING OUTCOMES

Terri Manning, Ed.D

Questions About Assessment

1. We’re spending time and resources trying to achieve student learning – is it working?

2. When we claim to be graduating students with qualities such as “critical thinking” or “scientific literacy,” do we have evidence of our claims?

3. We have the impression our students are weak in area ___ would more systematic research back up this impression and help us understand the weakness more thoroughly?

4. When we identify a weakness in our students” learning, how can we best address the problem?

5. How can we improve learning more effectively in a time of tight resources?

Assessment Clear and Simple, Walvoord, 2004

In a perfect world

Faculty have the time to think about outcomes before they develop and begin teaching a course

In Reality Faculty have been teaching the course for years

before they ever think about learning outcomes assessment

The Perfect World Method

A new faculty member teaching within a program spends some time planning and examining her curriculum in relationship to all the courses in the program.

She asks herself these questions? What are the critical skills, knowledge, attitudes, behaviors and values students in our program are supposed to be getting from my courses?

She analyzes the general education competencies and determines how she defines those according to her content and which competencies she is enhancing in her courses?


She creates a short list of important items. She asks herself “what assignments and assessments

am I using to measure if students have these skills or not?”

She establishes that three of her assessments in each course measure the critical outcomes students are supposed to achieve from her courses.

She teaches her courses, she gives her assessments. She identifies which items on her exams correlate to

the learning outcomes for the course.


She creates a grading rubric for several assignments with sub-scores based on the learning outcomes for her courses.

Disaggregates her grades by outcomes measured with her assessment and records those scores in her grade book.

At the end of the term she knows how many students have accomplished each outcome.

For those outcomes on which students don’t do well, she begins a process of identifying difficulties and barriers to change her teaching strategies.

The Real World

How do we bring a group of faculty together who have been teaching within a program area for decades and create a meaningful and useful outcome assessment process?

A Six Step Process for Outcomes

Step One2. During the discovery phase, faculty members are

involved in a working session where they are trained on learning outcomes assessment. They move into an activity on identifying “reasons for being, what we want them to get out of it and overall benefits for students.”

• Forms to help: Outcome Identification Form• and Outcome Statements Form

Step Two

1. During the prioritization phase they begin the narrowing of a long list of potential outcomes to the critical ones. This process involves them working through 1) what would be useful for faculty and departments to know, 2) what the faculty/department values, 3) what would inform decision-making, and 4) what is most critical for student success.

* A form to help: Outcome Prioritization Form

Steps Three

1. During the operationalization phase they create operational definitions of each outcome based on the curriculum course or program (e.g. effective writing is defined differently for English faculty than it is for early childhood faculty).

Step Four

1. During the mapping phase, faculty look at the natural fit between 1) courses and outcomes and 2) assessments/projects and outcomes. They create assessment processes where good ones are lacking.

* A form to help: Outcome Assessment Form

Step Five

1. The measurement phase involves setting realistic outcome targets, developing/using assessment tools, the process of distribution and collection of data and the reporting chain of command.

Step Six

� The final phase involves analysis among appropriate faculty, determining a plan of action and writing up results. Follow-up reporting is strongly recommended.

Based on Middle States Accrediting Agency Publications, Linda Suskie

Characteristics of Good Assessment

Good Assessment is Used

Results are used to inform important decisions on important goals.

Planned and purposeful Focus on clear and important goals Active participation of stakeholders Assess teaching-learning process as well as

outcomes Results communicated widely and transparently Results used fairly, ethically and responsibly

Good Assessment is Cost Effective

Efficient and economically (especially time). Focus on clear and important goals (3-6 goals) Start with what you have Simple (short surveys) Realistic expectations

Good Assessment is Accurate and Truthful

Flow from clear and accurate goals Represent a balanced sample of key goals

including thinking skills Use a variety of approaches, including direct

evidence of student learning Recognize diverse approaches to teaching and

learning developed thoughtfully Perpetual works in progress

Good Assessment is Valued

Assessment results/efforts are recognized/honored Innovation, risk-taking and efforts to improve

teaching and learning are recognized and honored Supported with appropriate resources: time,

guidance, support and feedback

Good Assessment Arise from Clear and Important Goals Flow from and focus on clear and important goals Inform important decisions on important goals Have clear and important standards for acceptable

and exemplary student performance (Spellings)

Let’s Talk Best Practices

The Council for Regional Accrediting Commissions (CRAC) – sponsored by all six regional accrediting agencies have produced two documents that identify:

2. What accrediting commissions should reasonably expect of an institution

3. What an accrediting commission should expect of itself

4. Best practices for institutions around student learning

Written Resources Regional Accreditation and Student Learning: A

Guide for Institutions and Evaluators (on SACS-COC website)

Regional Accreditation and Student Learning: Principles for Good Practice (on SACS-COC website) “to give a central focus to student learning as a

demonstration of institutional quality”

Five Principles

What an Accrediting Commission Should Expect of Institutions

Principle One

The Role of Student Learning in Accreditation

the institution defines educational quality – one of its core purposes – by how well it fulfills its declared mission of student learning.

“An Institution’s Learning Mission

…reflects its aspirations for students, and is stated in terms of how students are expected to benefit from its course of study.” (p. 7)

Quote

“Educational quality refers to the quality of student learning itself, both the extent to which the institution provides an environment conducive to student learning, and the extent to which this environment leads to the development of knowledge, skills, behaviors, and predispositions of value to students and the society they are preparing to serve.” (p. 7)

Key Points - Quote

“The literature on collegiate student learning is remarkably clear on what it takes to produce quality learning. An institution that takes its learning mission seriously and that views the quality of student learning as one of its core purposes:

Quote

Is clear and public about the learning outcomes to which it aspires for its students;

Uses these learning goals as well as knowledge about learning as drivers for organizing instruction;

Provides an environment which signals support for student learning at all levels; and

Promotes an atmosphere of critical reflection about teaching and learning.” (p. 7)

Principle Two

Documentation of Student Learning the institution demonstrates that student

learning is appropriate for the certificate or degree awarded and is consistent with the institution’s own standards of academic performance.

How An Institution Accomplishes This

Set clear learning goals which speak to both content and level of attainment.

Collect evidence of goal attainment using appropriate assessment tools.

Applying collective judgment as to the meaning and utility of the evidence

Using this evidence to effect improvements in its programs

Simply put…

Specify what students should learn – the learning outcomes

Identify potential sources of evidence to show if students have attained learning goalsEvidence is – information that has been

deliberately organized to support a claim

Questions to ask…

What constitutes learning that is good enough?

What should the standard be? Are the standards adequately met?

Principle Three

Compilation of Evidence the institution derives evidence of student learning

from multiple sources, such as courses, curricula, and co-curricular programming, and includes effects of both intentional and unintentional learning experiences. Evidence collected from these sources is complementary and demonstrates the impact of the institution as a whole on the student.

The Process

Because learning is complex and varies from learner to learner, the best approach

Identify the broad learning goals you have for all students, and

Begin to identify sources of information to show student learning

Multiple methods from multiple sources works best

Assessment

Should be more than a compilation of all the course assessments but should include Interaction between curricular and co-curricular

experiences Should demonstrate how the institution has impacted

the student’s learning Not be only at the end but along the way… Multiple sources – data from institution, strategic plans,

direct assessments, surveys, focus groups – don’t rely on just one

Principle Four

Stakeholder Involvement the collection, interpretation and use of student

learning evidence is a collective endeavor, and is not viewed as the sole responsibility of a single office or position. Those in the institution with a stake in definitions of educational quality participate in the process.

A Stakeholder

Anyone with interest in the institution – who stands to gain or lose by what the institution does… faculty, staff, students, parents, board members, etc.

Hold focused conversations on the challenges to student learning or use the results of surveys, or focus groups.

Stakeholders should participate in the interpretation and use of student learning evidence.

Different perspectives and opinions leads to a thorough analysis of data.

Principle Five

Capacity Building the institution uses broad participation in

reflecting about student learning outcomes as a means of building a commitment to educational improvement

Capacity Building

The tendency to treat accreditation as a task to complete does not build capacity

Accrediting agencies want institutions to internalize assessment of student learning and use it to monitor its own quality

Characteristics of Capacity that Lead to Institutional Improvement

Leadership of engagement – frame things correctly, put clear choices before the faculty, open to negotiation about what informs decisions. Provide incentives for risk-taking of departments (experiment with assessment).

Culture of peer collaboration and peer review – based on shared understanding of faculty work

Flexible and decentralized evaluation policies – units/programs can define for themselves the critical evaluation questions, sources of evidence, analysis and interpretation of data.

Characteristics, cont.

Willingness to make assumptions explicit and to question them – challenge existing perspectives is key to institutional vitality

Recognize than sources of evidence already exist and best expertise is inside the institution – not comprehensive and standardized assessment instruments and external consultants to implement them.

Characteristics, cont.

Time and resources to reflect on what is learned from assessment, to determine appropriate action, resources needed to implement, opportunities for discussion, invitation to adopt new and different approaches

Need for evidence to make decisions (culture of evidence) – spirit of reflection ad continuous improvement

Do We Need to Worry About the Six Critical Factors in Assessment Tools

Reliability Validity Sensitivity Appropriateness Objectivity Feasibility Is their a difference between your English essay

exam and a researcher who is trying to create an assessment tool to diagnose learning disbilities in children?

Reliability

Dependability or trustworthiness over time A perfect instrument, when given twice under the

same circumstances, will yield identical results The degree to which a test consistently measures

whatever it measures (over time)

What is important to you and your “tests” or “assignment?”

Reliability

Consistency When you have two sections of the same course and

they receive the same exam, are their differences in their scores?

Can students master the material tested for in exam 2 when they failed exam 1 – sequencing of material impacts reliability

Understanding of student variables that impact the reliability of a test – like time of day the course is taught, proximity of the exam to a holiday or campus event, etc.

Methods of testing reliability

Stability (test/retest) = one form of the test given to one sample twice

Equivalence (alternate forms) = two forms on a test given to one sample

Internal consistency (split halves) = two halves on one test given to one sample

Interrater (interscorer, interjudge) = consistency of two or more raters observing the same phenomena

Most of this can be done with statistical test

Validity

The degree to which a test measures what it is supposed to measure

Does the test provide data on the phenomena under consideration so that I can answer the question – “does the student understand this material?”

Students complain the most about this. I studied al six chapters and she only asked questions about chapter 2.

Is this as critical for your math exam as it is for the nursing licensure exam?

Content Validity

Was the test content selected with a theoretical or rational basis to provide the necessary data

Is it representative of all possible content How well does it represent a given content area

How to determine: Panels of experts Comparison to other tests Expect opinion

Common exams take care of this

Construct Validity

The degree to which it measures a hypothetical construct - behavioral or observational tests

Two concerns: 1) whether the test is able to distinguish between groups known to behave differently on the variable under study, and 2) which construct account for differences in test scores and which constructs do not account for differences in test scores

How to determine: A number of independent tests

Predictive Validity

Whether the test provides data that accurately predicts the future behavior of subjects

Placement tests??? Entrance exams???

How to determine: A number of independent tests

Appropriateness

Is the examinee capable of meeting the requirements of the test (e.g. reading level, physical format, age appropriateness, administration setting)

More important to us.

Objectivity

Is it free from bias Are the results a function of what is being measured

Language issues in tests – important to students

Feasibility

Am I capable of administering the test (e.g. license, training, equipment, access)

Not an issue for us.

Sensitivity

Can the test make fine distinctions required for the detection of true differences among subjects.

Difficult with likert scales.

Important in grading.

Making Expectations Clear and Explicit

Rubrics

Rubrics

Rubrics are scales in which the criteria used for grading or assessment are clearly spelled out along a continuum. Rubrics can be used to assess a wide range of assignments and activities in the classroom, from oral presentations to term papers to class participation. There are two main types: Analytic Rubrics: Separate scales for each trait, or learning

outcome, being assessed within the assignment (e.g., separate scales for "Argument,” “Organization,” “Use of Evidence,” etc.)

Holistic Rubrics: One scale for the assignment considered as a whole. (e.g., one scale describing the characteristics of an “A” paper, a “B” paper, or a “C” paper, etc.)

How They Help Students

Rubrics enhance student learning by: Anchoring grading to specific learning objectives, rather than

more subjective, distracting considerations of rank or effort Improving assignment design by clarifying desired learning

outcomes Contributing to fairness and consistency across sections. Reducing student anxieties about the subjectivity of grading Identifying traits that are important and describes levels of

performance within each of the traits

How Rubrics Help Faculty

Rubrics help you save time by: Narrowing the field of evaluation to desired learning

outcomes Facilitating constructive written comments Reducing grade challenges Reducing graders’ anxieties about grade inflation and

the subjectivity of grading

Look at examples

Grade/Point

Characteristics

5Argument pertains to relationship between social factors and educational opportunity and is clearly-stated and defensible.

4Argument pertains to relationship between social factors and educational opportunity and is defensible, but it is not clearly-stated.

3Argument pertains to relationship between social factors and educational opportunity but is not defensible using the evidence available.

2Argument is presented, but it does not pertain to relationship between social factors and educational opportunity.

1Social factors and educational opportunity are discussed, but no argument is presented.

Five-point Scales

Grade/Point

Characteristics

3Argument pertains to relationship between social factors and educational opportunity and is clearly-stated and defensible.

2Argument pertains to relationship between social factors and educational opportunity but may not be clear or sufficiently narrow in scope.

1Social factors and educational opportunity are discussed, but no argument is presented.

Three Point Scales

Ideal Outcome

Argument pertains to relationship between social factors and educational opportunity and is clearly-stated and defensible

3 2 1

Simplified Three-point Scale

Practice on Rubrics

Let’s complete some of these today For each of the general education goals, let’s work

on a set of rubrics You might need only one rubric for your subject but

with some you will need a rubric to be applied to a written assignment, one for an oral assignment and one for a project

If you created a library of rubrics for these gen ed goals that were posted somewhere – all faculty could use them and it would save everyone time.

Faculty Fear

We are going to have to spend all of our time assessing student learning and will have no time for teaching.

Change must occur in your thinking about why you are assessing students – the purpose

Change must occur in how it is collected Change doesn’t have to occur in the assessment

itself – unless it is not a valid measure of what you are doing.

How Do We Use What We Learn You have spent time creating a list of agreed-upon

outcomes You have created an assessment tool You have created a grading process and established a

benchmark for “success” You use the assessment tool for multiple sections, score it

and pull together the results Your students do not pass at the level you expected Why didn’t the pass? What do you do?

Using Results

If faculty can easily figure it out – make changes and move on

Do some research Methods that have been proven to work by discipline –

to improve retention, engage students, help explain content, create authentic assignments, improve content mastery, etc.

May have to make some curricular changes May need more info – to drill down further

Use of Results

What processes are in place to make sure you use results to improve programs, services and student learning.

Just because you have data showing that students are having difficulty doesn’t mean you know what to do to fix it.

Can you measure, report and forget it? Then why do it?

Major Issue Facing Faculty

What do we do when they don’t learn? And is this my fault?

I see that they didn’t get it, but why, what is missing and what do I do about it?

What have you tried that worked?

This can take time.

You need two things

1. Research and theories of student learning that suggest what steps might work to improve student learning in a given situation (may differ by content area).

2. Data on factors that may be affecting your students in particular. Two ways:

1. Ask them – the most feasible and best way to discuss what factors are affecting students’ learning is to ask them – directly, in focus groups, on surveys/questionnaires

2. Compare them – to each other (pre-post) or to others – can experiment with methods


When Assessment Doesn’t Work

When done only to comply with accreditation Triggering resistance and hostility of faculty Gathering data no one will use Letting administrators do it Making it too complicated


Bring Faculty On-board

Pat Hutchings suggests six ways to bring the purposes of assessment and the regular work of faculty closer together in the publication: Opening Doors to Faculty Involvement in Assessment

1. Build assessment around the regular, ongoing work of teaching and learning

Build on the process of grading in every course every semester. This can bring forth questions about course design, assignments and exams and feedback to students. This is where faculty talents and interests lie. Embedding assessment in the classroom sets the stage for discussions at the program level and draws on what faculty care most about: their discipline or field. When assessment reflects and respects disciplinary interest, it is more likely to lead to important faculty engagement.

2. Integrate assessment into the preparation of graduate students;

Finding this is the exception rather than the rule but it is occurring more often.

3. Reframe assessment as scholarship

Student learning should be seen as an important phenomena for investigators. The work should not be seen as service to the college (serving on the assessment committee) but as an important intellectual exercise. Attention should be paid to the use of new forms formulas, and genres for capturing the scholarly work of teaching, learning and assessment.

4. Make a place for assessment in faculty development

Many institutions of higher education have created centers for teaching and learning and assessment can be part of these centers. Bringing faculty together and facilitating constructive conversations around assessments, meaning, implications for pedagogy and a commitment to evidence can increase faculty engagement.

5. Create campus spaces and occasions for constructive assessment conversations and action

Teaching and learning have been traditionally seen and undertaken as private activities. Faculty assume responsibility for their own students and classrooms. It is foreign for them to come together in groups and discuss student learning, difficulties, strategies and possible actions as groups. Institutions have done various things from faculty learning communities, setting aside time in department meetings, multidisciplinary reading groups and inquiry groups to foster these conversations.

6. Involve students in assessment.

Student self-assessment where they monitor and direct their own development is gaining groups. New products exist such as e-portfolios, rubrics to guide student work and involve students in campus conversations about learning and assessment.

Questions We Should Ask Ourselves

1. We’re spending time and resources trying to achieve student learning – is it working?

2. When we claim to be graduating students with qualifications, do they actually have them?

3. We think they have problems with certain concepts or are weak in certain areas, do we have data to back that up?

4. When we identify weaknesses, how can we best address the problem?

5. What are our most cost effective methods of improving learning?

Things to Remember

Outcome measurement must be initiated from the unit/department level (promotes ownership of process).

Measure only what you are teaching or facilitating. Measure what is “important” to you or your program. Prioritize your outcomes. Be selective (2-3 outcomes only for a course or

programs). Use what you find.