11

Reporting Outcomes Results Reporting Outcomes Results and Improvementsand Improvements

Reporting Outcomes Results Reporting Outcomes Results and Improvementsand Improvements

Produced by Non-Instructional

Subcommittee of

Assessment Committee

22

Review of PET FormReview of PET FormReview of PET FormReview of PET Form

• Past year’s PET Form instruction included developing– Purpose Statement – Goals – Objectives (preferably as Outcomes)

33

44

Review of PurposeReview of PurposeReview of PurposeReview of Purpose

• Purpose Statement should: – fall within AC’s mission– identify THE reason for the

department’s/division’s/program’s existence– encompass all the key aspects of the

department/division/program

55

Review of GoalsReview of GoalsReview of GoalsReview of Goals

• Goals should be:

– based on the purpose for the division/department/program

– stated as broad/general/vague expectations for a division/department/program

– recommend 2 to 4 with no more than 5

(address the key areas desired for assessment and resulting improvement and/or growth)

66

Review of ObjectivesReview of ObjectivesReview of ObjectivesReview of Objectives

– the person or group expected to achieve the objective

– the specific task to be accomplished– the standard/benchmark/target to be achieved– the timeline or deadline for completion

• Objectives should identify:

77

Review of Objective TypesReview of Objective TypesReview of Objective TypesReview of Objective Types

• Types of objectives and definitions: – Inputs = resources– Outputs= types, levels and targets of

services (may include satisfaction)

– Processes = efficiencies or intended accomplishments

– Outcomes = specific changes in student’s/client’s knowledge,

skill, expertise, attitude or behavior

88

Review of Outcomes as Review of Outcomes as ObjectivesObjectives

Review of Outcomes as Review of Outcomes as ObjectivesObjectives

• Outcomes should be:

– a statement of what someone should be able to do or develop (knowledge, skill, expertise, attitude or behavior)

– written from the audience’s (e.g. student or client) perspective

– the preferred type of objective

NOTE: Each dept./program should develop at least one outcome and include in PET form. May have other types of objectives (e.g. output objectives, process objectives) in PET form as well.

99

Review of Assessing OutcomesReview of Assessing OutcomesReview of Assessing OutcomesReview of Assessing Outcomes

• All outcomes (objectives) do not have to be assessed all of the time

• All outcomes (objectives) do have to be assessed sometime (maybe some every semester and others annually or monthly)

Adapted from James Madison University

1010

New information: New information: Results ColumnResults Column

New information: New information: Results ColumnResults Column

• ANNUALLY:

– Report the Results for each objective (outcomes, outputs, processes or inputs)

– Reported results will be data for quantitative objectives

– Reported results will be summary narrative for qualitative objectives

1111

ResultsResultsResultsResults

1212

ResultsResultsResultsResults

• Quantitative– Statistical – Time reference

• ? Participation %(N=H)

• Qualitative

1313

New information: New information: Use of Results ColumnUse of Results Column

New information: New information: Use of Results ColumnUse of Results Column

• ANNUALLY:—Report the Use of Results for each objective

(outcomes, outputs, processes or inputs)

—Reported Use of Results will be the revisions and/or improvements recommended or made as a result of analysis of the evidence in the Results column

—Be sure to provide a summary of the analysis in the Use of Results column before identifying the revisions and/or improvements recommended or made

1414

Use of ResultsUse of ResultsUse of ResultsUse of Results

1515

Use of ResultsUse of ResultsUse of ResultsUse of Results

• Sentences describing how the department will make improvements based on the data

• Provide an answer of what will be done next

1616

Outcomes - Example 1Outcomes - Example 1Outcomes - Example 1Outcomes - Example 1

• Outcomes (objectives) should be worded to express what the student/client will learn, know, or do as a result of the intervention – NOT what the instructor or service will do for the student/client.– Typical Outcome: Provide student/client with

knowledge about how the library works.– Better Outcome: After taking the Research Methods

course, student/client will be able to demonstrate their knowledge of how the library works by finding 10 sources for a research paper in the library.

Adapted from James Madison University

1717

Outcomes - Example 2Outcomes - Example 2Outcomes - Example 2Outcomes - Example 2

• Outcomes should be reasonable. They should reflect learning that the student/client can accomplish as a result of the division/department/program.– Typical Outcome: Students will demonstrate open-

mindedness for all cultures by strongly agreeing with all of the items on the Open-Mindedness Inventory (OMI).

– Better Outcome: Upon completion of the Study Abroad program, participants will show an increase in open-mindedness through at least a 10-point increase on the Open-Mindedness Inventory (OMI).

Adapted from James Madison University

1818

Outcomes - Example 3Outcomes - Example 3Outcomes - Example 3Outcomes - Example 3

• Student/client learning should be assessed with an observable, measurable outcome. Verbs such as “know” and “understand” are not observable.– Typical Outcome: Students will understand the

electoral college.– Better Outcome at intervention method: Students

will be able to match at least 40 of the United States with their number of electoral votes.

– Or: Students will be able to evaluate (e.g. Govt of US Courses) the effectiveness of the electoral college in modern elections.

Adapted from James Madison University

1919

Outcomes - Example 4Outcomes - Example 4Outcomes - Example 4Outcomes - Example 4

Outcome should specify the criterion of ACCEPTABLE student/client performance.

Goal: Understand the concepts that contribute to career decision-making.

• Typical Outcome: Students will be able to select a career choice.

• Better Outcomes: Upon completion of the career and life planning course, students will be able to match a list of majors to appropriate career choices.

OR:

Upon completion of the career and life planning course, students will be able to state their “work personality” as measured by the Work Abilities, Values, and Interests (WAVI) Inventory.

Adapted from James Madison University

2020

Outcomes - The ABCDE MethodOutcomes - The ABCDE MethodOutcomes - The ABCDE MethodOutcomes - The ABCDE Method

• A = Audience (Who are you assessing?)• B = Behavior (What is expected of the

audience?)• C = Conditions (What intervention is required?)• D = Degree (To what degree is the behavior to be

reached?)• E = Evaluation (What tool will be used to

measure this?)

Recipe for Writing an Outcome Statement =C + A,B + D,E

Adapted from James Madison University

2121

Outcomes - ABCDE Outcomes - ABCDE Example 1Example 1 (Police (Police Dept)Dept)

Outcomes - ABCDE Outcomes - ABCDE Example 1Example 1 (Police (Police Dept)Dept)

• A = Audience (employees)

• B = Behavior (will take precautions)

• C = Conditions (After mock drills)

• D = Degree (decrease by 20% annually)

• E = Evaluation (number of reported campus burglaries)

Outcome statement = After mock drills, employees will take precautions, which will result in at least a 20% decrease annually in the number of reported campus burglaries.

Adapted from James Madison University

2222

Outcomes - ABCDE Example 2 Outcomes - ABCDE Example 2 (IST HelpDesk)(IST HelpDesk)

Outcomes - ABCDE Example 2 Outcomes - ABCDE Example 2 (IST HelpDesk)(IST HelpDesk)

• A = Audience (employees)

• B = Behavior (indicate that the technology problem was resolved during the time the problem was identified)

• C = Conditions (After seeking assistance with a technology request)

• D = Degree (75%)

• E = Evaluation (automatically generate an email with an anonymous web survey link)

Outcome statement = After requesting assistance with a technology request, employees seeking assistance with a technology request will indicate that the technology problem was resolved during the time the problem was identified 75% of the time and automatically generate an email with an anonymous web survey link.

Adapted from James Madison University

2323

Outcomes - ABCDE Example 3 Outcomes - ABCDE Example 3 (Student Services)(Student Services)

Outcomes - ABCDE Example 3 Outcomes - ABCDE Example 3 (Student Services)(Student Services)

• A = Audience (employees)• B = Behavior (demonstrate mastery of FERPA

regulations)• C = Conditions (After training session on FERPA

requirements)• D = Degree (80%)• E = Evaluation (on post-test)

Outcome statement = After attending a training session on FERPA requirements, all participating employees will demonstrate mastery of FERPA regulations at 80% on the post-test.

Adapted from James Madison University

2424

OutcomesOutcomesABCDE Example 4 (Testing Services)ABCDE Example 4 (Testing Services)

OutcomesOutcomesABCDE Example 4 (Testing Services)ABCDE Example 4 (Testing Services)

• A = Audience (faculty (FT & PT) requesting testing services)

• B = Behavior (will follow identified procedures)

• C = Conditions (After reading the website on testing instructions and

completing corresponding survey)• D = Degree (60%)

• E = Evaluation (based on log of all procedure violations)

Outcome statement = After reading instructions regarding testing from a website and then completing a corresponding survey, 60% of faculty (full-time & part-time) requesting testing services will follow identified procedures based on log of all procedure violations.

Adapted from James Madison University

2525

Outcomes - ABCDE Example 5 Outcomes - ABCDE Example 5 (Continuing Education)(Continuing Education)

Outcomes - ABCDE Example 5 Outcomes - ABCDE Example 5 (Continuing Education)(Continuing Education)

• A = Audience (companies with 25+ employees)

• B = Behavior (implement the program addressing generational differences in the workplace)

• C = Conditions (After attending a pilot program)

• D = Degree (25% of companies)

• E = Evaluation (based upon signed Training Agreements.)

Outcome statement = After attending a pilot program, 25% of companies with 25+ employees will implement the program addressing generational differences in the workplace based upon signed Training Agreements.

Adapted from James Madison University

2626

Outcomes - ABCDE Example 6 Outcomes - ABCDE Example 6 (Business Office)(Business Office)

Outcomes - ABCDE Example 6 Outcomes - ABCDE Example 6 (Business Office)(Business Office)

• A = Audience (employees with purchasing privileges)

• B = Behavior (will file paperwork in a complete and timely fashion in order that vendors will be paid by due date)

• C = Conditions (After receiving procedures instructions)

• D = Degree (75% of the time)

• E = Evaluation (based upon a transaction log.)

Outcome statement = After receiving procedural instructions, employees with purchasing privileges will file paperwork in a complete and timely fashion so that vendors will be paid by due date 75% of the time based upon a transaction log.

Adapted from James Madison University

2727

Outcomes PracticeOutcomes PracticeBRAINSTORMBRAINSTORM

Outcomes PracticeOutcomes PracticeBRAINSTORMBRAINSTORM

1.Purpose - restate the purpose statement:

(What is the primary reason that this division/department/program exists?)

2. Goals - select 1 or 2 key goals:

(Based on the purpose statement and experiences of the staff, what are the biggest challenges or problems that this division/department/program desires to improve?)

2828

Outcomes PracticeOutcomes Practice BRAINSTORM BRAINSTORM

Outcomes PracticeOutcomes Practice BRAINSTORM BRAINSTORM

3. Outcomes:Fill in each of the following for one outcome of

one goal:• A = Audience - • B = Behavior - • C = Conditions - • D = Degree - • E = Evaluation -

Write as a sentence (see recipe):Outcome Statement =

Adapted from James Madison University

2929

AssignmentAssignmentAssignmentAssignment

PET form: • Use the skills you’ve

developed to write at least one outcome for the division’s/department’s/program’s PET form.

3030

AssignmentAssignmentAssignmentAssignment

PET form

AC Website (www.actx.edu)

Scroll to bottom of screen

AC Forms & Policies

Planning & Evaluation Tracking (PET) Form

3131

AssignmentAssignmentAssignmentAssignment

Recommendation:• All employees in the

division/department/program should work together to brainstorm and decide on finalized PET form.

3232

AssignmentAssignmentAssignmentAssignment

Finalized PET form should be approved through appropriate President’s Cabinet member.

3333

AssignmentAssignmentAssignmentAssignment

Submit finalized PET form electronically to: Brandy Hayes in the Office of I.E. & Advancement

Deadline for submission: December 8, 2006