Compiling and Reporting Your Assessment Plan Data

Office of Institutional Assessment & EffectivenessSUNY OneontaSpring 2012

Providing a Context: Oneonta’s Process for Assessing Student

Learning Fall 2009: Approval by President’s Cabinet of APAC Guidelines for Academic Program Assessment

Spring 2009: Submission by programs of Step 1 (Establishing Objectives) of guidelines

December 2010: Submission by programs of Step 2 (Activities & Strategies) of guidelines

June 1, 2011: Submission of Steps 3 (Assessment) and 4 (Closing the Loop) [plans only] All plans approved by academic deans by the end of Fall 2011

2011-12 academic year: First round of data collection, with reports due June 1, 2012 as part of annual reports

Emphases Throughout the Process

Establishing congruence among institutional goals, programmatic and course objectives, learning opportunities, and assessments

Linkages to disciplinary (and, as appropriate, accreditation/certification) standards

Using a variety of assessment measures, both quantitative and qualitative, in search of convergence

Value of course-embedded assessment Course- vs. program-level assessment

Course- Vs. Program-Level Assessment

Focus of SUNY Oneonta assessment planning is programmatic student learning objectives, not individual students or faculty

For the most part, data collection described in approved assessment plans will take place in the context of the classroom (i.e., course-embedded assessment)

Focus at this point, however, is how to compile and aggregate data across courses and course sections, as appropriate, and to report these results in a meaningful way

What You’ve Done So Far

1. Development of programmatic student learning objectives

2. Curriculum mapping3. Development of plan for collecting data across a

three-year period APAC guidelines recommend that programs assess 1/3 of SLOs

every year

4. Development of plan for reviewing data and using that information to revise courses and programs, as appropriate (i.e., “closing the loop”)

Follow Your Curriculum Map – It’s Your Assessment Database!

COURSE

SLOs 1 2 3 4 5 6 7

Introductory Course E E History/Theories E P P Methods E, L E, L L Required Course 1 E E E, P P Required Course 2 P P P

Required Course 3 E E P Required Course 4 I, PO I, PO Capstone PO PO PO PO PO Assessment Key:

P-Paper E-Exam PO-Portfolio O=Oral Presentation L-Lab Assignment I-Internship

Designed correctly, your curriculum map tells you what to assess, when to

assess, and how to assess it!

Collecting Data: Important Points from Spring 2011

Workshop Collaborative work among faculty to assure: SLOs are being assessed using direct measures Measures are of sufficient quality, either by review process or

through the use of a checklist that includes good practice criteria A priori success indicators are in place for each measure as well

as common categories for results (e.g., exceeding, meeting, approaching, not meeting standards)

Decisions on how issue of multiple course sections will be addressed

Process in place for assuring: Reliable scoring of qualitative measures Courses (and students) to be included in assessment are

representative

Value of Assessment Data Will Be Maximized IF:

A variety of assessment measures is used Quantitative and qualitative Course-embedded and “stand alone” measures Benchmarking as available

For each SLO, data are collected from multiple courses/course sections

Criteria for scoring and determining what meets standards are consistent across courses Do measures have to be the SAME? No, but…….

And Perhaps Most Important

There’s no point in collecting useless

data!

Compiling and Reporting Assessment Data Across Courses

Aggregating and Documenting Your Assessment Results

Approaching the Task Systematically

1. Start with your curriculum map, which depicts: The courses in which you’re covering – and

assessing – SLOs And, depending on the detail of your map, the levels

at which SLOs are being addressed and the measures being used to assess them

2. Select courses – and faculty – to be included in the assessment and communicate that expectation clearly and establish a time frame for administering assessment and submitting results

Approaching the Task Systematically (cont.)

3. Collect and compile data (categorized as agreed upon by department) on spreadsheet for review

4. Review results collectively, reach conclusions regarding strengths and weaknesses, and make recommendations for change as appropriate

Be sure to document all this (i.e., write it down)

A Step-by-Step Example

1. Following Your Curriculum MapThe Simplest Case: Which Courses Cover Which SLOs?

To What Extent Do Courses Cover SLOs?

COURSE

SLOs 1 2 3 4 5 6 7

Introductory Course 4 2 History/Theories 4 4 3 Methods 4 4 4 Required Course 1 3 3 2 4 Required Course 2 4 3 4

Required Course 3 4 4 4 Required Course 4 4 4 Capstone 4 3 3 2 4

COURSE

SLOs 1 2 3 4 5 6 7

Introductory Course X X History/Theories X X X Methods X X X Required Course 1 X X X X Required Course 2 X X X

Required Course 3 X X X Required Course 4 X X Capstone X X X X X

1. Following Your Curriculum MapAt What Level Do Courses Cover SLOs?

How Do Courses Assess SLOs?

COURSE

SLOs 1 2 3 4 5 6 7

Introductory Course I I History/Theories I R R Methods I R R Required Course 1 R R R R Required Course 2 R M M

Required Course 3 R R M Required Course 4 M M Capstone M M M M M

Assessment Key:

I-Introduced R-Reinforced M-Mastery

COURSE

SLOs 1 2 3 4 5 6 7

Introductory Course E E History/Theories E P P Methods E, L E, L L Required Course 1 E E E, P P Required Course 2 P P P

Required Course 3 E E P Required Course 4 I, PO I, PO Capstone PO PO PO PO PO Assessment Key:

P-Paper E-Exam PO-Portfolio O=Oral Presentation L-Lab Assignment I-Internship

2. Selecting Courses to be Included in Assessment

Do all courses covering a particular SLO need to be included?

Ideally, yes – remember, each SLO is only assessed every 3 years

Especially important if SLOs are being assessed at different “levels” (in terms of proficiency and/or course levels)

Do all course sections covering a particular SLO need to be included?

No, but why not? If not, important that:

Adequate number of students are included in the assessment Sections are selected randomly

2. Selecting Courses to be Included in Assessment

(cont.) Once courses are selected, communicate this fact quickly and clearly to faculty, being sure to emphasize:

SLOs to be assessed and categories to be used in reporting results

Mapping of assignments – and eventually assessment results – to specific SLOs and the need for a priori criteria in categorizing student performance

Time line for submitting results (late enough in semester for students to demonstrate competence but early enough for data to be compiled)

Must assignments and a priori criteria be the same? No, but the more comparable, the more meaningful the results Ultimately, it’s a programmatic decision

A Few More Hints About Assignments

Final grades are NEVER an adequate measure for a specific SLO

Overall assignment grades are rarely – if ever – an adequate measure for a specific SLO

Often only minor adjustments are needed in existing assignments to gather information on specific SLOs

Limit assignments to one SLO Break assignments into subparts, each of which addresses a

specific SLO Identify test questions that map to specific SLOs

3. Compiling Data for Review and Discussion

Important for all results be submitted in agreed-upon format, organized, and maintained in a single place (an Excel spreadsheet is perfect)

Different ways to organize results for review and deliberation

Best to start with all data for each SLO, showing each course and its overall results for student performance organized by performance categories

The more you can aggregate, the better (i.e., by course, course level, level of proficiency), being attentive to variations within aggregating categories

Importance of removing identifiers

Sample Spreadsheet (for Internal Review by Program

Faculty)Course # # Students Exceeding Meeting Approaching Not Meeting

GER 100 25 24 56 12 8

GER 100 22 95 0 5 0

Sub-Total - GER 100 47 59.5 28 8.5 4

GER 150 24 76 14 10 0

GER 175 12 24 42 27 7

Sub-Total 100-level 83 54.75 28 13.5 3.75

GER 225 47 53.2 29.8 12.8 4.3GER 275 19 15.7 73.7 5.3 5.3

GER 280 23 61 22 13 4

Sub-Total 200-level 89 43.3 41.83 10.37 4.53

GER 330 28 21 50 25 4

GER 335 28 39 29 27 5

GER 350 17 24 46 29 0

Sub-Total 300-level 73 28 41.67 27 3

GER 481 9 33 66 0 0

GER 481 5 0 100 0 0

GER 481 21 24 43 33 0

Sub-Total 400-level 35 19 69.67 11 0

Total 280 37.68 43.96 15.32 2.89

4. Reviewing and Discussing Results and

Closing the Loop Important points of discussion How are data consistent?

Do students at different course levels perform similarly? Eventually, it will be possible to look at this issue over time

How are they distinctive? Do students perform better on some objectives than others?

Are assessment results “acceptable” to program’s faculty? What strengths (and weaknesses) are revealed and what

explains them? What should (and will) the program do to improve areas of

weakness?

Incorporating Assessment Data into Annual Report

At present, directions for Annual Report include the following:

“Descriptions of student learning outcome assessments conducted for your students during the year. These descriptions should include a summary of the assessment methods that were used and the results, including benchmarking as appropriate. Also, whenever possible, detail course-specific student learning outcomes and their relationship(s) to departmental or programmatic expectations.”

Incorporating Assessment Data into Annual Report

(cont.) APAC’s recommendation to the Provost and Deans is that Annual Reports include two distinct components with respect to the assessment of student learning, effective June 2012 Chart that summarizes the assessments that were

done during the academic year and results Narrative that summarizes the program’s discussion of

the assessment results, overall conclusions, and planned changes based on the assessment data

Programs may also report other relevant data (e.g., results from senior/alumni surveys)

Sample Outcomes Assessment

Chart – Aggregate DataSimplest Case (and All That’s Required)

CoursesAssessment Measure(s)

Performance Criteria

# of Students

% Exceeding Standards

% Meeting Standards

% Approaching

Standards

% Not Meeting

StandardsAll Portfolio 4-5=Exceeding

3=Meeting2=Approaching1=Not Meeting 

338 18% 59% 13% 10%

Sample Outcomes Assessment

Charts – More DetailedBy Course Level

CoursesAssessment Measure(s)

Performance Criteria

# of Students

% Exceeding Standards

% Meeting Standards

% Approaching

Standards

% Not Meeting

Standards100-Level Tests 90%+=Exceeding

80-89%=Meeting70-79%=Approaching69% and below=Not Meeting 

203 18% 59% 13% 10%

200-Level Papers 4=Exceeding3=Meeting 2=Approaching1=Not Meeting

78 18% 59% 13% 10%

Tests 90%+=Exceeding80-89%=Meeting70-79%=Approaching69% and below=Not Meeting

112 18% 59% 13% 10%

300-Level Portfolios 4-5=Exceeding3=Meeting2=Approaching1=Not Meeting

96 18% 59% 13% 10%

Sample Outcomes Assessment

Charts – More DetailedBy Competency Level

CoursesAssessment Measure(s)

Performance Criteria

# of Students

% Exceeding Standards

% Meeting Standards

% Approaching

Standards

% Not Meeting

StandardsBeginning Tests 90%+=Exceeding

80-89%=Meeting70-79%=Approaching69% and below=Not Meeting 

203 18% 59% 13% 10%

Inter-mediate

Papers 4=Exceeding3=Meeting 2=Approaching1=Not Meeting

78 9% 61% 20% 10%

Tests 90%+=Exceeding80-89%=Meeting70-79%=Approaching69% and below=Not Meeting

112 11% 54% 25% 10%

Advanced Portfolios 4-5=Exceeding3=Meeting2=Approaching1=Not Meeting

96 15% 54% 18% 13%

Issues to Be Addressed in Narrative Overall conclusions of student performance on

SLO as revealed by assessment Description of strengths and weaknesses

revealed by data Planned revisions for improvements as

appropriate Planned revisions to the assessment process

itself as appropriate

APAC Members

Paul French Josh Hammonds Michael Koch Julie Licata Richard Lee

Patrice Macaluso Anuradhaa Shastri Bill Wilkerson (Chair) Patty Francis (ex

officio)