From Here, Go Anywhere Overview of the Reporting Process Program/Discipline Review and Outcomes Assessment Report Dr. Michael E. Bryan, Coordinator Office

From Here, Go Anywhere

Overview of the Reporting ProcessOverview of the Reporting ProcessProgram/Discipline Review Program/Discipline Review

and Outcomes Assessment Reportand Outcomes Assessment Report

Dr. Michael E. Bryan, CoordinatorOffice of Student Outcomes Assessment

May 12, 2004

“Effective assessment must begin with real concerns of the

stakeholders and result in useful information and recommendations

related to the purpose of assessment.”

– D.W. Farmer & E.A. Napieralski


Intended Intended OutcomesOutcomes

Audience will understand the Program/Discipline Review Initiative including:

History and driving forces

Supporting resources

Reporting process (e.g., deadline, schedule, tasks)

Report format with various components (e.g., intended outcomes, assessment measures, analysis)

Various uses (e.g., program-, discipline-, and course-level)

“Many faculty have been ‘doing assessment’ for their own sake, and have not been overwhelmed with the task, without particular

experience in evaluation methodology.” – O.J. Nichols (1995)


What is Assessment?What is Assessment?The systematic collection and analysis of

information to improve student learning.

“…a learner-centered, teacher-directed approach designed to improve student learning in the individual classroom.”

– B.D. Wright (1991)


Assessment and GradesAssessment and GradesGrades are global evaluations that represent the overall

proficiency of students. They don’t tell you about student performance on individual learning goals.

“…assessment involves taking a second look at materials generated in the classroom so that in addition

to providing a basis for grading students, these materials allow

faculty to evaluate their teaching.” – B.D. Wright (1991)


Why Assess?Why Assess?

Rationale and Driving ForcesRationale and Driving Forces

“…assessment involves taking a second look at materials generated in the classroom so that in addition to providing a basis for grading students, these materials allow faculty to evaluate their teaching.” – B.D. Wright (1991)

“The institution demonstrates that each educational program for which academic credit is awarded is (a) approved by the faculty and the administration, and (b) establishes and evaluates program and learning outcomes.”

“The institution places primary responsibility for the content, quality, and effectiveness of its curriculum with its faculty.”

Principles of Accreditation, Standards for All Educational Programs, Section III, 1 and 12, August 2003


Reporting Reporting MandatesMandates

“Reports of Institutional Effectiveness (ROIE) is an annual report required by the State Council of Higher Education for Virginia (SCHEV) and is intended to provide meaningful information on the academic quality and operational efficiency of Virginia’s public institutions of higher education.”

“Each institutional report is organized into five sections: institutional mission, college profile measures, system-wide measures, and institution-specific measures, and the core competencies.”

ROIE Homepage, http://research.schev.edu/roie/

“One of the most untruthful things possible, you know, is a collection of facts, because they can be made to appear so many

different ways.” - Karl A. Menninger


Reporting Reporting MandatesMandates

Assessment processes at VCCS colleges have been mandated for the past eighteen years. Senate Document No. 14, 1986 Virginia General Assembly, provided the legislative mandate for assessment reporting and SCHEV established the initial guidelines for colleges and universities to respond.

The Final Report of the Governor’s Blue Ribbon Commission on Higher Education (February 2000) called for the creation of a Quality Assurance Plan to:

(1) define core competencies for written communications, oral communications, quantitative reasoning, scientific reasoning, critical thinking, and information literacy;

(2) identify measures to assess students’ knowledge and skills;

(3) provide a vehicle to present results publicly.Report of the VCCS Task Force on Assessing Core Competencies,

July 18, 2002


Local Reporting Local Reporting MandatesMandates

“Annual reports will be produced to account for what has been accomplished in enhancing current programs and implementing new ones, and to identify emerging developments that call for updates to the curriculum planning process.”

TCC Comprehensive Five-Year Curriculum Plan (2003-2008), Page 1, August 2003

“While you are experimenting, do not remain

content with the surface of things. Don't become a mere recorder of facts, but try to

penetrate the mystery of their origin.” - Ivan Pavlov



Curriculum planning at TCC will occur in accord with the following principles:

The curriculum will be structured and articulated in ways that provide students with multiple opportunities for connection and advancement as they enter, progress through, exit, and potentially return to the college.

The curriculum will be responsive to the distinctive needs of business, industry, government, agencies, and the larger community, continually reinventing itself in relation to the immediate requests and long-term developments in various occupational and technical fields.

The curriculum will be developed in consultation with employers and transfer institutions that are the direct recipients of graduates.

All curricula will be built and maintained on a foundation of excellence. “In the college transfer area, the college will benchmark itself to the most rigorous standards of achievement in 4-year baccalaureate degree-granting institutions. In the fields of occupational-technical study, the college will look to business and benchmark its performance goals to the standards of the relevant industries themselves” (Bearings on the Future: The TCC Strategic Plan).

TCC Comprehensive Five-Year Curriculum Plan (2003-2008), Pages 1 & 2, August 2003



TCC will pursue the following corresponding strategies:

The college will regularly review programming at peer community colleges across the nation with the goal of identifying current and emerging curricular areas for possible development and implementation. As resources permit, faculty and administrative teams will… bring back model programs and practices for adaptation and implementation at TCC.

Through focus groups, advisory committees, and other means of formal and informal relationships with employers and the higher education community, the college will actively solicit input for needed program enhancements and development as well as communicate the range and availability of existing programs.

The college will actively partner with those who are the primary stakeholders in the successful outcome of students, looking for ways to engage their support and sponsorship of students throughout the learning process.

The college will put into place a systematic process for assessing programmatic and student learning outcomes, the results from which will be used for continuous improvement of disciplines and programs. Other college activities, processes, and initiatives (e.g., faculty hiring, adjunct faculty initiative, accreditation, professional development and renewal, budget development, grant proposals, major gifts campaign) will be aligned with the pursuit of curricular excellence.


Recap Why We Recap Why We AssessAssess

External NeedsRequirements for SACS Accreditation

Report of Institutional Effectiveness for SCHEV

Quality Assurance Plan for VCCS

Internal NeedsCurriculum Planning Process & Annual Reports for TCC

Intellectual CuriosityIntellectual Curiosity

ProfessionalismProfessionalism


Supporting Supporting ResourcesResources

Consultation Services provided by the Coordinator for the Office of Student Outcomes Assessment (OSOA)

Assessment-centered assistance such as word-smithing intended outcomes or developing assessment measures

Related pedagogical assistance such as instructional strategies, syllabus design, recordkeeping or tracking data

OSOA Website http://www.tcc.edu/welcome/

collegeadmin/OIE/SOA/ index.htm


Reporting Reporting ProcessProcess

TasksTasks“Lead Deans, in consultation with program/discipline faculty, complete in its entirety the Program/Discipline Review and Outcomes Assessment Report for each of their respective programs and/or disciplines scheduled for review.”

Approved by the Vice President for Academic and Student Affairs on April 9, 2004 http://www.tcc.edu/welcome/collegeadmin/OIE/SOA/review/Process.htm

Provosts screen the reports and offer feedback prior final submission to the Vice President for Academic and Student Affairs.

“One important distinction in assessment methods is between techniques that directly determine whether students have mastered the content of their academic programs and those that ask students to reflect on their learning.” – C.A. Palomba & W. Banta (1999)


Reporting Reporting ProcessProcess

Annual DeadlinesAnnual DeadlinesApril 1st – Lead Deans submit draft report to Provost(s) for review.

May 1st – Lead Deans submit final report to Vice President for Academic and Student Affairs.

Scheduled Program/DisciplinesScheduled Program/Disciplines4 Programs and 2 Disciplines were in the planning phase last year, but are in the implementation phase this year.

An Additional 15 Programs and 24 Disciplines are in the planning phase this year (2003-04), and will implement their review and assessment plan next year (2004-05).


Report FormatReport Format

Sections to the Sections to the PlanPlan

Background/ Descriptive Section

Programmatic Outcomes Section

Student Learning Outcomes Section

Other Noted Data Section


Report FormatReport FormatElements to the Outcomes SectionsElements to the Outcomes Sections

Intended Student Outcomes: Knowledge, skills, or values students should be able to demonstrate after completing your program or discipline. Intended Programmatic Outcomes: Non-student aspects that contribute to your program’s viability.Assessment Measures: Tool or indicator that yields the most timely and accurate data used to determine attainment of a stated outcome.Minimum Competency: A standard that represents a satisfactory, acceptable-level of performance.Actual Results: The real results of the data collected for each assessment measure.Highlights: Analysis and cited evidence that confirms outcome attainment.Enhancements: Analysis and cited evidence that identifies areas for improvement; strategies planned for the coming year to address these areas.Enhancement Update: Describe the impact and level of success of each enhancement activity (COMING NEXT YEAR).


Report FormatReport Format

Working with the FormWorking with the FormMS Word Template Online Form

Simply download the form from the OSOA website to begin working with it

http://www.tcc.edu/welcome/collegeadmin/ OIE/SOA/review/Form.htm

Demonstrate functions, then we’ll get back to writing outcomes and selecting assessment measures


Step #1: Step #1: OutcomesOutcomes

Writing Intended Student Learning Writing Intended Student Learning OutcomesOutcomes

Outcomes describe specific learning behaviors that students should exhibit in the context of the program/discipline/course.Outcomes are the specific skills, values and attitudes students should exhibit that reflect the broader goals (e.g., for students in a freshman writing course, this might be “students are able to develop a cogent argument to support a position”). Often in the assessment literature, “objectives”and “outcomes” are used interchangeably.

“A student drove himself so hard that he missed the learning curve.” - Pun of the Day



Effectives OutcomesEffectives Outcomes Use action words that specify definite, observable behaviors.Indicate an appropriate level of attainment.Are assessable through one or more indicators.Comprehensively and meaningfully define a goal.Are realistic and achievable.Use simple language.

“Two mathematicians arguing about even

numbers were at odds.”

– Pun of the Day







Difficulty Writing Outcomes?Difficulty Writing Outcomes?For each of your stated goals, what are the specific student behaviors, skills, or abilities that would tell you this outcome is being achieved?Ideally and briefly, what would a skeptic need (what evidence needs to be present, what specific behavior needs to be visible) in order to see that your students are achieving the major outcomes you have set out for them?In your experience, what evidence tells you when students have met these outcomes – how do you know when they’re “getting” it?



Curriculum MappingCurriculum MappingProgram

GoalProgram Outcome

Depth of Coverage

Required Courses

ABC 101

ABC 110

ABC 200

ABC 210

ABC 220

ABC 295

Goal 1

Outcome 1

Basic Introduction or Review X X X

Advanced Application X X

Outcome 2



Goal 2

Outcome 3



Outcome 4

Basic Introduction or Review X X



Identify & Review Existing Identify & Review Existing Assessment MeasuresAssessment Measures

What information on student learning/performance do you currently collect? What assessments are administered?

Direct and Indirect Measures

How do these data sources relate to your newly articulated outcomes? Does the data provide rich understanding?

Are there gaps between the information you collect and your outcomes?

What other information do you need to have to understand whether students are meeting these outcomes?

Step #2: Step #2: MeasuresMeasures


Step# 2: Step# 2: MeasuresMeasures

Outcome-Assessment Outcome-Assessment Comparison MatrixComparison Matrix

ACC 201Homework

(HW)1HW2 Quiz1 HW3

Midterm

Outcome 1: Accounting Cycle

X X X X

Outcome 2: Closing Entries

X X X

Outcome 3: Financial Statements

X X


Step #2: Step #2: MeasuresMeasures

Outcome-Test Item MatrixOutcome-Test Item Matrix ACC 201 Homework 2

Question1

Question 2

Question 3

Question4

Question 5

Outcome 1 (75% of Questions)

X X X

Outcome 2(25% of Questions)

X

Outcome 3(Bonus Question)

X


Step #2: MeasuresStep #2: Measures

Assess at Specific Points in Time or Assess at Specific Points in Time or Over Time? Over Time?

Think about the length and scope of the assessment techniques you are currently administer or you’d like to implement.

“Education is what survives when what

has been learned has been

forgotten.” - B. F. Skinner


Step #3: Step #3: ResultsResults

Establish Minimum CompetencyEstablish Minimum CompetencyWhat constitutes exemplary, satisfactory, needs improvement (e.g., rubrics/scoring guides with examples of A/B/C/F-level work)?What are your realistic expectations (e.g., 90% will graduate/pass)?

“The great difficulty in education is to get

experience out of ideas.” - George Santayana



Collect Data & Collect Data & Compute Actual ResultsCompute Actual Results

Determine when and how to collect the data.For course-based assessment data faculty of selected courses administer assessments and compile their results.Lead Deans decide how to collect and enter the course-based assessment data for analysis.Compare expectations against actual results.

“You can observe a lot by just watching.”“We made too many wrong mistakes.”

- Yogi Berra






Step #4: Highlights Step #4: Highlights & Enhancements& Enhancements

Closing the Loop: Interpreting and Using the ResultsClosing the Loop: Interpreting and Using the Results

HighlightsAnalyze the assessment measure data and cite evidence that confirms the program/discipline/ course is accomplishing its intended outcome.

EnhancementsCite evidence that identifies areas for improvement and state enhancement strategies/activities planned for the coming year that address these areas.Implement planned strategies the following year (Fall 04) and report their impact (Spring 05).


Step #5: Other Step #5: Other Noted DataNoted Data

Other Noted DataOther Noted DataNote all significant trends, developments, accomplishments, issues, opportunities, threats, etc.. that were NOT cited elsewhere in the report.Make this the “brag book” portion of the report (e.g., faculty accolades, regional/state/national awards).


Programmatic Programmatic OutcomesOutcomes

Programmatic OutcomesProgrammatic OutcomesTop portion of the report form (between the Background/Descriptive Section and the Student Learning Outcomes Section).Apply same process to the non-student portion of the report.Programmatic outcomes ensure your program’s viability (e.g., FTEs, headcounts, modernized equipment).


Closure ActivityClosure Activity

Writing Intended Outcomes and Selecting Appropriate Assessment Measures (Time Permitting)

Write an outcome

Then pick a…

course

Measure, rubric

and/or test item

time to collect

way to collect


Additional Additional InformationInformation

Need Assistance? Clarification?Visit the Student Outcomes Assessment (SOA) Website

http://www.tcc.edu/welcome/collegeadmin/OIE/SOA/ index.htm

Call or Email Mike Bryan

(757) 822-1073

[email protected]


Questions?Questions?

Documents

From Here, Go Anywhere Overview of the Reporting Process Program/Discipline Review and Outcomes Assessment Report Dr. Michael E. Bryan, Coordinator Office