Academic Program Assessment & Evaluation

Academic Program Assessment & Evaluation

Fred RovaiJim Downey

Background

• “College standards are becoming diluted and there is a fuzziness about what faculty teach and what is expected from students” (Commission on the Future of Higher Education).

• “Outcome-based education means starting with a clear picture of what is important for students to be able to do, then organizing the curriculum, instruction, and assessment to make sure that this learning ultimately happens” (Spady, 1994).

SACS Comprehensive Standards

• 3.3.1 – The Institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.

• 3.4.1 – …establishes and evaluates program and learning outcomes.

• 3.5.1 – …identifies college-level competencies within the general education core and provides evidence that graduates have attained those competencies.

Student Learning Outcomes (SLOs)

• Skills, knowledge, and attitudes that students are expected to acquire in a program and be able to demonstrate upon course and program completion.

• Similar to instructional objectives .• Characteristics

– Specific – SLOs should specify what students are to achieve.– Measurable – You should be able to measure the degree to which

SLOs are met.– Achievable - Are the SLOs achievable and attainable?– Realistic – Can students realistically achieve the outcomes with

available resources?– Time bounded – When are the students to achieve the outcomes?

SLOs

• SLOs are expressed and measured at the course, program, or institutional level.

• SLOs reflect a shift of focus from “What am I teaching” to “What are my students learning”?

Curriculum Alignment

• SLOs• Essential questions

Intended Curriculum

Achieved Curriculum

• Assessment for Learning– Formative– “In-progress”– Provides corrective feedback

• Assessment of Learning– Summative– “After the fact”– High stakes

Taxonomy of the Cognitive Domain

Level Sample Action Verbs

Creating Generating, planningEvaluating Critiquing, recommendingAnalyzing Differentiating, organizingApplying Executing, implementingUnderstanding Summarizing, explainingRemembering Retrieving, recalling

Taxonomy of the Affective DomainLevel Sample Action Verbs

Characterization by value set

Avoiding, embracing

Organization Theorizing, formulating

Valuing Supporting, debating

Responding Following, volunteering

Receiving Accepting, listening

Taxonomy of the Psychomotor DomainLevel Sample Action Verbs

Non-Discursive Communication

Choreographing, gesturing

Skilled movements Dancing, acting

Physical activities Moving precisely

Perceptual Punting, catching

Basic Fundamental Movement

Running, manipulating

Reflex movements Stretching, extending

Significant Learning Categories• Learning how to learn: self-regulation, organizing

time, data-driven decision making, etc.• Motivation/interest/values• Human dimension: acquiring professional behaviors,

leadership, teamwork, etc.• Cognition• Integration/connection• Application/problem solving/critical thinking• Skills: using technology/communication, etc.

Course Syllabus

• Course goals, instructional objectives, and/or SLOs must be consistent with: – University mission and goals– School mission and goals– Program goals and SLOs– National disciplinary norms

• Assessments must be identified and linked to each instructional objective or SLO

Assessment

• Dressel, P. (1957) writes:"...a grade (is)...an inadequate report of an

inaccurate judgment by a biased and variable judge of the extent to which a student has attained an undefined level of mastery of an unknown proportion of an indefinite amount of material." (p. 6.)

Assessment

• Anderson et al. (1975) provide the classical description of assessment:

“Assessment, as opposed to simple one-dimensional measurement, is frequently described as multitrait-multimethod; that is, it focuses upon a number of variables judged to be important and utilizes a number of techniques to assay them – Its techniques may also be multisource … and/or multijudge” (p. 27).

Assessing SLOs• Focus is on assessing student:

– Skills (psychomotor domain)– Knowledge (cognitive domain)– Attitudes (affective domain)as a result of the educational program.

• Purpose: To obtain information that can be used by program faculty to answer the following questions:– Are our students learning what we think is important?– Are they learning what they need to succeed in this field or profession?– Are we improving in our ability to help students learn?– Should our curriculum or teaching strategies be modified?– Are there other techniques or additional resources that would help our

students learn more effectively?

Sample SLO Assessments

Direct Measures• Standardized test results• Performance on licensure or

professional exams• Embedded assignments in

specific courses• Internships• Performances• Portfolios of work samples• Job placement rates

Indirect Measures• Student surveys• Graduate follow-ups• Focus groups• Exit interviews

Student Learning Assessment Cycle

Define SLOs

Identify evidence

Identify tasks & methods

Administer assessments

Review results & identify changes

Evidence Requirements

• Identification of SLOs and how they are addressed for each program, e.g., curriculum maps

• Assessment of each SLO• Efforts to:– improve student learning – make the program more effective

based on these assessments

Simple Program Curriculum MapProgram

SLOs Courses

Course 1 Course 2 Course 3 Course 5 Capstone Course

SLO 1 Introduced Reinforced & assessed

Reinforced & assessed

Reviewed

SLO 2 Introduced & assessed

Reviewed & assessed

SLO 3 Introduced & assessed

Reinforced& assessed

Reviewed

SLO 4 Introduced & assessed

Reviewed & assessed

SLO 5 Introduced & assessed

Reinforced Reviewed & assessed

Assessment Opportunities

• Embedded assignments in specific courses– Traditional vs. authentic, application driven

• Student performances– Student thesis/dissertation defenses

• Internships• Program comprehensive examinations• Course evaluations (self, peer, student, & supervisor)• Standardized surveys• Focus groups

Putting Assessment into Practice at Regent• Assessment schedule will be advanced to address SACS concerns

and start building the culture• New assessment matrices will be used• Executive summaries of results will be generated for reporting to

school deans and Office of IE• Identify one high-stakes outcome for you program that would be a

major issue if not achieved• Begin to think about program outcomes beyond SLOs

– Job placement rates– Retention rates– Employer satisfaction– Etc.