Assurance of Learning:Why, what, who, when and how?

Marta Colón de Toro, SPHRAssessment Coordinator

College of Business AdministrationUPR-Mayagüez

2

Today’s Learning Goals

After completing this workshop you should be able to:

1. Explain the importance and purpose of assessing student learning outcomes.

2. Recognize AACSB’s standards on assessment.

3. Describe the process of assessment and the challenges it represents.

4. Develop a plan to conduct classroom assessment.

5. Identify and apply appropriate instruments to directly assess student learning in the classroom.

6. Create and maintain a course portfolio to document your classroom assessment.

3

A Transformation Process

• Demographics• Academics• Experience• Talents• Attitudes /Values

Completing

Student

Input Teaching-Learning OutputCurriculum

Seminars and Competitions

Internship/CoopStudent Associations

Extracurricular Experiences

• Knowledge•Skills•Abilities•Experience•Attitudes and Values

Incoming Student

4

Effective curricula is…

…pertinent

…coherent

…strategic

…dynamic…assessable

…intentionally designed!

5

Objective #1

Explain the importance and purpose of assessing student learning outcomes

6

Have you ever asked yourself…?

Are my students learning?

How much are they learning?

How well are they learning?

What affects their learning?

Objective #1

7

Assessment

…is a way of describing student

learning to an identified audience

for clearly- articulated reasons.

…produces information useful for communication and decision-making.

Objective #1

8

Purpose of Assessment

Assessment

Reveals the Gap

Intended Outcomes

1. Act ethically

2. Quantitative Analysis

3. Apply models

4. Decision making skills

5. Communication skills

Actual Outcomes

1.

2.

3.

4.

5.

Objective #1

9

For example…UPR-Utuado

MisiónEl Programa de Contabilidad de la Universidad de Puerto Rico en Utuado, reconoce que tiene ante si un reto y compromiso con la comunidad altamente industrializada y comercializada de nuestros tiempos. Su objetivo es capacitar al estudiantado con las herramientas necesarias: destrezas, conocimientos, aptitudes y actitudes que respondan a la demanda y expectativas del mundo laboral y en los principios técnicos necesarios en la Contabilidad General, en el análisis de información de transacciones financieras en forma sistemática y organizada.La finalidad de esta tecnología es desarrollar el potencial individual de estudiantes para que puedan competir en una economía global, ampliar sus oportunidades de empleo y proveer a la comunidad de personal técnico capacitado.

Objective #1

10

MISIÓNEl programa de Administración de Empresas tiene como misión lograr la formación humana, ética, intelectual y profesional del estudiante para que provea un servicio de excelencia en la administración de las empresas privadas y públicas, y en la comunidad. Además, el estudiante desarrollará las competencias necesarias para desempeñar posiciones de liderato y enfrentar retos con la disciplina y el grado de flexibilidad que exige el ambiente cambiante empresarial, y así contribuye a su éxito profesional.

Another example…AreciboObjective #1

11

Para lograr su misión, el Departamento tiene como metas: Familiarizar al estudiante con los conceptos, teorías, y técnicas

administrativas y fiscales de las empresas privadas y públicas. Capacitar al estudiante para recopilar datos y analizar y

presentar información de su área para que se pueda utilizar en el proceso decisional.

Fomentar en el estudiante la comprensión de su responsabilidad como ciudadano en el desarrollo de la organización, la economía, la sociedad y la cultura del país.

Capacitar al estudiante con las destrezas intelectuales, técnicas y profesionales que le permitan integrarse exitosamente a la constante innovación en la empresa y la sociedad.

Desarrollar en el estudiante la agudeza de percepción para reconocer los cambios del ambiente externo e integrar las competencias que faciliten identificar mecanismos administrativos que permitan la adaptación exitosa de la empresa.

Another example…AreciboObjective #1

12

Assess Learning to:

1. Provide feedback and guidance to individual students.

2. Assist the school and faculty members to improve courses and programs.

3. Assure external constituents that the institution meets its academic goals.

AACSB

Objective #1

13

Levels of Assessment

Institutional Learning Goals

Course Goals

Lesson Objectives

Program Learning Goals

Objective #1

AACSB’s Focus

Today’s Focus

14

When to Assess

Entry Demographics Academic achievement Previous Business KSAOs

Midpoint Measure progress / added value

Exit Measure attainment of expected learning outcomes

Follow-up Further education Workforce Personal goals

Objective #1

15

Effective assessment…

Is goal directedEthics, writing, critical thinking, teamwork

Uses multiple measuresTests, essays, projects, transcripts, surveys

Show student development overtimeEntry, midpoint, exit, follow-up

Uses multiple types of informationDemographics, behavioral, perceptual

Objective #1

16

Challenges of Assessment

Motivating faculty, staff, students, administration

Needs realistic goal settingData managementAssuring right purposeRequires taking actionCould be costly

Objective #1

17

Objective #2

Recognize AACSB’s standards on assessment

18

AACSB Assurance of Learning Standards

AACSB only requires assessment at the program level. Other levels of assessment bring support

Assessments using samples are allowed. Assessment must use direct measures of

achievement Course-embedded exercises Course-embedded examinations Stand-alone examinations

Indirect measures may be used to supplement direct measures of achievement.

Objective #2

19

AACSB’s Learning Expectations at the Bachelor’s Level

Educate students in a broad range of knowledge and skills as a basis for careers in business.

Build on the students’ pre-collegiate educations to prepare them to enter and sustain careers in the business world and contribute positively in the larger society.

Provide knowledge and skills for successful performance in a complex environment requiring intellectual ability to organize work, make and communicate sound decisions, and react successfully to unanticipated events.

Develop learning abilities suitable to continue higher-level intellectual development.

Objective #2

20

Standard 15 The school uses well documented, systematic

processes to develop, monitor, evaluate, and revise the substance and delivery of the curricula of degree programs and to assess the impact of the curricula on learning.

Curriculum management includes inputs from all appropriate constituencies which may include faculty, staff, administrators, students, faculty from non-business disciplines, alumni, and the business community served by the school.

Objective #2

21

Standard 16

Adapting expectations to the school’s mission and cultural circumstances, the school specifies learning goals and demonstrates achievement of learning goals for key general, management-specific, and/or appropriate discipline-specific knowledge and skills that its students achieve in each undergraduate degree program.

Objective #2

22

Standard 17

The bachelor’s or undergraduate level degree programs must provide sufficient time, content coverage, student effort, and student-faculty interaction to assure that the learning goals are accomplished.

Objective #2

23

Program-level learning goals

General, not specific. general knowledge and skills management-specific learning goals

State the broad educational expectations for each degree program.

Specify the intellectual and behavioral competencies a program is intended to instill.

Clarify how we intend for graduates to be different as a result of their completion of the program.

Need to be defined operationally in order to be assessable.

Normally, four to ten learning goals will be specified for each degree program.

Objective #2

24

Objective #3

• Describe the process of assessment.

25

The Assessment Process

Enable LearningImprove

Continuously

Assess Learning

Set Goals and Design Curriculum

External Reality Vision/Mission

Objective #3

26

1. Establish learning goals Consult appropriate constituencies Outcomes-based Specify level of learning

2. Identify/design enabling experiences Courses and Extracurricular activities

3. Select instruments and indicators of goal attainment

4. Assess

5. Interpret and report results

6. Identify strategies for change

The Assessment ProcessObjective #3

27

For example…1. Establish Learning goals: :

“The student will be able to discuss the importance of assessing student learning.”

2. Identify or design enabling activities: Brief presentation on topic Brainstorming Q&A

3. Select: Indicator: “Participant will present 2 arguments pro/con of assessment w/o

consulting notes.” Instrument: Focused Listing

4. Assess: List 2 arguments in favor and 2 arguments against doing assessment.

5. Measure, interpret and report results _________________________

6. Introduce change7. Revise plan

Assurance of Learning Standards

Objective #3

28

Objective #4

• Develop a plan to conduct classroom assessment.

29

Where to start? PLAN!

Step#1 - Revisit your course learning goals (CLGs). (Also known as: learning outcomes, course objectives)

A. Definition“…types of executions that the students can perform after

being taught in a way that they show that they have

learned what was expected.” Norman Gronlund

B. Purposea. Express expectations of what is to be learned in a

course

b. Provide a context for what will be learned

Objective #4

30

Learning Outcomes…

C. Should answer five questions:1. Are they aligned with the Program Learning

Goals?2. What type of learning do they address?3. Do they express the appropriate learning level?

Express the “ultimate goal”

4. Are they outcomes-based? Use action verbs!! List what will the student will be able to do as a result of

learning.

5. Are they student-centered?6. Do they address one element of learning?

Objective #4

31

Ejercicio

Al finalizar el curso, los(as) estudiantes podrán:

Manejar la calculadora para calcular datos específicos como la media y la desviación …

Solucionar problemas estadísticos utilizando la calculadora científica.

32

Hierarchy of Learning Goals

Institutional Goals

College Goals

Program Goals

Course Goals

Session Objectives

“…Apply scientific inquiry methods.”

“…Think logically and critically.”

“…basic and applied research.”

“…apply data collection techniques.”

“…list steps in the scientific research process.”

Objective #4

33

How to align goals

INSTITUTIONALLEARNING

GOALS

PROGRAMLEARNING GOALS(Those that apply)

COURSE LEARNING

GOALS

Communication Skills

Interpersonal Skills

Required

team project

…Information

Technology SkillsAssignments using

Peachtree

…Ethical

Behavior …

…Research

skills…

Complete the following for each course you teach: ADMI 3000

Objective #4

34

Program Goal Matrix

Goal #2 - “The students will use the appropriate software packages and hardware as a working tool in the daily operations of a business”.

Course/Experience Level of Learning Application of Learning

ADMI 3007 - Intro. to Data Processing

Application

Business related problems

ESTA 3001- 3002 - Business Statistics

Application

Business related problems

CONT 3005-3006 - Elementary Accounting

Application

Accounting problems

Internship/COOP

Application

Job environment

Computer-assisted courses Application

Business problems

Assigned projects in several courses.

Application

Objective #4

35

Types of Learning

Cognitive: mental skills (Knowledge) “…enumerate three major EEO laws…”

Affective: growth in feelings or emotional areas (Attitude) “…will value diversity…”

Psychomotor: manual or physical skills (Skills) “… correctly use the keyboard…”

Objective #4

36

You may want to use:

Know

Comprehend

Apply

Analyze

Synthesize

Evaluate

Define, describe, list, reproduce, enumerate

Classify, explain, discuss, give examples, summarize

Determine, develop, compute, utilize, conduct

Correlate, diagram, distinguish, outline, infer

Adapt, combine, compare, contrast, design, generate

Compare & contrast,

critique, justify

Objective #4

Bloom’s Taxonomy (Cognitive Domain)

37

Dealing with “…understand”…

Always ask yourself…How do you know the student understands?

“When he/she:Gives examples of …Determines the “correct” method to use…Discusses pros and cons… Identifies elements in a given case…etc.”

Objective #4

38

Using Measurable Action Verbs

analyze compute classify collaborate

compare appreciate contrast define

demonstrate direct derive designate

discuss display evaluate know

identify infer integrate interpret

justify list understand organize

grasp report respond solicit

state synthesize name explain

Objective #4

39

Course goals help…

The Student Clarify personal

goals Measure success Reduce anxiety Improve studying

effectiveness

The Professor Design relevant

classroom material Design relevant

assignments Design

tests/projects

Objective #4

40

Right focus! The student’s learning.

Focused on Teaching Focuses on the

professor’s effort, intent

or

Focused on Learning Focuses on the

student’s achievement

Example: “Expose the student to

recent research in organizational behavior”.

Example: “After completing this

course the student should be able to evaluate recent research in organizational behavior”.

Objective #4

41

Where to start? (cont.)

Learning Goals

1.

2.

3.

4.

5.

Objective #4

Step #2 - Identify/design activities to enable students to achieve each learning goal:

Teaching Methodologies Lecture Case analysis and discussion Oral presentations Essays/Reports Team Projects Experiential Learning

Extracurricular Activities Seminars/forums Field trips

42

Complete the course matrixPROGRAM

LEARNING GOALS(Indicate only those

that apply)

COURSE LEARNING

GOALS

ACADEMICACTIVITIES

(Methodology)

ASSESSMENTINSTRUMENTS

(Rubrics, pre/post tests…)

Interpersonal Skills

“…able to work in teams”

Required team project

Information Technology Skills

“…complete accounting cycle using Peachtree”

Assignments using Peachtree

Ethical Behavior

“…identify ethical dilemmas in marketing”

Research skills

“conduct a research study in accounting theory”

Objective #4

43

Objective #5

Identify and apply appropriate instruments to directly assess student learning at nd classroom levels.

44

Where to start? (cont.)Objective #5

Step# 3 - Select or design classroom assessment instruments (CATs) and indicators

“Classroom assessment is a simple method faculty can use to collect feedback on how well their students are learning what they are being taught”Angelo, T.A (1991)

45

Quality of student learning is directly, although not exclusively, related to the quality of teaching.

Learning is goal oriented. Students need to receive appropriate and focused feedback early and often

Classroom Assessment provides systematic inquiry and intellectual challenge.

Classroom Assessment does not require specialized training

With Classroom Assessment efforts, faculty (and students) enhance learning and personal satisfaction.

Classroom Assessment Assumptions: Objective #5

46

Classroom Assessment Techniques

Learner-Centered Teacher-Directed Mutually Beneficial Formative Context-Specific Ongoing Rooted in Good Teaching Practice

Objective #5

47

Three step process

STEP 1PLAN

STEP 3RESPOND

STEP 2IMPLEMENT

Objective #5

48

Classroom Assessment TechniquesObjective #5

Direct Measures 1 Minute paper Muddiest Point One-sentence summary Direct paraphrasing Application cards Empty outlines Pro-con grids Class journals/Portfolios What’s the principle? Rubrics

Indirect Measures Student evaluation of courses Alumni Feedback Internship/Coop

49

CATs

MINUTE PAPERAsk them…

The most important thing learnedWhat remains unanswered

Used in lecture/discussion courses

Objective #5

50

MUDDIEST POINTAsk them…

Identify what is least clear to you or the most confusing point

Used for large, lower level courses

CATs Objective #5

51

ONE SENTENCE SUMMARYAsk them…

Summarize today’s session.

CATs Objective #5

52

DIRECTED PARAPHRASINGAsk them

Explain ______ in your own words.

CATs Objective #5

53

APPLICATION CARDSAsk them

Connect today’s topic with…Mention two possible applications of today’s topic.

CATs Objective #5

54

REMEMBER…

CATs are suggestions to be adapted, not models to be adopted

Don’t ask if you don’t really want to know or are not going to do anything about it.

Objective #5

55

Examples of Measures of Achievement

Example 1 School A has defined a learning goal in ethical reasoning

for each of its four undergraduate majors. Student achievement on this goal is relevant to demonstrating satisfaction of Standard 16. The school’s faculty has defined the goal.

Learning Goal “Each student can recognize and analyze ethical problems and

choose and defend resolutions for practical situations that occur in accounting, human resource management, and marketing.”

Demonstration of Achievement The school uses course-embedded exercises in three required

introductory-level courses. Faculty in the three disciplines have developed different methods for instructing and assessing achievement toward this learning goal.

Objective #5

56

In Accounting A two-week module near the end of the introductory

course is devoted to “Ethical standards and fraud in accounting.”

A topic outline has been developed by faculty members to structure an exam on the materials of this module, and a standard set of expectations has been created for grading the exam.

In addition to this exam’s contribution to the course grade, it provides a pass/fail indication on the learning goal.

Examples of Measures of Achievement

Objective #5

57

In Human Resource Management Students must provide four written analyses of

problem situations during the course. On three of these analyses (on the topics of selection, reward systems, and job design), students are asked to respond to ethical issues.

A standard scoring key on the ethical component provides evaluation toward the course grade and a pass/fail indication on the learning goal.

Examples of Measures of Achievement

Objective #5

58

In Marketing Each student must compose a term paper analyzing

a current national or international marketing campaign.

The analysis must include a specified set of components, and ethical issues that have been presented in lectures are among the required components.

In addition to the overall grade of the paper, each student receives a pass/fail indicator on the ethics component.

Examples of Measures of Achievement

Objective #5

59

In addition to reporting course grades Each instructor of these three courses provides a

checklist of all of those students who successfully completed the ethics expectation.

This information is a part of each student’s record and all three parts of the learning goal must be achieved before graduation.

Students who fail the ethics evaluation while passing the course repeat the evaluation exercise or ethics module until they are successful.

Examples of Measures of Achievement

Objective #5

60

Step #4 Conduct Assessment

61

Objective #6

Create and maintain a course portfolio to document your classroom assessment.

62

Step #5 Document results with course portfolios

“ The portfolio is to teaching what lists of publications, grants, and honors are to research and scholarship.”

Peter Seldin

The Teaching Portfolio (Anker, 1991)

63

Uses of Course Portfolios

Document existing courses in order to:Constitute an official record of courses syllabiFacilitate the assessment of our students’

learning outcomesProvide quantitative and qualitative information

for continuous improvementServe as reference for professors that will

teach the course in the futureRender accountability to stakeholders

Objective #6

64

Proposed ContentI. Course Syllabus (In compliance with Certification #130 – UPR-BT)II. Educational Goals Addressed in the Course

1. Institutional Learning Goals Matrix2. Program Learning Goals Matrix3. Concentration Learning Goals Matrix

III. Classroom Assessment Techniques – For each assessment instrument used please include the following documentation:

1. Brief description of each technique used2. Graded Sample Instrument (High, average and low grades)3. Outcomes4. Analysis of Outcomes5. Lessons Learned6. Recommendations to improve teaching/learning7. Specific Action(s) Taken or To Be Taken

IV. Evaluation Instruments (Tests, Quizzes, Projects and Special Activities) - For each evaluation instrument used include the following documentation: (Same as above)

V. Recommendations and Proposed Action Plan

Objective #6

65

Documenting CATs

CAT’s Diagnostic Tests Pre-Post Test Muddiest Point Minute Paper Application Cards Direct Paraphrasing Course Final General

Assessment

Documentation Brief Description or

Sample copy Sample of Grades

(H,M,L) Sample Answers Analysis of Outcomes Conclusions Recommendations Actions Taken

Objective #6

66

Analysis of Outcomes

QuantitativeDescriptive statistics

Average, Median, Mode, Standard deviationQuestion by question analysis

QualitativeLook for

MisconceptionsAttitudes/prejudices

Objective #6

67

Conclusions/Recommendations

Summarize lessons learnedList actions to be taken for improvement

Step #6 - TAKE ACTION!!! Next time you teach the course read the list

and apply recommendations Refer pertinent recommendations to

appropriate forum

Objective #6

68

Stages of Grieving

Denial Anger Bargaining Depression Acceptance

On Death and Dying by Elisabeth Kubler-Ross

69

Thank You!

colon-m@rigel.uprm.edu