Effective SLO Assessment in the Online Environment

8/13/2019 Effective SLO Assessment in the Online Environment

1/61

Effective SLO Assessment

in the Online Environment

Jim Julius, MiraCosta College

Kevin Kelly, Wiley Learning Institute

Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute


2/61



Before we get started




3/61

What is your role at your institution?

Faculty

Professional Development/Instructional Design

Academic Technology/eLearning

Student support staff

AdministrationAcademic Affairs

Administration - Student Affairs

Administration - Other

3Effective SLO Assessment in the Online Environment - Module 1


4/61


5/61

What are your SLO responsibilities?

SLO and/or assessment committee member

Department coordinator for SLO/assessment

Course author/creator

Instructorconduct assessment of SLOs

Staffsupport the systems where SLO data lives

SLO lead for department

Assessment officer



6/61

Overall agenda

Choose appropriate assessment strategies

Choose appropriate assessment tools

Managing consequencesof online SLOassessment

Analyze, report, and act upon online SLO data



7/61



Module 1:

Choose appropriateassessment strategies




8/61


9/61

(Re)imagine Assessing

Learning Outcomes


10/61

(Re)imagine Assessing Learning Outcomes

Key factors to keep in mind:

Consider the level(s) of learning you want students to

achieve or demonstrate Make the objectives measurable

Avoid common misconceptions

TRUE or FALSE:onlinemeans all learning or assessment must take

place behind a computer screen.

TRUE or FALSE:distanceeducation means students are located far

away

Effective SLO Assessment in the Online Environment - Module 1 10


11/61


12/61


Consider Level(s) of Learning

Psychomotor (skills)

Observing: Active mental attending of a

physical eventImitating: Attempt to copy a physical behavior

Practicing: Try a specific physical activity over

and over

Adapting: Fine tuning, make minor adjustmentsin a physical activity

12Effective SLO Assessment in the Online Environment - Module 1See http://ets.tlt.psu.edu/learningdesign/objectives/psychomotor

Image source:

http://www.sharjah.ac.ae
http://ets.tlt.psu.edu/learningdesign/objectives/psychomotorhttp://ets.tlt.psu.edu/learningdesign/objectives/psychomotor


13/61


Consider Level(s) of Learning

Affective (attitudes or feelings)

Receiving: Shows awareness.

Responding: Shows attention and motivation.

Valuing: Demonstrates commitment.

Organizing: Integrates and prioritizes new

values.

Characterizing by value: Behavior reflects new

values.

13Effective SLO Assessment in the Online Environment - Module 1http://www.uwsp.edu/education/lwilson/curric/affectiv.htm

Image source:

Karin Kirk, SERC

http://serc.carleton.edu
http://www.uwsp.edu/education/lwilson/curric/affectiv.htmhttp://www.uwsp.edu/education/lwilson/curric/affectiv.htm


14/61

Thought prompt

To what extent is it

possible for students

to achieve higherlevel outcomes in

online/distance

environments? Harder

Roughly the same

Easier14Effective SLO Assessment in the Online Environment - Module 1

Creative

CommonsAttribution:

billsoPHOTO
http://www.flickr.com/photos/billsophoto/5243121852http://www.flickr.com/photos/billsophoto/5243121852http://www.flickr.com/photos/billsophoto/5243121852http://www.flickr.com/photos/billsophoto/5243121852http://www.flickr.com/photos/billsophoto/5243121852http://www.flickr.com/photos/billsophoto/5243121852http://www.flickr.com/photos/billsophoto/5243121852http://www.flickr.com/photos/billsophoto/5243121852http://www.flickr.com/photos/billsophoto/5243121852


15/61


Consider Environment

Tech-enabled F2F classes

Assessing outcomes might reflect

enhanced capabilities with tech

Flipped classes

Assessing outcomes might reflect

action or performance focus

Online classes

Assessing outcomes might include

student interaction with the world



16/61


17/61

Select Appropriate

Assessment Strategies


18/61


19/61

Assessing performance

Different strategies used for class assessments

Written: tests, quizzes, essays, written assignments,

case studies Oral: recitations, student presentations, oral exams

Manual: lab practicum, individual and team projects,skills observation

What other types of assessments do you use?



20/61

Reasons to assess learning differently

Traditional F2F Methods

Standardized testing

(multiple choice, etc.)

Written work (essays,research reports, etc.)

Student presentations

Lab work

Team projects

Possible F2F Issues

Does not support all

learners

Hard to require studentfeedback in classroom

Not enough time in class;

usually not recorded

Hard to make up if youmiss the lab exam

Hard to determine who

did what, and when



21/61

Thought prompt

Does one assessment provide enough

information to show that a student has

learned something?



22/61

Apply Universal Design for Learning principles

Universal Design for Learning (UDL)

The core goal is to Teach every student

Principles guide instructors to provide multiple pathways forstudents to succeed

Representation: Share content in more than one format

Engagement: Use multiple ways to make learning meaningful

Expression: Provide more than one way for students to show whatthey know

See full presentation about UDL and online assessment

http://prezi.com/1rm9g73wkic_/udl-and-online-assessment/http://prezi.com/1rm9g73wkic_/udl-and-online-assessment/http://prezi.com/1rm9g73wkic_/udl-and-online-assessment/http://prezi.com/1rm9g73wkic_/udl-and-online-assessment/


23/61

Providing assessment choices and levels

Provide alternative questions

Provide different assignments

Let students submit in different formats

Use two or three of these strategies together



24/61


25/61



Module 2:

Choose appropriateassessment tools




26/61


27/61

Select tools for online activities


28/61

Using technology for assessment

Low-stakes quizzes/High-stakes exams

Quizzes through a Learning Mgmt System (e.g., Moodle)

Written assignments - Iterative/peer-review process

Calibrated Peer Review

Online paper submission tools (e.g., Turnitin)

Electronic portfolios

Written assignments - Grading/feedback process Screencast (e.g., Screenr, Echo360)

Tablet Apps (iAnnotate)



29/61


Individual or small group presentations

Online presentation tools (e.g., VoiceThread)

Labs

Commercial lab kits sent to studentshomes

Kitchen sciencelab

Virtual labs/simulations

Intelligent tutors (e.g., ALEKS)



30/61


Individual or small group projects

Google docs or wiki pages for team organization/coordination/workflow

Blogs or ePortfolios to share and assess final products

Performance-based assessment (science, creative arts)

Upload observation logs by trusted expert(s)

Audio/video recording

Other?



31/61

Choosing technology for assessment


synchronous

asynchronous

physical digital

collaborative

individual

x: where

y: when

z: who


32/61

Choosing appropriate technology

Questions to consider

How will using this technology help students learn?

Do students have access to this technology?

Do students have access to training and support to

use this technology?

Is this technology accessible to students with

disabilities?



33/61


34/61

Choosing appropriate technology

Resources to investigate

Virtual resource site - teaching with technology

http://www.umuc.edu/virtualteaching/

Discussion-based online teaching

http://cgi.stanford.edu/~dept-ctl/tomprof/postings.php (#561)

How technology influences student success

MERLOT & Pachyderm (CSU tools) http://www.merlot.org; http://www.pachyderm.org

http://www.umuc.edu/virtualteaching/http://cgi.stanford.edu/~dept-ctl/tomprof/postings.phphttp://caret.iste.org/index.cfm?fuseaction=answers&QuestionID=1http://www.merlot.org/http://www.pachyderm.org/http://www.pachyderm.org/http://www.merlot.org/http://caret.iste.org/index.cfm?fuseaction=answers&QuestionID=1http://cgi.stanford.edu/~dept-ctl/tomprof/postings.phphttp://cgi.stanford.edu/~dept-ctl/tomprof/postings.phphttp://cgi.stanford.edu/~dept-ctl/tomprof/postings.phphttp://www.umuc.edu/virtualteaching/


35/61


36/61

Activity

1. Make a teaching-with-technology plan

2. Share elements of your plan with the group

Objectives Resources Activities Assessments



37/61

The role of rubrics

Tie rubric criteria to specific outcomes

Repeatedly assess across multiple assignments



38/61

Consider support for assessment tools

Campus apps (e.g., LMS featuresquizzes or

tests, assignment submissions)

3rdparty apps (e.g., Turnitin, Google Picasa)

Publisher tools (e.g., WileyPLUS)

Digital feedback tools (e.g., digital rubric withinePortfolio, audio/video/screencast feedback)



39/61

Put it together with

ePortfolios


40/61

Gathering evidence over time

Electronic portfolios allow different stakeholders to view

progress of

ePortfolio resource site:

http://eportfolio.sfsu.edu

http://eportfolio.sfsu.edu/http://eportfolio.sfsu.edu/http://eportfolio.sfsu.edu/http://eportfolio.sfsu.edu/http://eportfolio.sfsu.edu/http://eportfolio.sfsu.edu/http://eportfolio.sfsu.edu/http://eportfolio.sfsu.edu/http://eportfolio.sfsu.edu/


41/61

Align with Outcomes at Multiple Levels



42/61

ePortfolios enable



43/61

Assessment via ePortfolios

ePortfolios to document:

Self-assessment

Peer review

Instructor

feedback (rubric)



44/61

Assessment

examples

Rubric

comments

can be sharedin addition to

automated

systems



45/61

provide

feedback

& rubrics

encourage& share

exemplars

cultivate

critical

thinking

elevate

self

evaluation

reward

peer

review

use game

& UDL

elements

provide

guidelines

& grades

foster

learning

improvement

require

attention

to purpose

make

collaboration

a priority

Synthesis



46/61

Added benefits of ePortfolios

Demonstration of student achievement

Accreditation

Program Assessment

Student Bridge to Transfer/Workforce

Student Recruitment



47/61



Module 3:

Managing consequencesof online SLO assessment




48/61


49/61

Potential Benefits

Students may develop showcase

resources beneficial to their next steps

Thinking about improving online SLO

assessment may result in improvements to

on-ground courses

Other?



50/61


51/61

Discuss:

Authentication & Academic Integrity

Do you (should you) require use of an

institutional course management system

for online assessment?

What do you do about third party

systems?

Under what circumstances should

proctoring be encouraged? Required? And

what should proctoringinclude?



52/61



Module 4:

Analyze, report, and actupon online SLO data




53/61

Potential Concerns

Institutions may not be collecting/

aggregating assessment data in useful

ways related to outcomes

Institutions may not be disaggregating

assessment data into DE and non-DE

Institutions may not be seriously looking at

DE vs. non-DE results and following

through with improvement plans



54/61

Activity

In your table, look at the data packet

provided. Assume you are all part of an

academic discipline/department.

What does the data tell you? What does it

not tell you? What questions does it cause

you to ask?

What is required to get buy-in for this kind

of review process?



55/61


56/61


57/61



58/61



59/61

Points to Ponder

Do tools and processes support meaningful

collection & analysis of course SLO data by

course modality? Is there faculty buy-in for this? Are DE courses defined, tracked, and reported

in a consistent manner?

Is it possible/reasonable to make comparisonsacross departmental, college-wide and

statewide DE data? What are the limitations?



60/61

Points to Ponder

Do you have a way of tracking DE

program outcomes? (Do you have a way

of tracking DE programs?)

What will the ultimate outcome be of SLO

mania combined with ratcheting up DE

standards?



61/61

Documents

Effective SLO Assessment in the Online Environment