Is it working?Assessing hybrid faculty development

and the hybrid classroom

Andreas Brockhaus and Carol Leppa University of Washington Bothell

Hybrid Course Development Institute (HCDI) Main goal: Create a peer-reviewed, new or

redeveloped course syllabus for a hybrid format class

Cohort of 12 faculty Provide faculty incentives (currently $1000) HCDI faculty team: Learning Technologies,

Nursing, Library Designed around Community of Inquiry model

HCDI structure evolution

2010 Structure 10-week model

– 3 face-to-face meetings• 3 hours long

– 1.5 to 2 hours online each week

– Up to 4 weeks between face-to-face meetings

– Ran twice– $500 stipend

2013 Structure 6-week hybrid model

– 6 face-to-face meetings• 1 hour long• Met weekly at the same

time – 2 hours online work weekly– Ran twice– $1000 stipend

Moving from 10 to 6 weeks

Made HCDI shorter and more hybrid Moved most content and lecture online More focus on a couple specific tools Focused mainly on social presence More templates and peer reviews

Results

47% full-time participants

79% full-time participants

29% tenure-track

61% tenure-track

10 weeks 6 weeks

53% completed all requirements

96% completed all requirements

hybrid

How to assess impact

Pre- and post-HCDI participant surveys

Faculty interviews (post-HCDI) Student evaluations

Feedback

Participants found the HCDI useful Being a student was helpful Interacting with colleagues and instructors was invigorating Experiencing a hybrid course and defining hybrid was useful Wanted even more technology focus Challenges teaching hybrid included:

– Technology– Course design– Time out of class

Feedback

“There is no real substitute for an institute of this nature that asks participants to work through the many stages essential to (course) development.”

“Being a student in the class taught me to think like a student.”

Course Evaluations

Do Standard course evaluations reflect hybrid course COI evaluations?

Completed and compared COI and standard course evaluation form X

N=371 cases– 19 class sections Sum 12 thru Spr 13– 8 to 40 evaluations per section

Garrison, Vaughn; Blended Learning in Higher Education (2010)

Compared Factor AnalysisCOI confirmatory Factor Analysis 3 factor model confirmed 60% variance explained

Teaching Presence

Social Presence

Cognitive Presence

Factor analysis Form X 3 factor model 71% variance explained

Effective content learning– Content– Instructor action

Effective Communication– Relative to expectations

Student reflection– Effort and engagement

Combined COI and Form X Factor Analysis 4 factor model – 62% of variance

Teaching Presence/Instruction Cognitive Presence/content Social Presence as separate factor (COI only)

Student self reflection as separate factor (Form X only)

Correlations of items: Form X with COI questions Highly skewed data for both standard and

COI evaluations (any > than .15=“statistically significant”)

Teaching Presence– 7 items with r= .55 to .67

Cognitive Presence– 5 items with r = .50 to .60

Social Presence– 5 items with r = .25 to .39

Global course evaluation scores/Teaching Presence All forms N=1232 Mean(SD)

5-point scale questions

HCDI hybridn=32

Other hybridn=21

All other n=1179

Course as a whole 4.3(.50)

4.1(.73)

4.1(.63)

Course content 4.3(.42)

4.1(.72)

4.1(.60)

Instructor’s contribution

4.5(.46)

4.3(.68)

4.3(.63)

Instructor’s effectiveness

4.4(.48)

4.2(.85)

4.2(.71)

Comb items 1-4 4.4(.45)

4.4(.45)

4.2(.62)

CEI (7-point scale)

5.0(1.10)

5.2(.52)

4.9(1.20)

Social Presence proxy meansN=896 course sections Forms A&X Mean(SD)

7-point scale questions

HCDI alumsN=31

Hybrid NON-HCDIN=20

All other coursesN=845

Sessions engaging 6.4 (.49)

6.1 (.80)

6.2 (.60)

Student participation encouraged

6.5 (.49)

6.4 (.74)

6.4 (.56)

Help available 6.3 (.56)

6.3 (.69)

6.3 (.54)

Learning concepts in course

6.4 (.51)

6.1 (.84)

6.2 (.65)

Applying material in real world

6.4 (.46)

6.2 (.80)

6.3 (.60)

Cognitive Presence proxy means Form X only 101 Sections Mean(SD)

7-point scale questions

HCDI alumsN=13

Hybrid NON-HCDIN=11

All other courses

N=77

Learn conceptual knowledge

6.4(.63)

5.9(.99)

6.2(.71)

Dev appreciation for field

6.5(.54)

6.1(.95)

6.4(.70)

Understand material in field

6.2(.74)

5.9(.94)

6.3(.62)

Solve problems in field

6.3(.59)

5.8(1.0)

6.2(.60)

Gen intellectual development

6.4(.66)

6.0(.99)

6.4(.57)

Global course evaluation questions Pre- Post-HCDI

 N=9

Pre-HCDI Mean (SD)

Post-HCDI Mean (SD)

Course as a whole5=highest

4.0 (.51)

4.3 (.60)

Course content 4.0 (.58)

4.2 (.55)

Instructor’s contribution 4.3 (.58)

4.4 (.61)

Instructor effectiveness/content

4.1 (.54)

4.3 (.67)

Combined5=highest

4.1 (.53)

4.3 (.60)

Challenge/Engagement Index7=highest

5.4 (.58)

5.4 (.60)

Social Presence proxy evaluation items

 N=9

Pre-HCDI Mean (SD)

Post-HCDI Mean (SD)

Sessions engaging7=highest

6.0 (.42)

6.3 (.70)

Student participation encouraged

6.6 (.33)

6.7 (.27)

Help available 6.4 (.33)

6.5 (.46)

Learning concepts in course 6.2 (.46)

6.5 (.59)

Applying material in real world

6.3 (.35)

6.5 (.50)

Challenges

Identifying and tracking number of hybrid courses

Mixed institutional interest Student evals are outdated

– Inadequate for capturing social presence– Inadequate for capturing online/hybrid info

Questions

Do you have hybrid-specific course evaluations on your campus? If so, how does it differ from other course evaluations. If not is it reasonable to use CoI model to design student surveys?

References sources for COI instrument questions

Garrison, D.R., Cleveland-Innes, M., Fung, T.S. (2010) Exploring causal relationships among teaching, cognitive and social presence: Student perceptions of the community of inquiry framework, Internet and Higher Education, 13, 31-36.

Diaz, S.R., Swan, K., Ice, P., Kupczynski, L. (2010). Student ratings of the importance of survey items, multiplicative factor analysis, and the validity of the community of inquiry survey, Internet and Higher Education, 13, 22-30.

Shea, P. & Bidjerano, T. (2012) Learning presence as a moderator in the community of inquiry model, Computers & Education, 59, 316-326.

Carlon, S., Bennett-Woods, D., Berg, L. et al (2012) The community of inquiry instrument: Validation and results in online health care disciplines. Computers & Education, 59, 215-221.

AppendixTeaching Presence Spearman’s rho = .50 to .65

Form X question The course as a whole

The course content

Instructor’s effectiveness

Instructor explanations

COI Questions Facilitation questions

Facilitation

Design/facilitation

Design/facilitation

AppendixCognitive Presence Spearman’s rho = .50 to .60

Form X questions Learning content

Understanding material in field

Solving problems in field

General intellectual development

COI questions Integration

Trigger events/resolution

Integration/resolution

Integration/resolution

AppendixSocial Presence Spearman’s rho = .20 to .40

Form X questions Class sessions interesting

and engaging

Student participation encouraged

Aware expectations

Apply real world problems

COI questions Affective experience/open

communication

Affective experience/open communication

Affective exp/group cohesion

Affective exp/open communication/group cohesion

AppendixCOI Evaluation questions Teaching Presence

– Design/organization 4 items– Facilitation 6 items

• (teacher-group support)– Direct instruction 3 items

AppendixCOI Evaluation questions Social Presence

• Student-student and student-teacher

– Affective expression 3 items– Open communication 3 items– Group cohesion 3 items

AppendixCOI Evaluation questions Cognitive Presence

– Trigger events 3 items• Activities/assignments

– Exploration 3 items• Breadth of topic discussions

– Integration 3 items• Constructing solutions

– Resolution 3 items• Application