Http://jw65/ Engaging Students in Distance Learning Jim Waters The iSchool at Drexel

http://www.pages.drexel.edu/~jw65/

Engaging Students in Distance Learning

Jim WatersThe iSchool at Drexel

Jim Waters, Susan Gasson Drexel University, 2009

If students have already paid us, why should we care if they are engaged ? Altruism – the long view?

Pragmatism – they can always leave taking their tuition money with them

Student feedback and tenure decisions?

Word gets out! – social networks Students can vote with their virtual feet and

Their very real wallets


Agenda

How do I know if students are engaged? What is the effect of:

Question design? Course scaffolding? Instructor moderation? Being an entertaining instructor?

So what?


Research Study Analyzed course interactions via discussion board on

Blackboard learning system. 12 online MS courses (info. systems./info. Science) 313 Students, 11,497 messages Posts to discussion board + small group discussions

Analyzed Thread depth, thread length, participants Cognitive content of message Interactive intent of message Patterns of message sequences

Examined student outcomes related to interaction Pre and Post questionnaires Demographics and Attitudinal data


Three modes of learning engagement

Individual Participation Active Course Involvement Iterative Social Engagement

Fluid: students can move between modes reacting to drivers


Mode 1: Individual Participation The semi-transparent participant

Interacts with materials Internalizes knowledge Contractual obligation postings Broadcast messages Superficial learning Hermit!


Mode 2: Active Course Involvement Demonstrates (some) genuine interest

Interacts with peers (after a fashion) Translates community knowledge Relates posts to own experience or knowledge Internalizes community knowledge Ego-centric approach Small group or clique interactions


Mode 3: Interactive Social Engagement Motivated for interactive learning

Committed to greater group learning Interacts freely with peers Looped learning cycles Iterative internalizations/externalizations Social construction of knowledge


How can we tell what is going on?


Mode 1: Course Participation

How Do We Know A Project Is On Track? Instructor 10/21/07 2:36 PMRE:How Do We Know A Project Is On Track? S18 10/25/07 12:09 PMRE:How Do We Know A Project Is On Track? S10 10/25/07 6:29 PMRE:How Do We Know A Project Is On Track? S21 10/25/07 8:30 PMRE:How Do We Know A Project Is On Track? S17 10/26/07 7:38 PMRE:How Do We Know A Project Is On Track? S22 10/28/07 6:19 PMRE:How Do We Know A Project Is On Track? S2 10/28/07 7:04 PMRE:How Do We Know A Project Is On Track? S4 10/28/07 10:24 PMRE:How Do We Know A Project Is On Track? S10 10/28/07 10:26 PMRE:How Do We Know A Project Is On Track? S7 10/28/07 10:46 PMRE:How Do We Know A Project Is On Track? S8 10/29/07 12:59 AMRE:How Do We Know A Project Is On Track? S11 12/14/07 11:34 AM


Mode 1: Course Participation


Mode 2: Active Course InvolvementUnanswered questions Instructor 10/5/07 3:23 PM

RE:Unanswered questions S1 10/5/07 6:59 PMRE:Unanswered questions S17 10/6/07 3:41 AM

RE:RE:Unanswered questions S19 10/8/07 12:33 AMRE:RE:RE:Unanswered questions S20 10/8/07 10:52

AMRE:Unanswered questions S13 10/6/07 10:52 AM

RE:RE:Unanswered questions S18 10/7/07 4:14 PMRE:RE:RE:Unanswered questions S6 10/9/07 9:45

PMRE:Unanswered questions S12 10/6/07 11:04 AM

RE:RE:Unanswered questions S20 10/7/07 10:34 AMRE:Unanswered questions S9 10/7/07 6:49 AMRE:Unanswered questions S21 10/7/07 4:36 PMRE:Unanswered questions S10 10/7/07 5:31 PMRE:Unanswered questions S4 10/7/07 10:59 PMRE:Unanswered questions S19 10/8/07 12:07 AMRE:Unanswered questions S8 10/8/07 2:21 PM

BOK as a communications/marketing tool S12 10/9/07 1:18 PMRE:Unanswered questions S8 10/8/07 2:48 PMRE:Unanswered questions S14 10/12/07 2:33 PM

RE:RE:Unanswered questions S12 10/12/07 3:22 PMRE:Unanswered questions S15 10/12/07 3:26


Mode 2: Active Course Involvement


Mode 3: Interactive Social Engagement


Mode 3: Interactive Social Engagement


What makes students engage at higher

levels in a distance-learning course?


Question Frequency vs. response

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00L

IS-1

b

LIS

-1a

IS-3

IS-2

IS-4

IS-1

b

LIS

-2c

LIS

-2b

CO

M-1

b

CO

M-1

a

IS-1

a

LIS

-2a

Questions

Posts/question

Linear (Posts/question)


Question Design

I want you to cook up a systems development project (real or imagined). Describe the goal(s), the objective(s) of the project and the scope of the work the systems analyst for the project. Post your goals, objectives and scope by around Thursday of this week. I'd then like each of you to comment a bit on each other's work. [Cooking up a new project]

Critically evaluate the author's FAST approach. Is it useful? Practical? What are some alternatives? Is this a "real" model that could be used on "real" projects? [Fast or slow]

I would like each of you to initially focus on one fact finding technique, your contribution should be a critical (but brief) examination of that technique within the domain of systems analysis. [Fact-finding]


And the Winner is

I would like each of you to initially focus on one fact finding technique, your contribution should be a critical (but brief) examination of that technique within the domain of systems analysis. [Fact-finding]


I want you to cook up a systems development project (real or imagined). Describe the goal(s), the objective(s) of the project and the scope of the work the systems analyst for the project. Post your goals, objectives and scope by around Thursday of this week. I'd then like each of you to comment a bit on each other's work. [Cooking up a new project]


GoodI want you to cook up a systems development project (real or imagined). Describe the goal(s), the objective(s) of the project and the scope of the work the systems analyst for the project. Post your goals, objectives and scope by around Thursday of this week. I'd then like each of you to comment a bit on each other's work. [Cooking up a new project]

150 posts Several sub-threads extremely deep (7 or 8 levels) Critique, feedback, support and facilitationWell-placed faculty moderation, nudges rather than cattle

prods Well-bounded but open-ended: students define problem Deliberately pitched as a cooperative task Concrete (well-defined) task Students negotiate the task meaning collaboratively


AverageI would like each of you to initially focus on one fact finding technique, your contribution should be a critical (but brief) examination of that technique within the domain of systems analysis. [Fact-finding]

85 postsModerate sub-thread depth (mostly 3 or 4 levels) 31% were messages from Instructor to students20% were messages from students to InstructorWell-placed faculty moderation, focus on challenging assumptions. Reasonably open-ended problem Far less cooperative inter-student activity Not pitched as a cooperative activity Students not answering a common question, but question is

defined


Bad


46 postsLimited sub-thread depth - mostly 2 (question then single response)45% were messages from Instructor to students37% were messages from students to Instructor18% were student-student messages Faculty intervention much more critical (didactic) Five questions in one: 1 was open-ended 4 bounded Very little cooperative inter-student activity Not pitched as a cooperative activity


Good questions tended to be

Early rather than later First question in the week Early weeks better than later weeks

Open (scope), but bounded (problem structure) Permitted students to call upon their personal experience

with IT or organizations Permitted many ways to approach the issues

Negotiated rather than defined Permitted collaborative interpretation Allowed students to contribute by defining their own take

on the question. Relevance to students helps – war stories,

company policies and approaches


Bad questions tended to be

Following a highly-interactive question or later in the course Later questions much less interactive and constructive

across courses than earlier questions Cognitively complex

Containing multiple parts that needed to be considered in turn, or

Overly abstract, so students could not draw on their personal experience.

Socially isolating Fewer opportunities for interpretation and collaboration

in answering the question.


How does course scaffolding affect

student engagement?


Scaffolding Something to hang construction on Solid foundation for task

Materials Discussion Support for knowledgeable peers to contribute

Task requires an extension of prior abilities Provides a structure on which students can

build knowledge Task must be just beyond current unaided

skills but doable with help


Poor ScaffoldingYou've been asked to read the ALA Code of Ethics plus two other codes of ethics of your choice. What did you learn from this process? Did any common themes or concerns tend to emerge? What did you relate to in the ALA Code of Ethics? Were there things that seemed problematic, or that you disagreed with?

Codes of Ethics Instructor 1/28/08 3:15 PMRE: Codes of Ethics S5 1/31/08 7:00 AMRE: Codes of Ethics S15 1/31/08 5:49 PMRE: Codes of Ethics S13 1/31/08 9:17 PMRE: Codes of Ethics S14 2/1/08 12:05 AM

RE: Codes of Ethics S16 2/1/08 12:54 PM RE: Codes of Ethics S9 2/1/08 1:08 PM

RE: Codes of Ethics S17 2/3/08 1:20 PM RE: Codes of Ethics S11 2/2/08 3:26 PMRE: Codes of Ethics S18 2/3/08 4:23 PMRE: Codes of Ethics S19 2/4/08 5:33 PMRE: Codes of Ethics S6 2/5/08 6:24 PMRE: Codes of Ethics S1 2/5/08 11:03 PM


Better ScaffoldingCan ethical behavior really be codified by a professional organization?Can ethical behavior be enforced? How?


Unscaffolded Learning materials

A long list of codes of ethics web-pages Three abstract ethics articles

A body of solid material but this did not directly relate to the posted question or give a framework for answering the question


Scaffolded Learning materials

A description of ethical models Worksheet for ethical decision making

Actions and consequences Responsibilities and obligations A theoretical and pragmatic platform from which

discourse could be built Three sparse pages of bullet-points

Materials contained less information but provided a structure for thinking.


How does instructor moderation affect

student engagement?


Moderated or leave-alone ?

Two sections of an IS course delivered at the same time – same basic syllabus

~Same number of students (23/24) Selected six “identical” questions on each

section Different Instructor approach Heavy moderation vs. lightweight

moderation


Question Heavy Moderation

Low Moderation

Systems Analyst as problem solver 69 74

Agile methods 96 97

Project design 150 97

Requirements Analysis 96 83

Fact Finding 85 90

Data Modeling Practice 182 180

Average 112 103

Tot messages 238 268

Tot words 26270 57128

Average words/student message

110.38 213.16

Questions and Approach (messages)


Agile methods Heavy Moderation Low Moderation

Total Messages 96 97Instructor – student

messages16 (17%) 0

Student messages 80 97

Student-student messages

28 73

Student-instructor messages

52 24

Deep thread messages (students)

65 44

Deep sub-threads ( 4 levels or greater)

10 8

Deep sub-threads w/o instructor intervention

2 8


Instructor interaction and student posts

0

500

1000

1500

2000

2500

INST-2

INST-4

INST-8

INST-8

INST-6

INST-8

INST-3

INST-4

INST-7

INST-3

INST-5

INST-1

Instructor DB postsStudent Posts


Are students more engaged when the

Professor is entertaining?


Deconstructing the Entertaining Professor

Highly knowledgeable industry professional Very Popular Instructor High level of interaction with students Regular internet chat presence Projects personality into discussion

High percentage of social interactions Voluntarily discusses hobbies, weather, music,

Disneyworld, cooking, children, Dickens, vintage cars, pets, gardening, insects, Star Wars, birds, Nintendo, Scrabble, foreign films, beer ….

Injects lots of jokes


Student participation

Students Discussion Board Visits

Posts Topic Threads

Posts/Thread

23 24095 2745 67 40.9

Student Posts

StudentPosts/Thread

FormalQuestions

StudentPosts/Question

AverageThreadDepth per question

1648 24.7 30 67 8.6


Never mind the width, feel the quality

Threads show evaluation and analysis Some hypothesis formation Fairly advanced cognitive activity Some student-student interactionBut, some chaff among the wheat

5% Un-focused Anecdotes 32% fluff posts: “LOL,” “Awesome [dead rock star] story!”

“OMG,” “Pictures of gardening implements,” “Lawyer Jokes”

50% contribute to learning 33% student-Instructor messages

Still, pretty successful overall High student satisfaction Grades were comparable with Prof. Serious


Introducing Professor Serious

Same course – Same Syllabus Highly knowledgeable industry

professional Very skilled Instructor Low level of direct interaction with students Strong Topic-focus Little social interaction


Student participation

Students Discussion Board Visits

Posts TopicThreads

Posts/Thread

24 13079 1458 352 4

Student Posts

StudentPosts/Thread

FormalQuestions

StudentPosts/Question

AverageThreadDepth per question

1334 3.79 14 95.3 5


Quality ? More collaborative learning Messages longer and more detailed Fairly advanced cognitive activity Much stronger student-student interaction

Stronger awareness of value of peer interactions

So what? High task-Focus 2.5% fluff posts 80% substantive knowledge-building posts 15% student-Instructor messages

Successful overall Moderate student satisfaction Grades were comparable with Prof. Entertaining


The tale of the tape Instructor participation of Prof. Entertaining

inflates thread depth slightly More posts, but smaller % focus on knowledge-building

Student-student posts more productive than student-instructor posts

Overall productive (knowledge building) activity was about the same for the two Professors

Student satisfaction slightly lower for Prof. Serious than Prof. Entertaining Fewer posts, but greater % focus on knowledge-building

But Stronger thought leaders for Prof. Serious Explicit kudos for peers in Prof. Serious


Conclusions Question design can be crucial to

engagement Discussion needs to be framed as

collaborative not competitive Relevance to students helps – war stories,

company policies and approaches No payoff for frequency of questions

Course scaffolding aids engagement Focus and framing of questions Must support task in concrete manner


Conclusions Successful course moderation hinges

on quality not quantity or frequency Knowing when to intervene Does not mean “being absent”

Being entertaining is not essential for success Tradeoff between popularity with students

and peer-engagement by students May shift focus from peers to instructor Excessive interaction is a lot of work


Students think they are learning more from peer-interaction

this was so helpful because often I was struggling with the same thing so I could learn from their errors and gain new information from the answers to their questions

I was moved to comment on how refreshing the lack of competition in the communications for the online classes seemed to me. It was a discussion and a sharing of experiences

Honestly, in the second half of the course, I have felt like I must be a pariah. Apart from the professor, I can't get anyone to respond to my posts- a very lonely feeling. I have posted to the the weekly board with little feedback

No question that the on line discussion was critical to getting me through the class. There were mostly questions about how to..I've never done this before.

I felt lost and inexperienced most of the time. I have no real full time work experience and I felt I had nothing much to contribute and compared to the rest of the posts mine would feel really insignificant.


Related References Waters, J. '

Social Network Behavior, Thought-Leaders and Knowledge Building In An Online Learning Community', Proceedings of Hawaii Intl. Conference on System Sciences (HICSS-41), Knowledge Management Track, Jan. 2008.

Gasson, S. and Waters, J. “How (not) to construct ALN course questions that encourage student participation in peer collaboration and knowledge construction,” 40th Hawaii International Conference on System Sciences, Hawaii, January 2007.

Waters, J., and Gasson, S. "Social Engagement in an Online Community of Inquiry," 27th International Conference on Information Systems (ICIS), Milwaukee WI, 2006.

Waters, J. “Determinants of Engagement in an Online Community of Inquiry,” The 12th Sloan-C International Conference on Online Learning, November 2006, http://www.sloanconsortium.org/conference/proceedings/2006/ppt/1162852287092.pot

Waters, J., and Gasson, S. "Strategies Employed By Participants In Virtual Learning Communities," Hawaii Intl. Conference on System Sciences (HICSS-38), Collaboration Systems and Technology track, IEEE Software Society, Hawaii, January 2005, p. 3b.

A full list of publications, with full copies of articles, is available at http://www.pages.drexel.edu/~jw65/publications.htm

Documents

Http://jw65/ Engaging Students in Distance Learning Jim Waters The iSchool at Drexel