ICT Efficacy and Efficiency ICT Efficacy and Efficiency for Academic Writingfor Academic Writing

William S. Warner, Ph.D.

William S. Warner

Combine audio, visual and written feedbackCompare efficacy and efficiencyAssess instructor and student response

William S. Warner

Of all the factors that make a difference to student outcomes, the power of feedback is paramount....

Hattie, J.C (2009) Visible Learning

Information and Information and Communication TechnologyCommunication Technology

Fronter-basedFronter-based

13 Assignments 2-3 Days for Feedback

William S. Warner

Efficiency vs. EfficacyEfficiency vs. Efficacy

William S. Warner

TIME

IMPACT

Pilot Project EvaluationPilot Project EvaluationEfficacy – 29 Students Efficiency – 5 TAs Appealing idea

Unhelpful % Helpful2 1 0 1

2Written 0 5 7 7 81 Tutor 0 0 3 21 76Fronter 0 2 14 30

54Rubric 0 4 12 35 49 Audio 3 10 3 38

45

• 10-30 seconds/comment• 1-2 comments/paragraph• Too soon to judge• Technical snags• Time-consuming

Relative to Fronter comments Relative to Fronter comments

Disagree % Agreeaudio is more… 21 0 1 2efficient 20 40 30 20 0effective 0 30 50 20 0More suite for encouraging than editing

William S. Warner

Hard Copy PreferredHard Copy Preferred

• Final 3 papers – with rubric• TAs

–Spot more errors–Rubric provides equitable quality-control–Ease of evaluation: 4X4 matrix

• Students–Written comments qualified detail–Rubric quantified standards: 16-24 points–Targets strengths and weaknesses

William S. Warner

0-1 2 3 4* Did not argue a debatable issue* No evidence* No counter argument and rebuttal* No insight: an “information dump” or opinion piece

*Argued a somewhat debatable issue *Significance of issue not clear*Attempted to reason claims* Logic sometimes faulty* Supporting evidence weak* Weak counter argument and rebuttal * Shows limited insight

*Argued a debatable issue *Justify issue’s significance* Adequately reasoned most claims*Logic generally sound* Supporting evidence adequatebut not always linked to thesis*Adequate counter argument and rebuttal*Demonstrates insight

*Argued a highly controversial issue or opposed conventional thinking *Compelling justification of issue* Persuasively reasoned all claims * Logical* Supporting evidence strong and directly linked to thesis* Insightful counter argument and strong, convincing rebuttal* Demonstrates original insight

IDEAS Score X 2:

_________(out of 8)

*No clear sense of beginning, middle, end*Supporting details are insufficient *No paragraph topics

*Contains introduction, body and conclusion, but not always distinct*Supporting details are often illogically sequenced* Some paragraph topics not visible

* Distinct introduction, body and conclusion*Supporting details usually sequenced logically * Each paragraph has a clear topic

* Introduction moves reader in three steps, followed by a distinct body, and a conclusion that does not merely repeat the body*Supporting details logically sequenced* Each paragraph has a clear topi

ORGANIZATIONScore:

_________(out of 4)

* Incoherent: most sentences not clear* Not cohesive: no transitions between paragraphs and sentences* Thesis not visible* Topic sentences lacking* Excessive nominalization* Excessive passive voice* Excessive 1st person or metawriting* Excessively abstract* Excessive jargon or slang* Contractions (e.g. don’t)* Excessive “to be” verb* Agent of action in sentence often missing

*Somewhat difficult to understand* Occasionally not cohesive: some transitions missing* Thesis easily misunderstood or does not reflect argument * Topic sentences rarely visible* Wordy* Unnecessary metawriting or 1st person * Frequent abstract language* Some jargon or slang* Some contractions (e.g.don’t)* Often unnecessary negative (e.g., did not remember)* Frequent, unnecessary “to be verb” (e.g. there is, there are)* Agent of action in sentence often missing

* Generally clear* Overall cohesive: transitions present but sometimes lacking or awkward* Thesis reflects argument * Topic sentences visible* No metawriting* Acceptable 1st person* Sometimes wordy* Occasional jargon* No slang* No contractions (e.g. don’t)* Some unnecessary negative (e.g., did not remember)* Agent of action in sentence generally visible

* Easy to understand: writing flows* Cohesive: purposeful transitions create a coherent essay* Clear thesis prepares reader* All topic sentences crystallize paragraphs* No metawriting or 1st person* Concise* Precise* No jargon or slang* No contractions (e.g. don’t)* Negative (e.g. did not remember) in the affirmative (forgot)* Strong verb instead of weak “to be” (e.g., there is, there are) * Agent of action always visible

STYLEScore X 2:

_________(out of 8)

* Did not follow instructions *Many spelling and punctuation errors* Abbreviation errors* Many citation errors* Not APA reference style* < 600 words text*> 750 words text

* Followed instructions * Several spelling and punctuation errors* Several abbreviation and citation errors* Irregular APA reference style

* Followed instructions * Some spelling or punctuation errors* Some citation or abbreviation errors* APA reference style

* Followed instructions * Few or no spelling or punctuation errors* Few or no citation or abbreviation errors* APA reference style

MECHANICS

Score:_________(out of 4)

TOTAL Score out of 24

Not Approved

< 12 Weak Approval

12 - 15 Approved 16- 19 Strong Approval

20-24 GRADE RANGE

RubricRubric

William S. Warner

StudentStudent TATA Rubric ScoresRubric Scores

William S. Warner

We’re not as smart as we thinkWe’re not as smart as we think

Autumn 2012Autumn 2012Audio replaced with JINGSKYPE introducedSocial media6 (45-min) video lectures

◦Introduction◦Outline◦Clarity ◦Cohesion◦Tables & Figures

William S. Warner

William S. Warner

Social MediaSocial Media

Writing Centre http://www.umb.no/nwc/

Writing Wrongs Blog http://writingwrongsblog.wordpress.com/

William S. Warner

Autumn – 3 assignments81 students: 49 BSc, 32 MSc

EffectEffect

Unhelpful % Helpful2 1 0 1 2

JING 0 0 0 18 82Tutor 0 0 3 15 82

Paper 0 0 11 39 50Rubric 0 1 12 41 45

I found JING as helpful as the tutorDisagree % Agree2 1 0 1 2

5 11 23 29 28

William S. Warner

Unhelpful % Helpful2 1 0 1 2

JING 0 2 2 16 80

Tutor 0 0 12 27 61

Fronter 0 4 7 29 60

Paper 0 6 6 41 44Rubric 0 2 10 45 43

I prefer JING to Fronter commentsDisagree % Agree2 1 0 1 2

6 4 14 19 57

Spring – 10 assignments75 students: 15 BSc, 60 MSc

STUDENT: Efficient EffectSTUDENT: Efficient EffectAutumn Spring

• 98 % students found JING increased writing efficiency

• Easier to understand than cryptic or loaded sentences

• Voice tone • Emphasize/prioritize• Confidence/support

JING saved me timeDisagree % Agree2 1 0 1 20 7 20 35 38

JING motivated/gave me confidenceDisagree % Agree2 1 0 1 23 3 20 40 35

JING improved my writingDisagree % Agree2 1 0 1 20 1 33 42 24

William S. Warner

Autumn6 TAs3 assignmentsDid not track student

Spring8 TAs4 and 6 assignmentsTracked 10 students

Efficacy◦ Very effective – 4◦ Effective – 2

Efficiency◦ Very efficient – 1◦ Efficient – 5

Efficacy◦ Very effective - 1◦ Effective - 7

Efficiency◦ Very efficient - 2◦ Efficient - 6

William S. Warner

TA EvaluationTA Evaluation

NotNot 22 11 00 11 22 VeryVery Effective/Efficient Effective/Efficient

Spring TAsSpring TAsJING’s impact on student writing is noticeable when tracking re-writes.

Agree Disagree2 1 0 1 25 3 0 0 0

Which feedback method provides the most help to a student in the least amount of time?

6 JING4 Face-to-face consultation

1 Rubric1 Writing comments on hard copy

William S. Warner

Notice the lack of improvement?

30 minutes for both?

30 minutes/session30 sessions/term

Unhelpful (students) Helpful

2 1 0 1 2Autumn 10 students 0 1 2 4 3Spring 17 students 1 0 5 4 7

For night owl or procrastinator?

William S. Warner

Video LecturesVideo Lectures

I suggest that you watch the video lecture on…Half watched the video lectures

Unhelpful % Helpful2 1 0 1 2

Autumn(40) 0 0 2 63 35Spring (34) 0 0 9 35 56

William S. Warner

SpringSpringFeedback TutorialsFeedback Tutorials

80% watched Principles of paraphrasinghttp://isites.harvard.edu/icb/icb.do?keyword=paraphrasing APA Format for Referencinghttp://isites.harvard.edu/icb/icb.do?keyword=apa_exposed

Unhelpful % Helpful 2 1 0 1 2

Autumn 1 3 13 33 50Spring 0 2 5 36 58

William S. Warner

Social Social MediaMedia HalfHalf (75) students found (75) students found

Unhelpful Very helpful2 1 0 1 2

Website 1 0 18 20 12 Writing Wrongs blog 0 0 12 28 15

Student Journal 0 0 10 11 7

William S. Warner

• Develop protocol Read first – not on the fly Balloon comment Color code highlight

• Green – good• Yellow – suggest/consider• Red - error

• I suggest you watch the video lecture on cohesion, which explains how to make transitions betweens paragraphs.

• Save document on Fronter

William S. Warner

RecommendationsRecommendations

ConclusionsConclusions

Integrate JING across curriculumRequire video tutorials (e.g., EndNote)Develop social media for peer-reviewExplore MOOC automated feedback

William S. Warner