Ghent University
Faculty of Arts and Philosophy
Writing Mentors Guiding Student Writers:
Evaluating Mentoring Efficacy
through Student Reflection and Essay Analysis
Paper submitted in partial fulfilment of the requirements for the degree of
“Master in de Taal- en Letterkunde: afstudeerrichting Engels-Nederlands”
Thijs Gillioen
Academic year 2015 - 2016
Supervisor: Prof. Dr. Mieke Van Herreweghe
English Studies Group - Linguistics Section
Co-Supervisor: Dr. Sarah Haas
English Studies Group - Linguistics Section
The author grants the permission for this master’s dissertation to be made available for
consultation. Every other use is restricted by the limitations imposed by copyright, in
particular with regard to the requirement to cite the source explicitly when referring to
results from this thesis.
Thijs Gillioen,
May 2015
Acknowledgements
I would like to offer due recognition to the following people, for their effort,
motivation, and support during the writing of this thesis.
Firstly, I would like to thank Prof. dr. Mieke Van Herreweghe, for her
willingness to support this research, and for supporting the peer writing mentoring
programme since its inception.
Secondly, I would like to extend my gratitude to dr. Sarah Haas, my co-
supervisor. It is only thanks to your excellent counsel, and boundless patience in the
face of countless questions and drafts, that this paper could become what it is now.
Likewise, it is only thanks to your guidance that I have become a better writer, and
researcher, than I could ever have hoped to be when I started studying English four
years ago. You have gone above and beyond the call of a supervisor, and for that, I am
eternally grateful.
Thirdly, I would also like to thank Alexander De Soete, who was kind enough
to dedicate much of his time to aid me in my research, by joining me and dr. Haas as
the third rater for my analysis of student writing. Aside from this, our many
conversations, discussions and laughs have helped shape this paper in more ways
than one, for which I am very grateful.
Fourthly, many thanks go to the UGent writing mentors, without whom this
research would not have been possible. Their continuous, and selfless, dedication of
their free time, energy, thoughts and contributions are what make the programme
work: it is my sincere pleasure to be part of such a wonderful group.
Finally, I would like to thank my parents, for giving me the opportunity to
study, for supporting me unconditionally, and for being there for me, even when I
was often too occupied to return your kindness.
Table of Contents
Abstract i
List of Tables ii
List of Figures iii
List of Abbreviations iv
I Introduction 1
II Research Situation: Methodology & Context 3
1 Methodology: Action Research 3
2 Context: Literature Review & Research Setting 5
2.1 Literature Overview: Teaching Writing Skills 5
2.2 Writing Instruction for English Majors at UGent 7
2.3 Literature Overview: Automated Essay Scoring: A Solution? 9
2.4 Writing Mentoring Programme at UGent: Inception and Grounding 10
2.5 AR Cycle 1: Setting up a Peer Mentor Programme 13
2.6 AR Cycle 2: Lowering Barriers for Engaging with Mentoring 14
III AR Cycle 3: Evaluating the Efficacy of the Writing Mentor Programme 16
1 Literature Overview: Evaluating Writing Mentor Programmes 16
2 Evaluating the Current Programme: Two Perspectives 20
3 Perspective 1: Analysis of Pre- and Post-Mentoring Texts 22
3.1 Data Collection: Student Papers 22
3.2 Text Analysis Framework: “Reader Engagement” (Haas 2016) 23
3.2.1 Overview of Engagement Framework 23
3.2.2 Developing a Rating Scale Based on the Framework 28
3.2.2.1 Pilot Test I: Devising an Initial Rating Scale 29
3.2.2.2 Pilot Test II: Refining the Rating Scale 30
LOCs vs. L-LOCs: Temporarily De-Emphasising Spelling and Grammar 30
Marking Each Instance of a HOC 33
Marking Each HOC Equally 34
Marking Essays That Do not Meet “General Responsibilities” 35
3.3 Analysing Student Essays with the Rating Scale 36
3.4 Overview of Results 37
3.5 Discussion of Results 41
3.5.1 General Remarks 41
3.5.2 Specific Areas of Discussion: Data Trends 44
3.5.2.1 ‘Apparent’ Number of HOCs Retained vs. ‘Actual’ Number of HOCs retained 44
3.5.2.2 Essays with ‘Complex Issues’ 50
3.5.2.3 Specific HOC Trends 52
3.6 Recommendations for Programme Improvements 56
4 Perspective 2: Analysis of Student Reflections 59
4.1 Data Collection: Student Reflective Writing 59
4.2 Methodology: Content Analysis 60
4.3 Overview of Results 67
4.4 Discussion of Results 68
4.4.1 General Discussion 68
4.4.2 Discussion of Reflections per Category 70
4.4.2.1 Practical Reasons 70
4.4.2.2 Confidence and/or Motivation Boost 74
4.4.2.3 Extra Explanation Offered and/or Helps to Understand Theory 75
4.4.2.4 Appreciation of Non-Directive Approach 76
4.4.2.5 Student Advice on Mentoring (Programme) 78
4.4.2.6 Attending Multiple Mentoring Sessions 79
4.4.2.7 Appreciation of Approachability 81
4.4.2.8 Students Who Did not Feel Helped 83
(27.449 words)
4.4.2.9 Writer Development 85
4.5 Recommendations for Programme Improvements 87
IV Limitations, Conclusion, and Future Research 91
1 Limitations of the Current Research 91
2 Conclusion of the Current Research 93
3 Suggestions for Future Research 97
V References 101
VI Appendices 110
Overview 110
Appendix 1: ETTVI Scribing 3 Assignment 111
Appendix 2: Overview of General Responsibilities 117
Appendix 3: Analysed Student Essays 118
Appendix 4: Samples from Reflective Writing 214
Abstract
At UGent, a major Belgian university, students of English are expected to attain a
high level of academic writing skills. However, due to the constraints of university
funding, a single teacher was responsible for teaching 250+ students how to write,
resulting in a situation where students could wait for weeks before getting any
feedback on their work. To combat this issue, a peer writing mentoring programme
was established during the academic year 2013 - 2014. Employing students as near-
peers, this programme aims to provide guidance in students’ development as writers
by presenting them additional opportunities for non-directive support. Through an
Action Research-based investigation, this paper aims to evaluate this programme in
its current form, and make suggestions for improvements in its conduct and
practices. It did so by assessing both students’ written product, and their reflection on
the programme’s influence on their development as writers. Students’ pre- and post-
mentoring texts were analysed using Haas’ “framework for teaching and assessing
writing” (2016), thus looking into the textual manifestation of the effects of writing
mentoring. This unveiled several areas where students are greatly helped by a
mentoring session, and some areas where further improvements could be made, both
by specific training for the mentors, and by more (focussed) sessions with students.
Secondly, it analysed students’ reflective writing in order to understand students’
perspective on the mentoring process and on their development as writers. Since the
concept of writing mentoring (and, by extension, a writing centre) is alien to Belgian
students, their perceptions of the mentoring process were vital to understanding how
they can be assisted more efficiently in their development as writers. The points of
discussion in this paper may be expanded to be applicable in other mentoring
situations, especially those in similar low-budget, students-as-peers-contexts.
!i
List of Tables
!ii
Table 01. HOCs and LOCs in Reader-Friendly Writing (Haas 2016: 27) 27
Table 02. Overview of Marks, L-LOCs/LOCs/HOCs in Pre- and Post- Mentoring Essays
39
Table 03. Overview of HOCs in Each Essay 40
Table 04. Apparent Retained HOC Issues vs. Actual Retained HOC Issues
45
Table 04. Apparent Retained HOC Issues vs. Actual Retained HOC Issues (second part)
46
Table 05. Overview of Complex Issues in Each Essay 50
Table 06. Number of HOC Issues 53
Table 07. Overview of Reflective Writing Categorisation Process 61
Table 08. Overview of Categorised Reflective Writing 67
List of Figures
!iii
Figure 01. Action Research (Lewin 1946) 3
Figure 02. Alphabetical List of “Building Blocks of Engaging Text” (Haas 2016: 26)
26
List of Abbreviations
AES Automated Essay Scoring
AR Action Research
AR cycle Action Research Cycle
AWE Automated Writing Evaluation
BA Bachelor of Arts
ESL English as a Second Language
ETTV Engelse Taal- en Tekstvaardigheid
GO! Onderwijs van de Vlaamse Gemeenschap
HOC Higher-Order Cognition
LOC Lower-Order Cognition
L-LOC Lower-Lower Order Cognition
VVSKO Vlaams Verbond van het Katholiek Secundair Onderwijs
!iv
!1
I Introduction 1
This applied linguistics dissertation aims to conduct an evaluation of the efficacy of
the peer writing mentoring programme that has recently been established at the
linguistics section of the English Studies Group at UGent. To do so, it uses the Action
Research methodology (hereafter referred to as “AR”), which seeks to “bring together
action and reflection, theory and practice, […] in the pursuit of practical solutions […]
and, more generally the flourishing of individual persons and their
communities” (Reason & Bradbury 2001: 1). By engaging in the AR reflective cycle
(Lewin 1946) and making decisions based on both literature and data, this paper will
use the result of this inquiry to propose practical recommendations for improving the
programme’s efficacy.
The writing mentoring programme at UGent was established during the
academic year 2013 - 2014; it was modelled on comparable mentoring programmes
in writing centres, which are present at most universities in the US, and are
appearing increasingly frequently at universities throughout Europe, particularly in
the UK, Ireland and Germany. This programme functions within the context of the
Engelse Taal- en Tekstvaardigheid (abbreviated ETTV) classes, which are
compulsory for students wishing to obtain an academic Bachelor’s degree in English
at UGent. In these classes, mentors have been employed to provide additional
opportunities for students to receive support for their written work.
The first section of this paper will introduce the AR framework, which
underpins this dissertation. To orient the reader, the second section will then
Due to the similar subject matter and research methodology of this paper and the research 1
conducted in the author’s Bachelor paper, elements of the methodology and literature review may appear in familiar guises in both that paper and this thesis, for the sole purpose of orienting the reader towards the nature of the programme and the current state of academic research into writing mentoring.
Introduction Thijs Gillioen
!2
summarise the first two Action Research Cycles (hereafter referred to as “AR Cycle”),
research into which was presented in the author’s bachelor paper, before
commencing the discussion on the third AR Cycle, which is the current paper’s focus.
The third AR Cycle will be examined in order to assess the efficacy of the peer
mentoring programme, and to make suggestions for improvements to the
programme.
Introduction Thijs Gillioen
!3
II Research Situation: Methodology & Context
1 Methodology: Action Research
In order for this paper to make suggestions for such programme improvements, it
employs the AR methodology first discussed by Lewin (1946), and since then well-
established particularly in organisational and educational research, and in applied
linguistics (Davies & Elder 2008: 476). As an interventionist methodology, used to
“assist the ‘actor’ in improving and/or refining his or her actions” (Sagor 2000: 3), it
“inevitably lead[s] to program improvements” (Sagor 2000: 8) and “bring[s] about
improvements in classroom practices” (Richards & Schmidt 2013: 8).
Action Research involves going through a succession of subsequent reflective
“Cycles” of the given situation it aims to evaluate. These cycles are represented in
Figure 01. Since AR actively aims to improve a real-world situation, engaging with
this situation is the first step of any AR Cycle (marked as (1) in Figure 01). For this
paper, this “setting” is the writing component of the previously-mentioned ETTV
Research Situation Thijs Gillioen
Figure 01. Action Research (Lewin 1946)
(1)
(2)
(3)
(4)
(5)
(7)
(6)
!4
classes, which is offered to students wishing to obtain a bachelor’s degree in English
at UGent.
Within the situation, Action Researchers define a specific “issue”, based upon
“initial observations [and] existing data” ((2) in Figure 01). Within the context of this
research, the “observation” is that, based on existing data from two previous AR
cycles, students reacted positively to mentoring sessions, and there is the subjective
notion that mentors can help guide students towards becoming better writers. The
issue defined here is that this positive perception is based on limited data, and there
is no data that objectively quantifies either text improvement or writer development
for students in the current setting.
The next step in the AR Cycle is to plan an action ((3) in Figure 01) to respond
to the issue defined in step (2). The action planned here is to evaluate the
programme’s efficacy from both a text- and reflection-based perspective.
Subsequently, the action is carried out and reflected upon ((4) and (5), respectively).
After these steps are taken, the findings are reported (6), which is where this thesis is
situated within the current AR cycle. Although the scope of this dissertation ends at
this step (6), the improvements suggested will be tested when the research(er)
reconnects with the real-world situation (7), which will simultaneously and
subsequently serve as the basis for the next AR Cycle.
Given this interconnectedness of these AR Cycles, it is neither feasible nor
desirable to present one AR Cycle as an independent entity. As such, before detailing
the current paper’s research questions, the next sections will summarise the research
and findings of the first two AR Cycles (which were investigated as part of this
author’s BA research paper) to orient the reader to the background of this
dissertation. Accordingly, to reflect the stages of AR and to ensure a logical
Research Situation Thijs Gillioen
!5
presentation of arguments, the relevant literature will be presented in each section,
rather than dedicating one full section to all relevant literature in its entirety.
2 Context: Literature Review & Research Setting
2.1 Literature Overview: Teaching Writing Skills
The ability for students to write “clear, well-structured texts of complex
subjects” (Council of Europe 2001: 61) is considered one of the core skills for students
to acquire over the course of their education, both during secondary school (where
this emphasis is most prominent in the primarily theoretical ASO strand), and during
their education at college or university. In Flanders, the official final attainment levels
for secondary education require students to be capable of taking into account “the
most important conventions of written language” (Flemish Government 2016), but it
is not specified what these conventions are.
The largest secondary school network in Flanders, Katholiek Onderwijs
Vlaanderen , narrows this down somewhat by requiring student-written texts to have 2
a “clear text structure”, although this is noted to be beyond the aims of the basic
teaching plan (VVKSO 2014: 20). It does not, however, define what a clear text
structure is: it only indicates it does not require students to be able to “combine
complex elements into a coherent text” (VVKSO 2014: 19-21). The other major
secondary school network, the public net GO!, does not specify beyond the
government’s requirements of students adhering to “the most important conventions
of language” (GO! 2014: 48). Therefore, in both settings, it is left up to individual
secondary school teachers to define what constitutes a well-written text.
Previously known as “Vlaams Verbond van het Katholiek Secundair Onderwijs” (VVKSO); 2
cited as such in this dissertation since the current official curriculum plans predate the name change.
Research Situation Thijs Gillioen
!6
When students enter higher education, research has indicated this scattered
writing education results in, at best, a widely diverse range of writing proficiencies
(De Wachter & Heeren 2012: 55), and, at worst, “serious problems when trying to
apply structure to texts” (Berckmoes & Rombouts 2009: 6 [translation]). In short,
texts written by these Flemish students force the reader to “distill the meaning from
the text, or ask the sender for extra information” (Berckmoes & Rombouts 2009: 18
[translation]), making it “very hard for the reader to follow” (Berckmoes & Rombouts
2009: 17 [translation]). Based on this, students in this education system thus, at
most, only reach the first stage in the “macro-stages in the cognitive development of
writing skill”, defined by Kellogg (2008: 4), “Knowledge-Telling”, wherein they
simply “tell what [they] know” (Kellogg 2008: 3), focussing on their “thoughts[,] and
not how the text itself reads” (Kellogg 2008: 6).
Therefore, it is left to institutes of higher education to teach more complex
cognitive principles: text structuring; Higher Order Cognitions (abbreviated HOCs
(Bloom et al. 1956)); or writing for a reader, which Kellogg considers a part of his
third (and final) macro-stage, that of “Knowledge-Crafting” (2008: 4). However, in
order for an aspiring writer (here, a student) to reach this stage of “Knowledge-
Crafting”, Kellogg argues a span “more than two decades” (2008: 2) is required,
which is not realistic within the setting of a student trying to attain a university
degree.
Although Kellogg’s demand for time is impossible, his other demands are
arguably at least partially feasible, but not met, in this specific context. He argues any
attempt to improve one’s writing skills can only be achieved through “repeated
opportunities to write and through timely and relevant feedback” (Kellogg &
Raulerson III 2007: 237 [emphasis added]). Kellogg’s demand for feedback is
Research Situation Thijs Gillioen
!7
supported by Hattie, who, in his meta-analysis of learning methods, noted feedback
to be one of the most effective tools for learning (2009: 162). In addition to feedback,
Kellogg argues the most successful model for learning writing skills is one wherein a
student can engage in “cognitive apprenticeship” (2008: 19), which involves a student
“learning by observing, rather than doing” (2008: 19), in which case the best results
can be attained by “observing readers who responded to one’s own written text plus
receiving additional written feedback” (Kellogg 2008: 19). This ‘ideal’ model of
writing instruction will be contrasted with the current system of writing instruction at
UGent in the following section.
2.2 Writing Instruction for English Majors at UGent
At Ghent University, Bachelor-level students of English have to attend three
compulsory (and successive) proficiency courses (ETTV), which include, among other
proficiencies, learning and improving writing skills. The first of these, ETTVI, enrols
between 250 and 300 students, and teaches these students about writing by gradually
presenting them the “building blocks of reader-friendly writing” (Haas 2016: 25) over
the course of a number of lectures. In between these lectures, students get to
experiment with these cognitively challenging HOCs in a number of smaller
seminars. As part of their final assessment for the course, students are given at least
three mandatory writing assignments, thus, arguably, at least partially fulfilling the
needs for repeated opportunities to write argued for by Kellogg - whose argument for
Research Situation Thijs Gillioen
!8
weekly opportunities to write and receive feedback is, again, not practically feasible
within the (European ) university setting. 3
However, due to the realities of the current academic climate, the university
cannot dedicate enough staff members to meet Kellogg’s demand of relevant, and
particularly timely, feedback. Whereas Kellogg’s research indicates feedback is best
delivered within minutes (2008: 18), it can often take one or more weeks for students
of ETTV classes to receive feedback on their written assignments, at which point they
may already have begun writing for the next assignment. In the case of the papers
analysed for this dissertation, there was a two-month gap between the due date and
availability of feedback. Although this two-month period did include exams and the
Christmas break, this does not change the fact that this much time went by before
feedback was provided - although no further essay assignments were to be made at
this point, students did have to partake in essay exams, thus timely feedback would
still have been relevant.
Additionally, the lack of sufficient staff members renders engaging into any
form of cognitive apprenticeship, as defined by Kellogg, impossible. Given the 250 to
300 students enrolled in the ETTVI course, engaging in any form of apprenticeship
would require about ten full-time teachers - which is currently not a viable option.
Research has noted significant differences between the “American model” and the “Central 3
European University model” (Harbord 2003: 3) for writing education and assignments. These differences have led to different approaches in writing instruction: whereas the current Belgian setting allows for about 15 hours of in-class writing instruction for English majors, American universities spend at least 100 hours of in-class writing instruction for non-English majors, thus arguably coming much closer to achieving Kellogg’s demand for repeated opportunities to write.
Research Situation Thijs Gillioen
!9
2.3 Literature Overview: Automated Essay Scoring: A Solution?
The issues of understaffing for the teaching of writing are not limited to the setting of
this dissertation, but are common throughout the world: Shermis calls into question
“the feasibility of recruiting a sufficient number of qualified human graders to
provide final scores [for writing assignments]” (2014: 54). Likewise, Vista et al.
mention “significant [financial and time-related] costs to mark, becoming exorbitant
for courses with large numbers of students” (2015: 1).
One often-cited solution to this problem is automated essay scoring (hereafter
referred to as “AES”), which hopes to significantly reduce both the time between
presenting an essay and getting feedback, and the pressure on teachers of writing. As
such, significant funding and research has been devoted to establishing the validity of
AES. However, the applicability of AES for ESL (English as a Second Language)
students is still subject of evaluation (Warschauer & Grimes 2008; Weigle 2013). In
addition, it may not be technically possible for AES in its current state to deal with
complex HOCs such as argumentation, as noted by Deane (2013: 15). Finally, current
research into AES systems shows “inter-human rater performance was significantly
better” (Shermis 2014: 55) than human-computer performance when marking
writing assignments. This human-computer reliability is even “lower in classroom
contexts” (Warschauer & Grimes 2008: 24). These realities currently prevent AES
from becoming standard practice for the purpose of marking writing assignments.
However, even if AES could be employed to reduce or eliminate the teachers’
time spent on marking essays, it does not resolve the lack of opportunities for
cognitive apprenticeship. There is only modest evidence that using AES or other
forms of Automated Writing Evaluation (AWE) has a positive influence on the quality
of the texts produced, or that the use of AWE can result in “more general
Research Situation Thijs Gillioen
!10
improvements in writing proficiency” (Stevenson & Phakiti 2014: 63). Additionally,
Salomon noted these automated systems “are not designed to upgrade students’
intelligent engagement” (1993: 180). This limitation was observed again in recent
research that claimed usage of AWE systems resulted in “none of the broadly iterative
process through which writers hone their content, sharpen their organization, and
thus learn to transition from writer-based to reader-based prose” (Warschauer &
Grimes 2008: 33). For these reasons, neither AES nor AWE can currently provide a
resolution to the lack of opportunities to engage in cognitive apprenticeship. Because
of this, cognitive apprenticeship is still highly time-intensive when teaching large
groups, and thus impossible to achieve, even if the teacher would no longer have to
deal with marking essays.
2.4 Writing Mentoring Programme at UGent: Inception and Grounding
Because Automated Essay Scoring is not currently a viable solution to the problem of
time-intensive marking, or the lacking opportunities for cognitive apprenticeship, a
different solution is needed to combat these issues. Therefore, during the academic
year 2013 - 2014 a peer mentoring programme was established at the linguistics
section of the English Studies Group at UGent. This programme aims to improve
students’ writing by providing students of ETTV courses with more opportunities to
receive individual support on their writing assignments, in effect making them “feel
like they have had individual attention with their writing problems” (personal
correspondence, Haas: 15/09/13). To do so, the programme enrols, as “near-
peers” (Murphy 1998), a number of students (on a voluntary basis) who have
successfully completed the ETTV courses. Through doing so, the programme
administrator (who is responsible for teaching writing) hopes to at least partially
Research Situation Thijs Gillioen
!11
fulfil Kellogg’s requirement of repeated opportunities for feedback, in addition to
providing at least some opportunities for cognitive near-peer apprenticeship. This
programme has gradually taken the form of an “Instrumental programme” (Karcher
et al. 2006: 714), which they describe as a mentoring set-up wherein “the mentor’s
role is to provide guidance and advice as the student conceptualises, researches, and
completes an assigned project” (Karcher et al. 2006: 718), which, within the context
of the ETTV courses and their assignments, is the programme’s focus.
Although concerns have been expressed at the idea of students taking a more
active (mentor) role in their fellow students’ education (as writers), research has
consistently supported the efficacy of peer mentoring (programmes), noting peer
feedback can be at least as effective as feedback provided by teachers (for example,
Topping & Ehly 1998; Gensemer 2000; Hattie 2009; Gielen et al. 2010).
Furthermore, peer mentoring has several added benefits for students, such as
reducing their writing anxiety (Martinez et al. 2011: 358), “improving (…)
understanding of concepts” (Cho & MacArthur 2010: 328) and “developing an
awareness of their writing” (Hutchings 2006: 259). On top of this, Cho & MacArthur
argue that peers are often in a better position to explain difficult concepts than
experts, “who use knowledge that novice students cannot refer to” (2010: 329).
Additionally, they indicate students’ close proximity to each other allows for them to
“detect problems from their own perspective”, thus “generating solutions to the
problems” (Cho & MacArthur 2010: 329). Some research has even posed that
“without peers’ assistance in both ideas and language, it is very difficult, if not
impossible, for [students] to grapple with these two issues [reading their text from an
outsider’s perspective and thinking of revision strategies] single-handedly” (Min
2006: 135). To further allay concerns related to mentors’ suitability to provide writing
Research Situation Thijs Gillioen
!12
support, students wishing to become mentors in the programme receive between
fifteen and twenty hours of additional specific training in both advanced writing and
mentoring skills. Therefore, one can argue that mentors can provide meaningful
support to near-peers during their writing process.
This mentoring programme, as described above, is in many of its aspects
similar to a writing centre. Writing centres have been attached to most American
universities for several decades (e.g. DuBois et al. 2002; Karcher et al. 2006), with
Girgensohn suggesting as many as 90% of all American universities have such a
centre (2012: 127). In these universities, research has indicated the “one-to-one
writing tutorials provided by a dedicated university writing support department or
Centre” (Gopee & Deane 2013: 1629) are one of the number one forms of institutional
support requested and appreciated by students learning how to write, even in non-
language majors (Gopee & Deane 2013: 1629). Europe (and Belgium, particularly), by
contrast, has seen a relatively slow development of writing centres, to such an extent
that “they remain so few and are often institutionally invisible , that there is not even 4
a number that could be cited”, even though there is a “recognis[able] growth in
writing center work in Europe” (Girgensohn 2012: 127). The popularity of writing
centres in the US is explained by their well-established effectiveness: Hoon, for one,
argues students’ writing apprehension was reduced, in addition to noting several
“positive affects to writing” (2009: 47). Hutchings provides similar arguments, and
notes these effects may be stronger the more incorporated the writing centre is within
This institutional invisibility is evidenced in the current setting: while UGent has arguably 4
had the first Belgian writing centre(s) with the peer mentoring programme (since 2013) and the Taalonthaal initiative (since 2014), the limited visibility of these programmes has made it difficult for them to stand out in the academic landscape. This is evidenced by KU Leuven being recognised as having the first Belgian writing centre with their ILT-Schrijfcentrum initiative (since 2015).
Research Situation Thijs Gillioen
!13
“the department […] and into disciplinary practices” (2006: 260). In this sense, the
writing mentoring programme fills a gap that is only recently, and slowly, being filled.
2.5 AR Cycle 1: Setting up a Peer Mentor Programme
It is within this setting that a peer writing mentoring programme was established
during the academic year 2013 - 2014, which is defined here as AR Cycle 1. This cycle
can be considered a “pilot year” for the programme: having only theoretical
knowledge gathered from literature, conference attendance and writing centre visits,
the programme coordinator, with the help of a few students, had to make choices
organically, based upon perceived needs of the mentors-to-be and the programme as
a whole. Although data were collected and analysed, reporting was, at this stage, only
done informally.
At the end of AR Cycle 1, the programme was in a functional state, and had
started to engage with students, both in one-on-one mentoring sessions and through
mentor attendance in ETTV seminars. It was at this stage the author of this
dissertation, in cooperation with the programme coordinator, decided that a
thorough and critical review of the programme’s workings was in order. This AR-
based review argued that significant improvements could be made to the programme
by improving its online components, and through these making the somewhat foreign
concept (for Belgian students) of mentoring easier to access for both mentors and
students (i.e. potential mentees). Having done this, the programme reconnected to
the real-world situation (step (1) in Figure 01), thus advancing into AR Cycle 2.
Research Situation Thijs Gillioen
!14
2.6 AR Cycle 2: Lowering Barriers for Engaging with Mentoring
Making mentoring easier to access, which was the focus of research conducted during
cycle 2, required the perspective of both mentors and students. The first perspective,
that of mentors, was crucial if the programme was to be successful: it was paramount
that mentors could be retained over the course of multiple years (thus allowing them
to gather mentoring knowledge and skills, and transfer these to new generations of
mentors). Since all mentors are volunteers, ensuring they could focus on doing
mentoring work, rather than being distracted by administrative work (such as dealing
with countless mentee e-mails) or other distractions, was imperative. This
perspective is supported by Dawson who argued “technology […] resources and tools”
are essential elements when establishing a mentoring programme (2014: 140).
One online component that was reviewed was the group’s communications
platform, which is a dedicated Facebook group. The communications in this group
were analysed and placed into categories based on the framework of English &
Duncan-Howell (2008: 598). These data were used to formulate suggestions to
improve the efficacy of this group (such as clearly marking the topic of each message
in a header), keeping in mind the relevancy of posts that were less immediately
mentoring-related for the purposes of community building, as demonstrated by
Brown (2001). These suggestions, thus, made it easier for mentors to quickly find
relevant info, which lessened mentors’ barriers to staying committed to the
programme.
The second perspective, that of the students who were the programme’s
potential mentees, was no less important. For most Belgian students, as mentioned
before, the concept of mentoring is still entirely alien, due to the specific Belgian
circumstances outlined above. Thus, making it easy and straightforward for students
Research Situation Thijs Gillioen
!15
to engage with a mentor is imperative. Through online questionnaires and an analysis
of students’ reflective writing pertaining to writing mentoring, students indicated
what barriers to mentoring they perceived. A significant barrier for many students
was the e-mailing back and forth with a mentor to establish where the mentoring
session would take place (since no dedicated room was available at this time). The
fact that mentors were, to most students, strangers, did not help this process. As a
result, student engagement with the programme was relatively low.
This situation improved when the request the programme receive a dedicated
room was granted. This change has seen significant increases in students’
engagement with the programme (for example, resulting in eighteen students
scheduling a voluntary mentoring session in November 2015, compared to two in
November 2014 (representing, percentually speaking, a 900% increase)). In addition,
a dedicated room has allowed mentoring sessions to be made mandatory for the final
ETTV writing assignment. Requiring that students see a mentor would otherwise be
impossible due to the logistical issues with arranging for locations for over 200
mentoring sessions.
At the end of AR Cycle 2, the programme had seen some substantial
improvements in its workings, which would increase student engagement with peer
mentoring during the next academic year, in which research was conducted as part of
the third AR cycle, which will be discussed below.
Research Situation Thijs Gillioen
!16
III AR Cycle 3: Evaluating the Efficacy of the Writing Mentor
Programme
Having summarised the research conducted during the first two AR Cycles of the
programme, the following section will transition to the analysis of the third (and
current) AR Cycle, which is this paper’s focus. The focus of this paper was made
possible thanks to the programme receiving a dedicated room, the necessity of which
was argued for in the research in AR cycle 2. Because of this room, mentoring
sessions could be made mandatory. This allowed for enough data to be gathered to
commit to a comparative analysis of the pre- and post-mentoring texts written by
students, which will be used to establish if (and, if so, how) their text had changed or
improved. Before commencing the discussion on this analysis, for the reader’s
convenience, the relevant literature will be presented first, to orient the reader
towards the state-of-the art concerning the evaluation of writing mentoring
programmes and writing centres.
1 Literature Overview: Evaluating Writing Mentor Programmes
As mentioned above, writing mentoring programmes, and by extent writing centres,
have been a staple of US education for several decades, which is now also slowly
finding its way into European universities. These programmes have several decades
of experience evaluating their practices and responding to changing contexts. Such
evaluations, traditionally, have been carried out from a reflective viewpoint, mainly
taking into consideration the writer’s development process, rather than looking at the
improvement of a given text (for example, Harrington et al. 2007; Archer 2008
(partially); Bromley et al. 2015). This process is analysed through analyses of the
AR Cycle 3 - Overview Thijs Gillioen
!17
writers’ (and, occasionally, the mentors’) reflections upon their own development
and, alternately, their (perceived) self-efficacy (for example, Schmidt & Alexander
2012).
This form of evaluation reflects the working practice of most writing
mentoring programmes (including the mentoring programme at UGent): the
“minimalist” non-directive approach. This approach centres on the idea that
“students write to learn, not to make perfect papers” (Brooks 1991: 3 [emphasis
added]). As such, the student is made the “primary agent”, who takes “full
responsibility” (Brooks 1991: 2) for the paper. The mentor, through this approach, is
not relegated to the position of being an editor, nor can they be held responsible for
making the text better. The mentor only serves as a ‘guide’ who helps the student
“focus[…] on his own writing” (Brooks 1991: 2). This is done by asking the student
questions about their own writing, thus making students reflect on their writing
choices. For example, in a paper that does not properly link components of its
argument together, a directive way to solve this problem would simply be to say “this
is not linked, here is how to solve it”. However, as Brooks argues, this does not bring
a student “any closer to his own paper than he was when he walked in” (1991: 2). The
non-directive alternative is to ask the student “how are these concepts linked?”, thus
provoking the student to reflect not only on the content of their essay, but also their
writing. For mentors of the UGent writing mentoring programme, who are still
students themselves, this also offers the additional benefit of shielding mentors from
leading students in the ‘wrong’ direction: the mentor only asks questions that help
the students develop as a writer, and does not directly engage with the construction
or the content of the text.
AR Cycle 3 - Overview Thijs Gillioen
!18
However, despite the non-directive approach having been a staple of writing
centres for several decades, research has questioned if an exclusively non-directive
mentoring relationship is the optimal way to ensure a student’s development as a
writer. Some research, such as that by Kullman, has called for flexibility in using
directive or non-directive mentoring, in accordance with “contextual
variables” (1998: 482). Similarly, Harbord argued that the emphasis on non-directive
mentoring is the symptom of “accommodation to traditional teaching, and fails to
take into account other views on teaching” (2003: 1), further specifying its
“shortcomings make it inappropriate for a European context” (Harbord 2003: 2).
This European context, Harbord argues, is different from the American context in
that European assignments tend to require longer and more academic essays (2003:
3), the goal of which is in fact to make a perfect paper. Further countering Brooks’
argument, Harbord gives the example of a student’s thesis: for many students, this is
the most elaborate (academic) writing they will ever undertake, thus making Brooks’
“write to learn” argument highly debatable (Harbord 2003: 3).
Harbord’s argument against an exclusively non-directive approach is
supported by more recent research conducted by Crasborn et al. which indicated the
most effective mentoring relationship is one that is “flexible in using the four
mentoring roles in mentoring dialogues” (2011: 328). These roles are defined as the
“Initiator”, the “Encourager”, the “Imperator” and the “Advisor” (Crasborn et al.
2011; 322); the former two use non-directive skills, whereas the latter two use
directive skills. They add “that it is important to strive for a successful match between
a mentor’s […] supervisory approach and a student’s […] learning needs during the
mentoring process” (Crasborn et al. 2011: 330), again demonstrating there is “no
single approach to mentoring that will work in the same way for every student […] in
AR Cycle 3 - Overview Thijs Gillioen
!19
every context” (Crasborn et al. 2011: 330). In this sense, Kullman’s, Harbord’s, and
Crasborn et al’s arguments against non-directive mentoring serve as an indication of
why the traditional method of evaluating writing mentoring programmes exclusively
through reflection on writer development , is increasingly considered inadequate. 5
This sense of self-reflection being an insufficient indicator for programme
efficacy is shared by a number of other researchers. As noted by Pleasant, the field of
research into writing mentoring programmes and writing centres is “edging toward
becoming the kind of discipline that does true social science research” (2015: 11),
which implies one cannot keep “believ[ing] that writing centers do good work simply
by relying on testimonials and anecdotes” (2015: 10). Among his specific advices for
researchers is to “pick an observable, measurable outcome” (Pleasant 2015: 10).
Likewise, Archer notes the traditional approach gathers information on “students’
perceptions of the consultation, rather than on their actual writing”, adding “that a
students’ perception of improvement may not necessarily translate into demonstrably
improved writing” (2011: 133). Furthermore, she argued “student responses were
often thin and did not allow for in-depth data analysis” (Archer 2011: 133). Although
Archer still examines both students’ and consultants’ (i.e. mentors’) comments, she
also takes into account other factors such as the actual writing and the grades
obtained. Finally, an additional argument for going beyond the traditional evaluation
method is noted by Cho & MacArthur, who pose that “many questions about the
nature of peer feedback and its effects on writing remain unanswered” (2010: 329).
This dissertation does not aim to reconcile the differences outlined above
regarding how to accurately assess the efficacy of writing mentoring programmes, nor
Since the mentoring programme aims to develop students with different backgrounds and 5
(writing) skill-sets into (reasonably) good writers, this dissertation uses “writer development” (Badley (2008), quoted in Haas (2009)) in lieu of the more standard (in this field) “writing development”.
AR Cycle 3 - Overview Thijs Gillioen
!20
does it elect to argue which side has more merit: in this author’s view, both
approaches, reflection and empirical, bring insights to the table that cannot be easily
brushed aside. As such, to assess the efficacy of the UGent writing mentoring
programme, this paper will evaluate the programme through both perspectives.
2 Evaluating the Current Programme: Two Perspectives
To evaluate from the traditional perspective of mentee reflection and perception, in
addition to the called-for perspective of a text-based analysis, the current research
poses the following two approaches through with the (workings of the) mentoring
programme will be evaluated, and through which suggestions for programme
improvements will be made.
Firstly, keeping with Pleasant’s call for an “observable, measurable outcome”
that goes beyond “testimonials” (2015: 10), this dissertation will commit to a text
analysis of the pre-mentoring version and the post-mentoring version of fifteen
different student essays. Both versions were analysed and rated using Haas’
framework for “Understanding Reader Engagement” (2016). From this, this paper
hopes to establish if, and (if so) how, students’ texts are improved as a result of
attending a session with a peer mentor. Determining the areas that have been most
improved can provide interesting data, but especially areas that have seen average to
little improvement will be of interest for the purposes of evaluating and improving
the mentoring programme’s effectiveness.
Secondly, keeping with the more traditional method of evaluating writing
mentoring programmes, students’ reflective writing was analysed to determine how
students perceive the writing mentoring programme. Although textual change may
have been noted in students’ post-mentoring essays, the process of becoming a writer
AR Cycle 3 - Overview Thijs Gillioen
!21
is predominantly a cognitive one: therefore, establishing how students have
developed as writers as a result of the mentoring process is vital to the efficacy of the
programme.
Having given this overview of the aims of the current research, this paper
turns to the first question outlined above, which seeks to confirm the hypothesis that
student writing is indeed improved by writing mentoring, and, if so, aims to discover
in which areas mentoring offers improvement. These data points will be used to
provide recommendations to further improve the programme and its practices.
AR Cycle 3 - Overview Thijs Gillioen
!22
3 Perspective 1: Analysis of Pre- and Post-Mentoring Texts
3.1 Data Collection: Student Papers
The papers collected as data for a text analysis of pre-and post-mentoring texts were
written as the third and final assignment for the ETTVI course, which is compulsory
for all students wishing to attain a BA in English at UGent. Students had previously
received over around thirteen and a half hours of class time instruction on writing,
and were given the following assignment:
[…] to write a short, engaging non-fiction essay. Your essay does not
need to be academic in topic, tone or style, but you must have an
explicit focus that is narrow enough to be appropriate for an essay of
500 words (+/-10%).
(ETTVI: Writer Development Assignment 3 (Appendix 1, p. 111) ) 6
This final assignment was further narrowed down by specifying the students’
task was to make “the reader want to start reading [their] essay” (S3 assignment,
Appendix 1, p. 111) and to “make sure the reader wants to keep reading your essay all
the way to the end” (S3 assignment, Appendix 1, p.111). These requirements
correspond with the “Initial Engagement” and “Continuing Engagement” sections of
Haas’ framework, which will be discussed below, in section 3.2.1.
One of the mandatory components of this assignment was for the students to
attend a mentoring session with one of the twenty writing mentors. They were
required to send in a draft of their assignment before coming to the session, so the
mentor could prepare. This draft is used as the “pre-mentoring” text for the purposes
of this dissertation. Based on the guidance given by the mentor, students could then
choose to adapt their text before they handed it in. The texts submitted to the teacher
Hereafter referred to as “S3 assignment, Appendix 1”.6
AR Cycle 3 - Student Essays Thijs Gillioen
!23
serve as the post-mentoring data point. Since no classes or other writing activities
were organised between the mandatory mentoring sessions and the deadline of the
writing assignment, this paper argues it is fair to assume the mentoring session was
responsible for the majority of changes made between the pre-mentoring text and the
final, post-mentoring essay.
In total, 175 students submitted an essay for the assignment. Out of this total,
fifteen essays were selected for analysis, resulting in a corpus of thirty texts (each
essay having both a pre-mentoring and post-mentoring version). These fifteen texts
were chosen based on the fact all authors of these texts had been to a mentoring
session with the same writing mentor (who is the author of this dissertation). This
was done to ensure potential differences in personal approach and mentoring style
between mentors can be eliminated as a cause for adjustments made to texts after the
mentoring session. All authors granted written permission for their texts to be
analysed for the purposes of academic research. This permission was reinforced via
an e-mail reaffirming they were comfortable with their texts being used for the
purposes of this specific dissertation.
3.2 Text Analysis Framework: “Reader Engagement” (Haas 2016)
3.2.1 Overview of Engagement Framework
If an analysis of students’ writing is to be done objectively and consistently, a
framework is needed that not only offers the tools to do so, but that is applicable to
the students’ specific writing instruction and setting. For these reasons, this paper
will use the “Framework for Teaching and Assessing Writing” developed by Haas
(2016). This framework has been developed and has been evolving over the course of
a seven year period, during five of which the developer has been a teacher of writing
AR Cycle 3 - Student Essays Thijs Gillioen
!24
at UGent. There, the framework has been used to teach and assess students’ writing
in the ETTV courses. As such, it has a history of practical application in an academic
setting. Using this framework to rate the pre- and post-mentoring versions of student
texts, this paper seeks to establish whether or not the programme is effective at
improving students’ writing. Before going into this analysis, the following section will
offer a brief overview of Haas’ framework.
In her framework, Haas starts her assessment of student writing from the
perspective of “Reader Engagement” (2016: 25). Reader engagement is broken down
into two major components: “Initial Engagement” (i.e. what allows for a reader want
“to start reading a text” (Haas 2016: 32)), which deals with elements like formatting,
title and presentation (Haas 2016: 34-36), and “Continuing Engagement” (i.e. what
ensures “it [is] easy for a reader to keep reading, or at least not [causing] a reader to
want to stop” (Haas 2016: 37)), which argues elements like coherence and cohesion
are essential. Focus (i.e. what is the underlying ‘red thread’ throughout the text that
connects everything for the reader) is additionally identified as a major contributor to
continuing reader engagement (Haas 2016: 52).
Throughout the framework, Haas uses the distinction first defined by Bloom et
al. (1956) between Higher Order Cognitions (hereafter referred to as “HOCs”) and
Lower Order Cognitions (hereafter referred to as “LOCs”). Higher Order Cognitions
are issues appearing in (academic) writing (such as global text structure) that are
cognitively hard(er) to resolve: these HOCs are, as a result, the focus of the writing
course, since they are fundamental to writing a good text. LOCs, on the other hand,
are issues such as faulty formatting, or spelling and/or grammar mistakes (Bloom et
al. 1956; Ferris 2016: 224-225). Since LOCs are cognitively easy (or, at least easier
when compared to HOCs) to deal with, it is expected that students get them right
AR Cycle 3 - Student Essays Thijs Gillioen
!25
without extensive instruction (again, as opposed to HOCs, which do require extensive
instruction). Together, HOCs and LOCs form the “building blocks of good
writing” (Haas 2016: 25). An alphabetical list of these building blocks is provided in
Figure 02, below. An overview of these issues, categorised into Initial and Continuing
Engagement, is provided below that in Table 01. A full list of definitions is beyond the
scope of this paper, but is provided in Haas’ framework (2016: 32-58).
AR Cycle 3 - Student Essays Thijs Gillioen
!26
AR Cycle 3 - Student Essays Thijs Gillioen
Figure 02. Alphabetical List of “Building Blocks of Engaging Text” (Haas 2016: 26)
!27
AR Cycle 3 - Student Essays Thijs Gillioen
Table 01. HOCs and LOCs in Reader-Friendly Writing (Haas 2016: 27)
LOCs HOCsGeneral
(components of initial, continuing and advanced engagement)
Initial Engagement Get the reader to want
to start reading
Or at least cultivate the good will of the
reader
Publishing Presentation Formatting Deadlines
Title
Mindset Writer is Responsible Reader is not obligated to read
Care About your reader About your writing
Responsibilities Engage you reader
Adjust To writing Situation Know the conventions Adhere to conventions
Continuing Engagement
Ensure the reader wants to keep reading
Or at least make it easy for the reader to keep reading (avoid making the reader
want to stop reading)
Grammar Spelling Punctuation Typing Formatting
Story Complete Linked Balanced
Proportion Promises kept
Self-Contained
Flow Coherence
Order Given/New No Gaps
Information Logic
Cohesion Linking Transitions Signalling
Relevance Concise Precision Consistent Construction
Sentences Paragraphs Entire text
Focus Narrow Precise Consistent
Advanced Engagement
Capture the reader
Something extra done for visual appeal or for otherwise making the reading effortless for a reader
Advanced ideas Advanced language use Advanced style
!28
3.2.2 Developing a Rating Scale Based on the Framework
Haas’ framework has been used (and refined) over the course of several years to teach
students how to write engaging texts (with the long-term view of transitioning into
academic writing), and has been used reliably, without major problems, by four staff
members. However, it did not have an explicit rating scale that was useable and
accessible to raters who were unfamiliar with the framework. As such, a first step in
this research was to devise a consistent and easily applicable rating scale. Because
designing a rating scale is an involved process, the designers decided to use this
opportunity to create a scale that would have applicability not only for this
dissertation (where it will be used to analyse 30 essays), but also for the purposes of
marking large-scale assignments (for example, the 250+ essays turned in for each
assignment of the ETTVI course). Rating scale development is an iterative process
(Banerjee et al. 2015), and, as such, a number of pilot tests were conducted to
construct and refine the rating scale used here.
The pilot tests were conducted with three raters: Dr. Sarah Haas, who
developed the framework; the author of this dissertation, who would be using the
framework to mark students’ essays; and an undergraduate student who will be
employing the same framework to analyse NPO newspapers for their own Bachelor
research. For the first pilot test, the three raters started with a randomly selected
student essay that was not part of this dissertation’s corpus, but that was made under
the guidelines of the same assignment.
AR Cycle 3 - Student Essays Thijs Gillioen
!29
3.2.2.1 Pilot Test I: Devising an Initial Rating Scale
The starting point of the rating scale was provided by the marking rubric for writing
assignment 3, which is based on Haas’ framework and defines a well-written text as a
text that has no HOC problems and no LOC issues whatsoever (Appendix 2, p. 117).
As such, a text that has no LOC or HOC problems was given a passing 10 out of 20
mark. This sets a high yet achievable mark for students learning writing. Since the
framework begins from a reader engagement standpoint, only a text that achieves this
minimum level of writing can be assessed additional marks for elements of
“Advanced Engagement” (Haas 2016: 27; cf. Table 01), such as a nice style. The
reason for withholding advanced engagement marks until the minimum
requirements are met is that texts with LOC or HOC problems risk disengaging a
reader before such content-related boons can be enjoyed. Because of this, texts that
do not first thoroughly attend to LOCs and HOCs do not merit credit for elements of
advanced engagement.
Essays that do not manage to keep a reader engaged throughout the entirety of
the text will therefore be given a mark lower than 10. The raters, after discussing the
pilot essay, agreed upon the following marking system. Each HOC issue in a given
text will cost a tenth of a mark (-0.1), whilst LOC issue is penalised by two tenths of a
mark (-0.2). Penalising LOCs harsher than HOCs is explained by the nature of LOCs:
since LOC issues are mostly related to formatting mistakes, which are not cognitively
hard to fix and are usually easy to pinpoint (thus indicating “the writer does not
particularly care about the reader” (Haas 2016: 34) if not properly attended to), any
LOC issue in a final text should be penalised harder than a HOC problem, which is
cognitively harder to pinpoint and fix.
AR Cycle 3 - Student Essays Thijs Gillioen
!30
3.2.2.2 Pilot Test II: Refining the Rating Scale
Based upon this initial rating system, a second round of pilot testing was conducted
by the three raters. In this round, four student essays, which were also made as part
of the same assignment (but are not in this dissertation’s corpus) were marked. As a
result of this, the raters agreed some further changes had to be made in order to
ensure the grades reflected their engagement with the essay correctly. Changes that
were made for these reasons include the introduction of L-LOCs (Lower-Lower Order
Cognitions), which will be discussed below, and the decision to penalise each instance
of a specific HOC (instead of only penalising a specific HOC once per text,
irrespective of its number of appearances), which will also be discussed in what
follows.
LOCs vs. L-LOCs: Temporarily De-Emphasising Spelling and Grammar
A point of contention when using this framework for analysing student texts was its
approach to issues related to spelling and/or grammar. Based on years of writing
(analysis) expertise, the framework developer does not believe the number of spelling
and/or grammar mistakes in a text, or alternately perfect spelling and/or grammar,
to be a sufficient metric to define ‘a good text’. In addition, the researcher wants to
move towards a focus on textual issues, thus temporarily de-emphasising spelling and
grammar in favour of HOCs, rather than simply focusing on grammar and spelling, as
students are used to from secondary school. This de-emphasis works to the benefit of
feedback givers as well: de-emphasising spelling and/or grammar issues in feedback
allows for more time and attention spent discussing Higher Order Cognition
problems. Furthermore, marking 250+ student essays limits the feasibility of
marking every single one of these mistakes successfully, when aiming to keep the
AR Cycle 3 - Student Essays Thijs Gillioen
!31
time between the due date of the essay and the moment of feedback as low as
possible. As such, the framework developer argued a certain number of grammar
and/or spelling mistakes should be given a pass, as long as they “would not annoy a
reader or impede understanding of the text”.
The other raters argued for a system that keeps the importance of the dangers
of emphasising spelling and/or grammar in mind, but that does also hold students
accountable for these lower-order mistakes and allows for a more objective metric
than “annoyance” for the reader. Solely focussing on reader annoyance allows for
subjective metrics to be implemented (and thus for variation in text scores) Indeed,
there was some disagreement amongst the raters as to how many, and what kind of
such mistakes caused annoyance. On top of this, the raters felt spelling and grammar
did still require some emphasis, given the context: the students assessed by the
framework are L2 learners of English and, by the end of their tenure at UGent, are
expected to attain a C1 level of English, which does not allow for these mistakes
(Council of Europe 2001).
A compromise between these positions was reached by taking spelling and/or
grammar issues into account as a new category, labelled “Lower-Lower Order
Cognitions” or L-LOCs (De Soete 2016; Haas 2016) . This categorisation expressed 7
the relatively low (textual) importance and low cognitive burden of these issues, but
simultaneously allows for them to be included in the rating scale and to be given the
proper amount of emphasis. Distinguishing L-LOCs from the regular Lower Order
Cognitions (LOCs - which still have a low cognitive burden) allowed for them to be
To reflect that this term is the result of a discussion between the three raters (as opposed to 7
being one individual’s terminology), this dissertation cites both Haas (2016) and De Soete (2016) for this term: it is used in each of the raters’ papers.
AR Cycle 3 - Student Essays Thijs Gillioen
!32
marked differently, and for the emphasis of ‘regular’ LOCs to remain on textual issues
rather than language issues.
Finally, to adhere to the researcher’s concern of marking 250+ essays for
language issues and to objectify what level of language issues can result in the reader
being disengaged, a set of “buffers” was decided upon that allows for a rater to both
de-emphasise these issues, but also penalise excessive amounts of such language
issues. The first buffer was set at one L-LOC issue per one hundred words: this
reflected the point at which the raters agreed the L-LOC issues started to pose a
problem for their engagement with the text. For the purposes of the 500-word essays
analysed as part of this research, this implies these essays were not penalised for the
first five L-LOC issues encountered during the analysis. After the first five L-LOC
issues, essays lost one and a half mark (-1.5). The second buffer was set between five
and ten L-LOC problems: the raters agreed that essays with this number of L-LOC
issues started to threaten reader engagement. Any essay that broke this second
buffer, i.e. in which more than ten L-LOC problems were encountered, were
penalised by three full marks (-3): this was agreed upon by the raters as being
reflective of the fact that such essays cause complete disengagement for most readers.
Thus, these buffers were agreed upon by the raters as representative of the “levels of
annoyance” that would be created for a reader when reading essays with these levels
of L-LOC issues. Although this system of buffers still reflects some personal
preference (i.e. other readers may disagree more than five L-LOCs cause
disengagement), this system allowed for a consistent rating scale to be used for the
current research; potential future research could focus on a large-scale investigation
of the effect of L-LOC frequency on reader disengagement.
AR Cycle 3 - Student Essays Thijs Gillioen
!33
Marking Each Instance of a HOC
A second point of discussion that arose during the second pilot test was how multiple
instances of a specific HOC should be approached. A certain text might, for example,
have several instances where “new” information is treated as “given” (an issue
defined as a “Given/New” HOC problem (Haas 2014; Haas 2016: 43)). The point of
discussion here pertains once more to marking: does one penalise an essay for each
instance of Given/New, or only once for each different HOC? For example, if an essay
has three Given/New problems, each appearing in a different section, does one mark
this problem once, or every time it occurs?
Like the discussion above, this point relates mostly to practicality: in longer
essays (or when facing large volumes of shorter essays), one can question if it is
feasible to start counting every single HOC to score an essay. On top of that, when
multiple instances of the same HOC appear in a single essay, one could argue each of
these instances has the same ‘cognitive’ issue underlying it. As such, one would be
penalising the same issue multiple times.
Despite these arguments, the raters agreed to mark each instance of each
HOC. Their reasons were as follows: firstly, although it is indeed arguable that
multiple instances of a given HOC problem (such as Given/New) can have the same
underlying cognitive issue, this issue does manifest itself (textually) in different ways,
resulting in variations on HOC issues that are significant enough for each to merit its
own mark. Secondly, since Haas’ framework considers ease of reading as its starting
point, it would be disingenuous not to mark each HOC problem separately, since each
HOC problem does fundamentally degrade the reading experience for the reader.
Finally, Haas’ framework does not allow for raters to internally ‘grade’ a single HOC
category (i.e., one cannot mark a certain Given/New issue as being a ‘worse’ HOC
AR Cycle 3 - Student Essays Thijs Gillioen
!34
than another Given/New problem). As such, a rater cannot determine which instance
of a HOC in a given text should be marked as ‘the most important’ or as being ‘the
biggest’. For these reasons, for the purposes of assessing disengagement in students’
texts, each HOC problem in students’ essays has been marked and pointed out as
such. A text that has three Given/New issues, for example, would lose 0.3 marks.
Marking Each HOC Equally
Linked to this discussion is the question whether or not each HOC should be
considered equal when marking essays. In its current form, the rating scale assigns
each HOC problem an ‘equal weight’: in other words, each HOC issue is penalised by
the same number (one tenth of a mark).
However, one might argue that this is an oversimplification of the
intricateness of writing an (academic) essay. Consider, for example, an essay that
lacks a sufficiently narrow Focus. In such an essay, it is easy for a reader to lose track
of the focus at multiple points throughout the text. Contrast this with an essay that
has a properly narrow focus, but at one point in the text lacks a clear link to this
Focus. To most readers, the second text would be more engaging than the first text.
However, the rating scale, in its current form, does not reflect this: both essays would
lose one tenth of a mark, since the HOC problems in these hypothetical essays, whilst
different (and thus having different implications), are graded as being equal.
The raters developing the rating scale spent some time discussing this. For the
sake of simplicity, and once again considering the implications of counting multiple
differently weighted issues in large volumes of essays, they decided against using
different weights for different HOC issues at this point, although they agreed to
AR Cycle 3 - Student Essays Thijs Gillioen
!35
review this issue after the essays in the corpus of this dissertation had been marked,
potentially changing it for the next academic year (and the next AR Cycle).
Marking Essays That Do not Meet “General Responsibilities”
A final issue that was noted was related to essays that do not meet the general
responsibilities outlined in the assignment brief for Scribing 3 (available in Appendix
1). One of these responsibilities, for example, is to write an essay that has “an explicit
focus that is narrow enough to be appropriate for an essay of 500 words” (S3
Assignment, Appendix 1, p. 111). An essay that is, for example, not focussed around a
single point, but rather tells an anecdote or a story in a chronological fashion, does
not meet these general requirements. This problem is comparable to students writing
a good response an an essay exam to some question, that is, however, not answering
the question posed by the exam.
An issue when marking essays that do not meet the general requirements rose
when applying the arguments which have argued in favour of analysing each HOC
problem equally (cf. above). An essay that, for example, does not meet the
requirements by being chronological, but is otherwise free of any HOC issues, would
score a 9.9 in the current rating scale (losing 0.1 for its lack of a Focus). The same
essay, that was in fact focussed, might score less due to having several Link to Focus
issues, for example. However, the raters felt that this rating solution was not
accurately representing reader engagement: despite the issues the second essay
would have, it might be more engaging due to it having an explicit Focus, which the
first essay lacked. Additionally, it would simply not be fair to grant the first essay a
higher mark: if there is no Focus, there can be no “Off-Focus” or “Link to Focus
Missing” issues, thus reducing the complexity of this essay compared to the second
AR Cycle 3 - Student Essays Thijs Gillioen
!36
one. On top of this, the raters felt students who did not fulfil the minimum
instructions for an assignment, thus not practising what they were supposed to be
practising, should be penalised more strictly for not living up to their writer’s
responsibilities.
As such, a new rule was introduced into the rating scheme to reflect these
concerns. Essays that do not meet the assignment requirements (see Appendix 2, p.
117) for any given assignment would not be marked starting at 10 (out of 20), but
rather at 8.5. This score was settled upon both as a realistic representation of the
annoyance caused for a reader by a text that does not follow all instructions, and as a
suitable reflection of the lack of proper practise. In addition, this prevents students
from getting additional marks for good style or developing good ideas, which
underscores the point made in Haas’ framework about good writing being essential to
developing a good text.
3.3 Analysing Student Essays with the Rating Scale
Having successfully developed and agreed upon a rating scale corresponding with
Haas’ framework, the student essays gathered as data for this dissertation were
analysed for L-LOC, LOC and HOC issues. Two steps were taken to help ensure
objectivity and to avoid personal preferences and potential issues of bias. Firstly, on
each essay, the name of the student was crossed out. Even though the identities of the
authors were known to two of the raters, a sufficient amount of time had passed for
this to arguably no longer present any bias issues. Secondly, to further avoid any
personal bias, and to eliminate any remaining element of personal preference when
marking essays, each essay, in both its pre-mentoring and post-mentoring form, was
marked by the same three raters who developed the rating scale discussed above.
AR Cycle 3 - Student Essays Thijs Gillioen
!37
Although, as noted by Saliu, “it is often argued that qualitative assessment is
subjective, imprecise and biased” (2005: 271), the varying backgrounds of the raters,
their differing levels of prior marking experience and their familiarity with the
framework and its marking system ensure a sufficiently diverse team of raters to
achieve a balance of viewpoints (Park 2015).
Because the current research focuses on finding changes in text quality
between pre- and post-mentoring essays, rather than objectifying the reliability of the
rating scale, the inter-rater reliability test conducted between the three raters was
concerned with establishing if each rater identified the same L-LOC, LOC and HOC
issues in each text. Therefore, the raters each independently analysed the text for
places of disengagement, and how these could be identified using Haas’ framework.
After this analysis, the raters compared notes and found they had, in nearly all cases,
identified the same issues, thus demonstrating a high level of inter-rater reliability (Li
Gwet 2014). Following this, the rating scale was used to mutually agree upon a mark
that was representative of text quality. Since each rater had identified the same
(number of) issues, agreeing upon a final score for each essay did not present any
significant discussions. This system allowed for a maximum focus on text issues
during the rating process. The result of this process, the analysed essays, are available
for consultation in Appendix 3. The results of this analysis are presented in the next
section.
3.4 Overview of Results
Table 02 provides an overview of the results of the data analysis of the student essays,
by listing the number of L-LOC, LOC and HOC problems encountered in each pre-
and post-mentoring essay, along with the mark agreed upon for each essay by the
AR Cycle 3 - Student Essays Thijs Gillioen
!38
raters. Table 03 then elaborates on HOC issues specifically, detailing which and how
many HOC problems were encountered in each essay. A more extensive discussion of
the data provided in both tables is provided in the next section.
AR Cycle 3 - Student Essays Thijs Gillioen
!39
AR Cycle 3 - Student Essays Thijs Gillioen
Table 02. Overview of Marks, L-LOCs/LOCs/HOCs in Pre- and Post-Mentoring Essays
(*) Indicates an essay did not meet the general requirements of the assignment (for example, by having a chronology-based text rather than a focus-based text - cf. Appendix 2). These essays could receive a maximum score of 8.5. (**) Scores higher than 10 are to be attributed to style, development of ideas and other forms of ‘advanced engagement’ (cf. Appendix 2 and Table 01). Since mentors focus on issues of structure, scores higher than 10 are not reflective of the efficacy of mentoring. They have, therefore, been represented as "10(+)". (***) These averages were rounded up or down to the full decimal, to reflect the fact that “half a(n) L-LOC/LOC/HOC” does not exist in Haas’ framework.
!40
AR Cycle 3 - Student Essays Thijs Gillioen
Note: the category “Complex Issues” was not included in the marking scheme, represented above in Table 02. This will be explained below; here, this explains why Table 02 had a total of 102 (pre) and 57 (post) HOC issues, versus Table 03’s 110 (pre) and 63 (post) HOC issues.
Table 03. Overview of HOCs in Each Essay, per HOC Category
!41
3.5 Discussion of Results
3.5.1 General Remarks
This analysis confirms the initial hypothesis brought forth by this paper (that
mentoring can guide students towards improving their written products), and the
existing literature on writing mentoring (which does not usually assess text changes,
but does indicate mentoring can help students): all analysed student essays displayed
improvements in their final version. On average, a student essay saw an improvement
of 27.29%, which is far enough above the statistic threshold of 5% to be considered
statistically relevant. Individual improvement rates vary from a low of 1.03% (for an
essay that already achieved a high score in its draft form) to a high of 81.48% (for an
essay that displayed a high number of both L-LOC and HOC problems). On top of
this, the essays that did not meet the minimum requirements in their pre-mentoring
form managed to amend this after the mentoring session(with essay 11 being the one
exception), an improvement in text quality that is arguably larger than indicated by
the numbers present. Finally, it is interesting to note that essays which do not have
any HOC problems (such as essay 15) manage to maintain this in their post-
mentoring form. Although seemingly evident, this indicates mentors are equally
capable of spotting a good essay (and thus withholding any non-directive support
that might hamper the essay) as they are of spotting issues in essays. Having given
this general overview, the next paragraphs will present a general overview of each
marking category and their relevant numbers.
Lower Order Cognition issues, in particular, are much less present in post-
mentoring essays. Most of these essays are completely devoid of LOC problems, while
those that are not are limited to a single LOC issue. For the former group, this means
a decrease of LOC problems by 100%; for the latter, it means an improvement of
AR Cycle 3 - Student Essays Thijs Gillioen
!42
66.67%. This high improvement rate is explainable by the nature of LOCs: these
Lower Order Cognitions, as defined above, do not require significant work to find or
fix. Therefore, it is relatively ‘easy’ for a mentor to guide their students towards fixing
these issues. As such, it is relatively unsurprising that this category sees the highest
improvement rates.
The category of Lower-Lower Order Cognition problems sees the second-
highest improvement rate, at 42.86%. Mentors, as guides along the path of a writer’s
development, do not serve as copywriters, editors or spelling checkers (Harbord
2003): they generally only point out that ‘language issues are present’, without
further specification. This explains why these spelling- and grammar issues continue
to appear in students’ final texts. Furthermore, as students rewrite (sections of) their
essays based on their session with the mentor, new issues may arise even if old ones
were fixed. This is the case, in, for example, essay 02, which has one additional L-
LOC problem in the final version of the essay: this L-LOC problem appears in a
section that was rewritten after the student’s session with a mentor.
Finally, the category of HOCs sees a rate of improvement that is equal to that
of the L-LOC category, with an average of 42.86% less HOCs per essay. This
demonstrates two points: firstly, how these Higher Order Cognitions are, cognitively,
the hardest ones to fix, even with the guiding help of a mentor. Therefore, the fact
that HOCs remain somewhat frequent in most of these essays is to be explained by
their cognitively hard nature, which underlines the importance of Kellogg’s demand
for “repeated opportunities for practise and feedback” (Kellogg & Raulerson III 2007:
237 [emphasis added]): it would be too simplistic to think any mentor could help a
writer resolve all issues in a cognitive writing process during one short, 30-minute
session. Research in this field has indicated as much: Harbord, for example, notes
AR Cycle 3 - Student Essays Thijs Gillioen
!43
such papers “are not written in a single sitting” (2003: 4), further adding that “the
mentor’s role is not to edit, but to […] help the student master conventions and
structure” (Harbord 2003: 4). Similarly, Irvin noted three tutorial sessions (in the
context of this paper, mentoring sessions) to be a “significant threshold” (2014: 2),
whereas these students were only given one. This, combined with Kellogg’s claim that
the “development of writing skills […] requires decades of learning” (2008: 20),
makes the improvement rate of 27.29% rather significant.
One other important note regarding the improvement rate of HOC issues is
that this ‘apparent’ number of HOCs that are retained in the post-mentoring version
of students’ essays is actually not entirely representative of the effects of a mentoring
session. Firstly, mentors only ‘guide’ students, non-directively: it is up to the students
to follow the mentor’s guidance and thus to actually eliminate these issues in their
essays. As such, one has to be wary of implying a too direct, causal link between the
mentoring session and the amount of HOCs fixed. Secondly, the ‘apparent’ number of
HOC issues that were solved is actually lower than the ‘actual’ number when one
takes in to consideration the effect of rewriting (sections of) an essay, as noted above
when discussing L-LOC issues. A student might rewrite one or more sentences or
sections of their essay as a result of the mentoring session, thus solving one problem,
but inadvertently creating a new HOC issue in the process. For example, essay twelve
had thirteen HOC problems in its pre-mentoring form; Table 02 indicates it had four
remaining issues in its post-mentoring form. However, in reality, none of the initial
thirteen issues remained: the four ‘apparently retained’ HOC issues found in the post-
mentoring essays were all results of the rewriting process. This is discussed in more
detail in the next section, which deals with a number of specific data trends.
AR Cycle 3 - Student Essays Thijs Gillioen
!44
3.5.2 Specific Areas of Discussion: Data Trends
3.5.2.1 ‘Apparent’ Number of HOCs Retained vs. ‘Actual’ Number of HOCs retained
As noted above, there is a difference between the ‘apparent’ number of HOC
problems that remain in the text after a mentoring session, and the ‘actual’ number.
When looking at Tables 02 and 03, one might assume that, in essay 3, for example,
which had twelve HOC problems before the mentoring session, and five HOC
problems in the post-mentoring text, five HOC issues were either not spotted by the
mentor, or not solved by the student after the mentoring session. However, this does
not take into account the reality of the writing process: to fix one issue, a writer often
has to rewrite large sections of a their text, or shift around existing parts of a draft
into new positions. During this process, whilst previous HOC issues might have been
fixed, the writer may have inadvertently created new issues. As such, the ‘actual’
number of retained HOC issues is lower than what the previous two tables indicate:
these tables assume that no such rewriting process took place. To demonstrate the
difference between the ‘apparent’ change (which includes HOC problems that were
created after the mentoring session) and the ‘actual’ change (which does not include
these post-mentoring HOC problems), Table 04 provides an overview of the HOC
issues in each essay (in both pre- and post-mentoring state). HOC issues that did not
change at all (textually) in between the different versions of an essay were underlined
and italicised. These are the ‘actual’ retained HOC issues: the issues that were present
in the pre-mentoring text, and remained in the post-mentoring text.
AR Cycle 3 - Student Essays Thijs Gillioen
!45
AR Cycle 3 - Student Essays Thijs Gillioen
Overview of “Higher Order Cognition” Issues in Student Essays: HOC Issues that remain, unchanged, represented in italics and underlined.
Essay Pre-Mentoring HOC Issue(s) (with paragraph n°)
Post-Mentoring HOC Issue(s) (with paragraph n°)
Number of Unchanged
HOC Problems
‘Actual’ HOCs retained/solved (in %)
(vs. ‘Apparent’ %)
1 Conciseness Problem (P1) Relevance Problem (P1) Focus too Wide (P1) Conciseness Problem (P2) Relevancy Problem (P2) Link to Focus Missing (P2) Construction Problem (P2/3) Link to Focus Missing (P3) Link to Focus Missing (P5) Transition Problem (P5/6) Link to Focus Missing (P6) Balance Problem (P7)
Gap in Information (P1) Focus too Wide (P1) Link to Focus Missing (P3(*)) Link to Focus Missing (P4) Link to Focus Missing (P5)
(*) Corresponds to P4 in pre-mentoring Essay.
216.67% / 83.33%
(41.67% / 58.33%)
2 Balance (P1) Conciseness (P1) Broken Promise (P1) Double Focus (P1)
Balance (P1) Conciseness (P1) Double Focus (P1) 3
75% / 25%
(75% / 25%)
3 Focus too Wide (P1) Link to Focus Missing (P2) Link to Focus Missing (P3) Shift in Focus (P3) Shift in Focus (P3) Shift in Focus (P4) Conciseness Problem (P4) Off-Focus (P4) Shift in Focus (P4) Balance Problem (P5) Self-Containment Problem (P5)
Focus too Wide (P1) Shift in Focus (P2) Link to Focus Missing (P2) Shift in Focus (P5) Consistency Issue (P5)
1 8.32% / 91.67%
(45.46% / 54.54%)
4 Implicit Focus (P1 - GR (*)) Transition Problem (P3/4) Shift in Focus (P5) Relevancy Issue (P6) (*)General Responsibility Issue
Shift in Focus (P4) Relevancy Issue (P5)
250% / 50%
(50% / 50%)
5 Order Problem (P1) Precision Problem (P2) Precision Problem (P2) Order Problem (P5)
Conciseness Problem (P2) Balance (P1-3) Shift in Focus (P4) Order Problem (P5)
125% / 75%
(100% / -)
6 Focus too Wide (P1) Relevancy Problem (P4) Consistency Problem (P4)
Gap in Logic (P2) Link to Focus Missing (P2) Order Problem (P4)
0- / 100%
(100% / -)
7 Self-Containment Problem (P1) Broken Promise (P1) Focus too Wide (P1) Consistency Issue (P2) Link to Focus Missing (P2) Order Problem (P2) Gap in Information (P2) Gap in Information (P3) Link to Focus Missing (P3) Gap in Information (P3) Relevancy Problem (P4) Transition Problem (P4) Shift in Focus (P4) Balance Problem (P5)
Focus too Wide (P1) Gap in Information (P2) Relevancy Problem (P3) Link to Focus Missing (P3) Gap in Information (P3) Transition Problem (P4) Relevancy Problem (P4) Gap in Information (P5) Balance (P6)
8 57.14% / 42.86%
(64.29% / 35.71%)
Table 04. ‘Apparent’ Retained HOC Issues vs. ‘Actual’ Retained HOC Issues
!46
AR Cycle 3 - Student Essays Thijs Gillioen
Essay Pre-Mentoring HOC Issue(s) (with paragraph n°)
Post-Mentoring HOC Issue(s) (with paragraph n°)
Number of Unchanged
HOC Problems
‘Actual’ HOCs retained/solved (in %)
(vs. ‘Apparent’ %)
8 Broken Promise (P1) Focus too Wide (P1) Coherence Problem (P3) Consistency Problem (P4) Balance (P5)
Broken Promise (P1) Focus too Wide (P1) Consistency Problem (P6) 1
20% / 80%
(60% / 40%)
9 Focus too Wide (P1) Shift in Focus (P3) Order Problem (P4)
Relevancy Problem (P3)0
- / 100%
(33.33% / 66.67%)
10 Order (P1) Conciseness Problem (P1) Balance (P1) Consistency Problem (P3) Relevancy Problem (P3) Gap in Information (P5) Relevancy Problem (P6)
Conciseness Problem (P1) Consistency Problem (P2 (*)) Relevancy Problem (P3)
(*) Corresponds to P3 in pre-mentoring Essay.
114. 86% / 85.14%
(42.86% / 57.14%)
11 Focus too Wide (P1) Relevancy Problem (P2) Shift in Focus (P3) Link to Focus Missing (P3) Shift in Focus (P4) Off-Focus (P5) Shift in Focus (P5) Link to Focus Missing (P5) Shift in Focus (P6)
Link to Focus Missing (P2) Shift in Focus (P3) Link to Focus Missing (P3) Conciseness Problem (P3) Conciseness Problem (P4) Link to Focus Missing (P5) Conciseness Problem (P5) Partly Chronological (P1-5(*))
(*) General Responsibility Issue
3 33.33% / 66.67%
(88.89% / 11.11%)
12 Consistency Problem (P1) Gap in Information (P2) Given / New Problem (P2) Gap in Information (P2) Gap in Information (P2) Order Problem (P2) Gap in Information (P2) Gap in Information (P3) Link to Focus Missing (P3) Link to Focus Missing (P3) Balance (P1-4) Order Problem (P4) Conciseness Problem (P4)
Given / New Problem (P1) Consistency Problem (P1) Given / New Problem (P2) Relevancy Problem (P2)
0- / 100%
(30.77% / 69.23%)
13 Broken Promise (Title) Broken Promise (P1) Gap in Information (P1) Gap in Information (P4) Relevancy Problem (P5)
Broken Promise (Title) Broken Promise (P1) Gap in Information (P4) Relevancy (P5)
480% / 20%
(80% / 20%)
14 Broken Promise (Title) Focus Unclear (P1) Balance (P1-4) Focus too Wide (P2) Shift in Focus (P3) Consistency Problem (P3) Shift in Focus (P3) Shift in Focus (P4) Consistency (P4)
Relevancy Issue (P2) Link to Focus Missing (P2) Shift in Focus (P5)
111. 11% / 88.89%
(33.33% / 66.67%)
15 - - - -
AVG 26. 65% / 67.24%
Table 04. ‘Apparent’ Retained HOC Issues vs. ‘Actual’ Retained HOC Issues (second part)
!47
Table 04 demonstrates the difference between ‘apparent’ HOCs retained and
‘actual’ HOCs retained, as outlined above. By isolating the HOC issues that remain
unchanged in between the pre- and post-mentoring version of an essay, it becomes
clear that the ‘apparent’ HOC improvement rate of 42.86%, as demonstrated in Table
02, is too pessimistic. The post-mentoring versions of the student essays analysed
here actually saw a decrease in HOC issues by 67.24%, or roughly two thirds, when
compared to their pre-mentoring state. This improvement rate then decreases once
new HOC issues are created during the rewriting process.
Of particular note, also, is the high individual improvement rate demonstrated
by the majority of these essays. Three students (essays six, nine and twelve) were able
to eliminate all HOC issues in their essay, after the session with their mentor. Five
more essays only retained a single HOC issue following the mentoring session,
displaying improvement rates of 80% and beyond. This clearly reinforces the
conclusion outlined above, that mentoring is in fact an effective way to help students
produce better texts. Furthermore, most post-mentoring essays also display a much
smaller number of (or, in some cases, no) new HOC problems when compared to
their pre-mentoring version, even when said essays underwent major rewrites
following the mentoring session. This could indicate the mentoring session not only
had a short-term impact (on the text itself), but also, possibly, on the student’s
long(er)-term development as a writer, thus helping them avoid making (some) HOC
mistakes in their (future) writing.
On the other hand, three essays (essays two, seven and thirteen) maintained
most of their original HOC issues, even if some of the original HOC issues were
solved. There are a number of possible explanations for this. Firstly, in the case of
essay two, the HOC issues present are relatively minor. Because the essay, in general,
AR Cycle 3 - Student Essays Thijs Gillioen
!48
manages to maintain a relatively high level (both in terms of language and writing),
the raters were able to delve deeper into the text than was the case in most other
essays. This team of three raters may have been able to spot issues that the mentor
(in the time-constrained setting of a single mentoring session) did not spot.
Alternately, it is possible the mentor did spot these issues, but the student did not
believe them to be significant enough to warrant fixing: as part of their writing
education, students are introduced to the idea that it is the writer who makes the final
decision on whether or not to take up the advice given by feedback.
Secondly, in the case of essay seven and thirteen, the prevalence of other issues
may offer an explanation as to why several HOC issues remained unchanged. Both
essays contain high numbers of both L-LOC issues and LOC problems: essay seven
exceeds the second L-LOC “buffer” (defined in section 3.2.2.2, p. 30), indicating a
severe level of reader disengagement, in addition to a high number of LOC problems.
Similarly, essay thirteen passes the first L-LOC “buffer”, thus also indicating an
advanced level of reader disengagement, in addition to containing an average amount
of LOC problems. The combination of these elements, i.e. a high amount of both L-
LOCs and LOCs, results in a high level of disengagement for a reader. Because of this,
such essays force a mentor to spend a significant amount of time (of the mentoring
session) working with the student to discuss these issues, thus losing time that could
have been spent on the discussion of the present HOCs. Furthermore, the high level
of disengagement makes it hard for mentors to focus on such Higher Order Cognition
issues, when an essay is already filled with Lower Order Cognition problems.
Because of these reasons, it is not unfair to pose that the authors of essays
seven and thirteen could have benefitted significantly from an additional mentoring
session after having rewritten their draft. Both essays have significantly reduced L-
AR Cycle 3 - Student Essays Thijs Gillioen
!49
LOC and LOC issues in their post-mentoring form: this would have allowed for both
the mentor and the student to focus on the HOC issues that were, at this stage, still
present in the essay. This is reinforced by Irvin’s argument about three mentoring
sessions being a “significant threshold” (2014: 2). The same is true for all other
essays: the remaining (and new) HOC issues could, potentially, have been solved by
an additional mentoring session.
However, it is interesting to take note of essay eight in this context. This essay
features an even higher number of L-LOC problems than the previous essays, in
addition to an average amount of LOC problems, thus, theoretically, causing an
equally high, if not higher, level of reader disengagement. Despite this, essay 8 only
retained a single HOC issue in its post-mentoring form, a decrease of 80% compared
to its pre-mentoring form. Therefore, one can question what sets this essay apart
from the essays discussed above that allowed the mentor (and the student) to
maintain their engagement to a point where HOC problems could, in fact, be
discussed. It is possible subjective elements such as personal interest in the topic, or
appreciation of the writing style (for example, the use of humour) play a role in
dampening the disengagement caused by L-LOC problems and LOC issues.
Alternately, it may be the case that, once a certain high number of L-LOC issues is
reached, these spelling and grammar issues actually become less relevant for a
reader/mentor, who may assume the writer simply has an insufficient command of
the language, rather than attributing it to sloppiness on the writer’s part. This would
allow for the mentor to set aside these issues temporarily, re-enabling them to look
for higher-level cognition issues. This specific topic warrants further study, which on
its own could be the focus of an entirely separate research paper. For now, this
dissertation will offer the hypotheses outlined above as an explanation for this
AR Cycle 3 - Student Essays Thijs Gillioen
!50
phenomenon, and move on to other data trends observed during the analysis of the
corpus.
3.5.2.2 Essays with ‘Complex Issues’
A second trend that emerged during the data analysis was the appearance of issues
that could not be easily labelled with Haas’ framework. The difficulty of categorising
these problems lies in the fact that they are not explainable by singularly pointing to
one HOC problem defined in the framework. Instead, they often appear to be
combinations of several ‘regular’ HOC problems that manifest in unison, thus
creating an issue that is familiar, but not entirely the same as those originally defined
by the framework. These issues were not initially covered by the framework: as such,
the three raters labelled them “complex issues” (De Soete 2016; Haas 2016) . An 8
overview of such issues appearing in this dissertation’s corpus is listed in Table 05,
below.
As Table 05 demonstrates, these complex issues are generally caused by
Higher Order Cognition problems, and lead to other HOC issues. To illustrate this
effect, an excerpt from essay 07 is provided below. This specific excerpt is the first
To reflect that the category ‘complex issues’ is the result of a discussion between the three 8
raters (as opposed to being one individual’s terminology), this dissertation cites both Haas (2016) and De Soete (2016) for this term, which is used in each of the raters’ essays.
AR Cycle 3 - Student Essays Thijs Gillioen
Table 05. Overview of ‘Complex Issues’ in Each Essay, per Essay
!51
half of the second paragraph of the essay, and demonstrates the complex issue that is
created by an underlying “Gap in Information” (Haas 2016: 45), which leads to a
“Consistency” problem (Haas 2016: 51). The relevant section has been underlined for
clarity; any L-LOC issues were italicised.
First of all, I eat my cookies and cakes and pies by hand. With a biscuit
as greasy as that, you won’t be able to leave the room without a stain on
your shirt. Above that you will leave a trail of grease on the floor,
because it keeps dripping of that fancy dessert.
(Appendix 3, Essay 07, p. 157 [emphasis added])
In this specific case, there is a Gap in Information between the ‘eating by hand’ in line
one, and the causing of ‘a stain on your shirt’. Arguably, the Gap in Information is not
incomprehensibly large: a reader can imagine this author wanted to indicate greasy
cookies leave one’s hands greasy, which usually results in stained shirts because
people did not wash their hands after eating the greasy cookie and they might then
touch their shirt, because of which the grease of the cookie ends up making stains on
one’s shirt. However, part of a well-written text, according to Haas’ framework, is that
the writer has “fill[ed] in all the necessary information or logical connections in a
text” (2016: 45). Additionally, such seemingly small issues can present serious
consequences if appearing in academic writing, underlining the importance of paying
attention to even the smallest HOC issues at an early stage. As such, since the
connection between ‘eating the cookie by hand’ and ‘a stain on a shirt’ is not made in
the essay itself, this section qualifies as a Gap in Information.
It is insufficient, however, to label this specific problem a Gap in Information
and to leave it as such. Because of the gap, the author has also (unconsciously)
created a Consistency problem in their own text. The essay’s focus is “disadvantages
AR Cycle 3 - Student Essays Thijs Gillioen
!52
of greasy desserts” (Appendix 3, Essay 07, p. 157). The Gap in Information results in
an inconsistent text because it creates a shift: is the disadvantage the greasy hands, or
the stained shirt? Again, it is easy for the reader to assume both of these are
disadvantages, but this is another example of something that should not have been
left to the reader’s interpretation in the first place. As such, this specific issue can be
labelled as a Consistency Problem (since ‘stain on the hands’ is inconsistent with
‘stain on the shirt’), but it has a Gap in Information at its base.
The nature of these complex issues, is, of course, tied to the inherently
complex nature of language itself and (academic) writing in particular. At the same
time, it demonstrates the difficulty of developing a straightforward rating scale that
can successfully delineate every single issue into a neat category. On top of this, it can
be useful to distinguish these complex issues from ‘regular’ HOC problems, for both
mentors and students. Given the high retention level of these complex issues, it
appears further instruction for mentors on these specific issues is warranted.
Similarly, making students aware of the fact that HOC issues can have an impact on
each other could increase their own critical reading when reviewing their writing.
3.5.2.3 Specific HOC Trends
Finally, this section will discuss a number of trends related to specific HOC
categories. Therefore, Table 06 offers an overview of the frequency of each HOC
category in both pre- and post-mentoring essays. Keeping in mind the point
discussed in section 3.5.2.1, about ‘apparent’ vs. ‘actual’ retained HOCs, this table
makes the same distinction for post-mentoring essays.
AR Cycle 3 - Student Essays Thijs Gillioen
!53
The HOC categories related to “Focus” are clearly the most frequent source of
textual problems, in both pre- and post mentoring texts. On its own, this is not
entirely surprising: “Focus” is defined by Haas as being one of the most difficult
“building blocks” to master (2016: 55), and this specific assignment was the first time
AR Cycle 3 - Student Essays Thijs Gillioen
Table 06. Number of HOC Issues, per HOC CategoryNumber of HOCs, per HOC Category, Pre-Mentoring vs. Post-Mentoring
HOC Category # Pre-Mentoring # Post-Mentoring (retained/new)
Delta retained (# and %)
Focus too Wide 8 4 / 0 -4 (-50%)
Focus Unclear 1 0 / 0 -1 (-100%)
Double Focus 1 1 / 0 -0 (-)
Focus Shift 14 3 / 3 -11 (-78.57%)
Off-Focus 2 1 / 0 -1 (-50%)
Link to Focus Missing 12 4 / 6 -8 (-66.67%)
Relevancy Problem 9 5 / 3 -4 (-44.44%)
Order Problem 7 1 /1 -6 (-85.71%)
Conciseness Problem 6 1 / 2 -5 (-83.33%)
Coherence Problem 1 0 / 0 -1 (-100%)
Consistency Problem 6 2 / 2 -4 (-66.67%)
Balance Problem 8 2 / 1 -6 (-75%)
Given / New Problem 1 2 / 0 +1 (+100%)
Transition Problem 3 1 / 0 -2 (-66.67%)
Construction Problem 1 0 / 0 -1 (-100%)
Gap in Information 13 4 / 1 -9 (-69.23%)
Gap in Logic 0 0 / 1 -
Broken Promise 6 2 / 1 -4 (-66.67%)
Self-Containment Problem 1 0 / 0 -1 (-100%)
Precision Problem 2 0 / 0 -2 (-100%)
Complex Issues 8 6 / 1 -2 (-25%)
!54
students had to grapple with it; thus, some levels of problems can be attributed to the
nature of this specific HOC and the circumstances of the assignment.
Despite this, however, the “Focus too Wide” problem in specific is one of the
most persistent issues, even in post-mentoring texts, seeing the second-lowest rate of
improvement following a mentoring session. This can indicate two things: either
mentors are (too) conservative in their mentoring approach to essays whose focus is
too wide, or mentors do mention this issue, but students do not (sufficiently) narrow
their focus following their conversation with the mentor. Either way, it is clear this is
an area where improvements to the programme’s conduct can be made, for example
by increasing mentors’ training in this specific area.
Similarly, the categories “Link to Focus Missing” and “Focus Shift” are very
frequent in pre-mentoring essays, and continue to cause reader disengagement in
post-mentoring essays. Unlike the “Focus too Wide” trend, though, these issues do
see relatively high improvement rates following the mentoring session (66.67% and
78.57%, respectively), indicating that some of the missing links were corrected.
However, this amelioration is impacted negatively by the creation of new problems in
these categories during the rewriting process. This, as noted above, underlines the
importance of the call for repeated conversations with a mentor. In addition, one can
also question if this is a proper place for a more flexible mentoring relationship
between the mentor and the student, as argued for by Harbord (2003) and Crasborn
et al. (2011): by non-directively pointing to and resolving the problem, then taking a
more directive approach (for example, by providing general examples on how this
issue could be eliminated), the creation of these new HOC issues might be prevented,
whilst also still primarily putting the focus on the students’ development as a writer,
rather than the improvement of one specific essay.
AR Cycle 3 - Student Essays Thijs Gillioen
!55
In the non-focus related (or, given the interconnectedness of these categories,
less-closely-to-focus-related) categories, “Relevancy” and “Gap in Information” are
the most frequently-appearing problems. This serves to demonstrate how
interconnected some of these categories are: it is often ‘because of’ a Gap in
Information, that a section becomes less relevant to the essay’s focus. Both categories
have relatively high improvement rates, though, indicating mentors are able to deal
with these issues when they occur. However, new relevancy issues continue to occur
in the post-mentoring essay, suggesting this is one area mentors may need to adopt a
more directive attitude, or assume the role of a more directive peer tutor (as opposed
to a non-directive peer mentor (Topping & Ehly 1998)), in order to ensure students
can prevent these problems from appearing in (sections of) their newly (re)written
text.
Finally, one must also look at the categories that have seen the highest rates of
improvement. Following the mentoring session, issues related to an “Unclear Focus”,
“Order”, “Coherence”, “Balance”, “Construction”, “Self-Containment” and “Precision”
were entirely or mostly resolved. This can be explained, partly, by the fact that these
are areas in which mentors have been extensively trained, and partly by the nature of
these issues: they are subjectively noted to be some of the “easier” HOC problems in
the sense that they are, cognitively, somewhat easier to distinguish than issues that
are, for example, related to Focus. Nevertheless, the high improvement rates in these
categories are an example of the efficacy of writing mentoring, demonstrating that
mentors are capable of guiding students to resolve a large number of any given text’s
issues.
AR Cycle 3 - Student Essays Thijs Gillioen
!56
3.6 Recommendations for Programme Improvements
Having discussed the results of the analysis of students’ pre- and post-mentoring
texts, the following section aims to suggest how these data can be used to improve the
mentoring programme’s practices, and its efficacy. Several of these suggestions have
been touched upon in the discussion of the data analysis; they will be expanded upon
here.
The possibility of students attending several mentoring sessions for one
specific assignment has already been mentioned during the discussion of the data
results, and is backed up by research conducted by Harbord (2003) and Irvin (2014),
and further reinforced by Kellogg’s demand for repeated opportunities for feedback
(Kellogg & Raulerson III 2007). From a subjective, experience-based standpoint, the
author can attest that working with the same student, during various (writing) stages
of their assignment, has proven to have a significant effect on both the students’
development as a writer, and especially on their written product. In the programme’s
current setting, mentoring sessions are slotted to have a maximum duration of half
an hour, a time period that is relatively narrow if the mentor is faced with an essay
that has a high amount of HOC problems (such as essays seven, eight and eleven).
Mentors are currently free to request for students to return for a follow-up session,
but this is rare in practice, partly due to time constraints on both students and on
mentors (who, in the current setting, are all contributing to the programme on a
volunteer basis), and partly due to the unfamiliarity of Belgian students with the
concept of peer mentoring, which is still proving to be somewhat of a barrier. Despite
this, the data discussed above clearly show mentors are capable of guiding students
towards significant textual improvements, an effect that can only increase when
multiple sessions can take place. Therefore, although this recommendation does not
AR Cycle 3 - Student Essays Thijs Gillioen
!57
go as far as to suggest attending multiple mentoring sessions be made mandatory in
the current setting, strongly encouraging mentors to ask students to return for (a)
follow-up session(s) - or, ideally, recommending them to reach “the three [session]
threshold” (Irvin 2014: 5) - is one path to increasing the efficacy of the programme in
its current form.
The second recommendation that can be made for programme improvements
relates to mentor instruction. There are a number of specific areas where further or
more intensive instruction for mentors could increase the programme’s efficacy. Of
the retained HOC problems in the essays analysed above, 44.44% relate to focus
(Focus too Wide, Link to Focus Missing, Double Focus, Focus Shift). Focus too Wide,
in particular, is an issue retained in 50% of the post-mentoring essays. Similarly,
complex issues are an area of low improvement in essays, with 75% of the identified
issues remaining in the post-mentoring essay. This can be explained partly by the
inherent complexity of these issues, and partly by the fact that these issues were only
identified as part of the inter-rater reliability process of this paper. As new additions
to Haas’ continually-evolving framework, mentors will need to receive additional
training in order to ensure they can identify and remedy such issues in future
students’ texts.
One way to familiarise mentors with HOC problems and practise identifying
these issues is by involving mentors in the marking process. This could have two
advantages for those involved in the programme: firstly, the teacher would receive
valuable (and, arguably, experienced) assistance in providing feedback on the large
number of essays (exceeding 200+ essays per assignment) that have to be marked for
the ETTVI course, allowing them more time to engage with students (and their
writing) personally.
AR Cycle 3 - Student Essays Thijs Gillioen
!58
Although unusual in the current (Belgian) setting, the inter-rater reliability
test at the onset of this chapter has demonstrated Haas’ framework might be applied
by non-experts with relatively little difficulty; furthermore, research has shown peer
assessment can be as effective as staff assessment (e.g. Zundert et al 2010; Iraji et al.
2016), especially when working with well-defined criteria or a specific rubric (in casu,
Haas’ framework) (Falchikov & Goldfinch 2000; Panadero et al. 2013; Jones &
Wheadon 2015). On top of this, mentors are already familiar with the framework they
would be using, having learned it in the ETTV classes and used it as mentors.
Familiarity with a framework enhances the validity of using it as a tool for assessment
(Falchikov & Goldfinch 2000: 315).
Secondly, being involved in marking can help not only the teacher, but
mentors as well. Their involvement could lead to a deeper understanding of HOC
issues “through meta-processes such as reflecting on and justifying what they have
done” (Liu & Carless 2006: 289). As such, it could provide considerable amounts of
real-life practise for mentors, helping them identify issues (faster). Aside from short-
term practice, research has also indicated peer assessors can gain other valuable
skills, such as problem-solving skills (Çevik 2015: 256), and can “[increase] student
learning and achievement” (Vista et al. 2015: 13), all of which could prove
advantageous as long-term improvements for mentors.
Thus, to address the issue of retained problems in the text, this paper offers
the recommendation that mentors be given specific instruction on the issues that are
retained most often, and that they be involved in the marking process. This offers
both immediate advantages for the teacher and mentors alike, additionally resulting
in long-term benefits such as mentors gaining valuable skills that can aid them in
their goal of guiding students to become better writers.
AR Cycle 3 - Student Essays Thijs Gillioen
!59
4 Perspective 2: Analysis of Student Reflections
Having analysed and discussed the textual improvements to students’ writing that
can be attributed to a mentoring session, and having made suggestions for improving
the programme’s efficacy further, the current section turns to the second perspective
on the research question outlined at the beginning of this dissertation.
Ultimately the goal of writing mentoring is not to improve texts, but rather to
assist students during their writing process and during their development as writers.
Therefore, an evaluation of programme efficacy can not solely be done from an
analysis of written products, but must also include the perspective of students as
writers-in-development, and, more specifically, the role and effect of writing
mentoring on this process. Getting the student perspective on this is important since
the writer development course does not solely focus on producing good text: it
focusses on turning an audience of students who have various writing backgrounds
into (reasonably) good (academic) writers. As such, understanding how students
experience the mentor process is an important component in establishing its efficacy,
and how said process can be improved in the future.
4.1 Data Collection: Student Reflective Writing
The reflections that will be used to study students’ view on writing mentoring were
written as part of their third writing assignment. In addition to being required to
write a formal essay, students were also asked to submit 1,000 words of reflective
writing. The assignment for this was to “write at least 1,000 words of Reflective
Scribbling. Reflecting on your writing decisions and on [if/how] giving/receiving
feedback helps you as a writer and helps other writers” (S3 assignment, Appendix 1,
AR Cycle 3 - Student Reflections Thijs Gillioen
!60
p. 112). This “reflective scribbling” was analysed for content that is relevant to
mentoring.
As with the students’ writing assignment, written permission to use their
reflective writing for academic research was obtained from all students prior to the
analysis of the texts for the purposes of this dissertation. To avoid any privacy
concerns, names and pronouns referring to either mentors or students were removed
and replaced with neutral references and pronouns; in addition, only the pieces of
reflective writing that are directly quoted from are included in the appendices to this
dissertation. To further ensure the privacy of those specific writers (and to ensure
maximum relevancy of the appendices), only the immediate context of the quote is
provided (usually the full paragraph from which a quote has been extracted). The
student comments presented in this dissertation were not otherwise altered unless
explicitly indicated. Before turning to the results of this analysis, the following section
will briefly discuss the analysis methodology that was employed to find out how
students perceived the programme.
4.2 Methodology: Content Analysis
A total of 162 pieces of reflective writing were submitted by students, and were
subsequently studied for the purposes of this dissertation. This reflective writing was
categorised in a number of ‘rounds’, based on lexical analysis. This categorisation
process is represented schematically in Table 07.
AR Cycle 3 - Student Reflections Thijs Gillioen
!61
In a first selection round, texts that offered some comment related to writing
mentoring and/or writer development were separated from those that did not; this
latter group was not studied further. Comments that were considered related to
writing mentoring and/or writer development were chosen based on keywords like -
and related to the concepts of - “(writing) mentor”, “mentoring session”, “writer
development”, “writing career” and “change in writing style”. This selection process
reduced the total number of relevant reflective essays to 119, with a total relevant
comment count of 229.
AR Cycle 3 - Student Reflections Thijs Gillioen
Reflection Categorisation Process
Original data set
Step 1: Categorisation
based on relevancy
Step 2: subcategorisation
Step 3: subdivision Step 4: further subdivision
where possible (final categories)
162 pieces of Student Reflective Writing
119 ‘relevant ‘ essays;
229 ‘relevant’ comments
“Helpful” category (213 comments)
“Practical Reasons” “Practical Reasons”
“Psychological Reasons”
“Confidence and/or Motivation Boost”
“Extra Explanation Offered and/or Helps to Understand Theory”
“Appreciation of Mentors(‘) (Style)”
"Appreciation of Approachability”
“Appreciation of Non-Directive Approach”
“Student Advice on Mentoring (Programme)”
“Student Advice on Mentoring (Programme)”
“Attending Multiple Mentoring Sessions”
“Attending Multiple Mentoring Sessions”
“Did not Feel Helped” “Did not Feel Helped”
“Writer Development”
category (16 comments)
“Writer Development”
(no further subcategorisation)
“Writer Development”
43 ‘non-relevant’ essays (not studied further)
- - -
Table 07. Overview of Reflective Writing Categorisation Process
!62
During step 2 (cf. Table 07), these data were then further analysed for
comments related to the writing mentoring programme, and students’ perceptions of
its effect on their development as a writer. From this analysis, these comments were
divided into two major subcategories, again based on lexical content. The first
subcategory contained 213 comments that were related to the short-term results of
the mentoring session, and was labelled “Helpful”: these students indicated the
mentoring session helped them in one or more ways (for example, “The session with
my mentor also gave me the right directions to find the real focus” (Appendix 4,
sample 1, p. 214) ). The second subcategory contained 16 comments that were related 9
to the long(er)-term effects of the mentoring session, and was labeled “Writer
Development”. An example of such a comment is: “After the mentoring session I am
now convinced that if I had to write another focus-based story, I would find a good
focus all by myself!” (A4S2, p. 215).
In the following step (3), the “Helpful” subcategory was further subdivided,
based on lexical indicators, into narrower categories that each reflect a central reason
why students perceived the mentoring session to be useful. This ultimately resulted
(step 4) in the following eight categories: “Practical Reasons”, “Confidence and/or
Motivation Boost”, “Extra Explanation Offered and/or Helps to Understand Theory”,
Appreciation of Approachability”, “Appreciation of Non-Directive Approach”,
“Student Advice on Mentoring (Programme)”,“Attending Multiple Mentoring
Sessions” and “Did not Feel Helped”. Each of these categories was identified on the
basis of lexical items, and are defined as follows.
The first category, “Practical Reasons”, was based on a trend that was
immediately observable: students finding short-term practical benefits (such as an
Hereafter, references to comments from the reflective scribbling will be formatted as 9
“(Appendix 4, sample [x], p. [x])”, which will be abbreviated as “(A4Sx, p. x)”.
AR Cycle 3 - Student Reflections Thijs Gillioen
!63
improved text, or getting specific pieces of advice or having received help to narrow
down one’s focus) of attending a mentoring session. Therefore, this category is
comprised of comments that relate to the immediate, practical effects of writing
mentoring on students’ texts. This was based on lexical items like, for example, “the
mentor I had was very helpful and gave a lot of advice to help me to finish and
perfect my scribing at home.” (A4S3, p. 215 [emphasis added]).
A second category that quickly became evident during the analysis in step 3
was “Psychological Reasons”. This category was defined by lexical items such as word
groups like “increased confidence” or “motivation boost” and by the use of emotive
verbs such as “comforting” and “reinforcing”. This category comprises the
psychological and/or emotional benefits to attending a mentoring session. One of
these benefits indicated by a substantial number of students, for example, is finding a
renewed motivation to write following the mentoring session. Similarly, many
comments also indicated students felt a confidence boost thanks to positive
reinforcement from the mentors on their current written product. An example of such
a comment is the following: “When I got home afterwards, I felt a lot more motivated
to start again than I did when I left my apartment in the morning.” (A4S4, p. 215).
Upon further analysis during step 4, this “Psychological Reasons” category was
narrowed down into two separate, final, categories: “Confidence and/or Motivation
Boost” and “Extra Explanation Offered and/or Helps to Understand Theory”. The
first of these, “Confidence and/or Motivation Boost”, retains those comments from
the “Psychological Reasons” category defined by the words groups and emotive verbs
defined above. The second category, “Extra Explanation Offered and/or Helps to
Understand Theory” was defined by lexical items such as “increased understanding”
and “explaining [a given HOC]”. This category also relates to the psychological (and
AR Cycle 3 - Student Reflections Thijs Gillioen
!64
cognitive) effects of a peer mentoring session, but, more specifically, is comprised of
comments from students indicating they felt more motivated not (primarily) by the
support, but because the mentor was successfully able to explain some concepts from
Haas’ theoretical framework that were previously not well-understood by the
student(s). One example of such a comment is the following: “Also, he once again
explained to me the difference between a topic and a focus and I think I’m finally on
the way of figuring out what a focus really is – or at least I think I discovered
something that day.” (A4S5, p. 216 [emphasis added]).
A fourth category defined during the analysis in step 3 was “Appreciation of
Mentors(‘) (Style). This category was mainly defined by emotive verbs, combined
with adjectives or word groups with adjectives, indicating appreciation. Examples of
such indicators are “appreciate”, “liked the relaxing attitude” and “relaxed and pretty
fun”. This category consists of comments by students indicating an important
element of their willingness to listen and talk to a mentor was the more casual,
‘relaxed’ setting of the mentoring session (as opposed to a regular class).
Following further analysis in step 4, this category too was refined into two
smaller groups. The first of these is “Appreciation of Approachability”. Comments in
this group specifically mention mentors’ status as peers (since mentors are all
students themselves) as an important element in establishing mentor-to-student
relatability and approachability, thus allowing students to feel less uncomfortable
when taking their text to, essentially, a stranger. An example of such a comment is the
following: “They understand our difficulties - regardless how petty - better in some
way, because, not that long ago, they were sitting exactly where we are sitting
now” (A4S6, p. 216).
AR Cycle 3 - Student Reflections Thijs Gillioen
!65
The second of the “Appreciation” subcategories defined in step 4 is
“Appreciation of Non-Directive Approach”. Lexical indicators for this category were
mentions of the non-directive mentoring style maintained by mentors, which could
be either explicit (by mentioning “non-directive”), or implicit (indicated by inferring
that the mentor did not do the work for them, instead merely asking questions that
lead to solutions for textual issues; for example, “thinking for myself”, “only asked
questions” and “helped but did not tell me what to do”). One student, for example,
wrote that “the fun thing was that he didn’t just tell me how I could do improve my
text and make everything link, he made me do it myself. He asked very direct
questions and my answers created the solution” (A4S7, p. 217).
This appreciation for the mentors’ approach is also expressed in a sixth
category, “Student Advice on Mentoring (Programme)”. This category was identified
in step 3 by the prompt “One piece of advice I would give to a first-year student next
year is …”, which is one of the prompts students were given as part of their Scribing 3
assignment (Appendix 1, p. 114) for use in their reflective writing. A group of students
responded to this question by saying that they would advise future students to go to a
mentoring session for any written assignment. The following piece of advice is one
example of this:
One piece of advice I would give to a first-year student next year is to go
to a mentoring session because it really helps. Even if you are too shy to
go just try it one time. The mentors give you really good feedback and
they are willing to help you with the mistakes or flaws in your text. It
can only help you and improve your assignment to go see a mentor.
(A4S3, p. 215)
AR Cycle 3 - Student Reflections Thijs Gillioen
!66
Similarly, many students either advised students to go see a mentor before it is made
mandatory to do so, or expressed a wish they had done so themselves: “If I had to
give someone advice for next year I would tell them to go see a mentor. It helped me a
lot and I regret I didn’t do it for the other assignments” (A4S8, p. 217). Finally, a
number of students also advised the course coordinator to make seeing a mentor
compulsory for earlier assignments, instead of only making it compulsory for the final
writing assignment of the course: “It might be a good idea to make this obligatory for
the first two essays too” (A4S9, p. 218). All of these advice-related comments were
gathered in the “Student Advice on Mentoring (Programme)” category.
The seventh category, “Attending Multiple Mentoring Sessions”, was identified
by students’ explicit mention of the idea of going to several mentoring sessions for a
given assignment, with a small number within this group explicitly mentioning
actually having done so. The following example is from one of those students:
I decided it was time to call in the help of a writing mentor (thank God
they exist). I went to our appointment […] In that moment it stroke to
me I still had no clue at all what focus meant. So luckily my writing
mentor helped me a lot trying to figure out what this whole focus thing
was all about. […] I decided to make another appointment with my
writing mentor and was hoping this time I would have it right. Luckily I
did and I was quite proud of myself I had mastered the focus-skill.
(A4S10, p. 218)
Finally, the eighth category, “Did not Feel Helped”, is made up of comments by
students who left the mentoring session not feeling very well-helped. Comments in
this category contain lexical items referring to disappointment or “having higher
expectations”, as is reflected in the following comment: “Don’t get me wrong, my
AR Cycle 3 - Student Reflections Thijs Gillioen
!67
mentor was the nicest [mentor] in the world, but I expected to get more specific
feedback” (A4S11, p. 218-219 [emphasis added]). The varying reasons for students
not feeling helped will be discussed in detail in the discussion section below.
4.3 Overview of Results
Once the categories detailed in section 4.2 had been established and refined based on
a full reading of all comments, all 229 reflective comments were analysed once more
and added to the relevant category. Table 08, below, offers an overview of the
frequency of each category, along with the total number of comments per category.
These findings will be discussed in detail in section 4.4.
AR Cycle 3 - Student Reflections Thijs Gillioen
Table 08. Overview of Categorised Reflective Writing
Reflection category # of comments % of total
“Helpful” categories 213 93.01%
Practical Reasons 77 33.62%
Confidence and/or Motivation Boost 33 14.41%
Extra Explanation Offered and/or Helps to Understand Theory 26 11.35%
Appreciation of Non-Directive Approach 23 10.04%
Student Advice on Mentoring (Programme) 19 8.30%
Attending Multiple Mentoring Sessions 15 6.55%
Appreciation of Approachability 13 5.68%
Did not Feel Helped 7 3.06%
Writer Development category 16 6.99%
Total 229 100%
!68
4.4 Discussion of Results
4.4.1 General Discussion
From Table 08, it is clear that a large majority of students find the peer mentoring
programme useful and/or important to their writing process in one way or another,
with 93.01% of all relevant comments indicating their mentoring session was helpful
primarily in the short term. For many students (33.62%), “Practical Reasons” are the
primary positive effect of a mentoring session. Seeing that practicality is the number
one concern among students (rather than long-term effects of mentoring, such as
assisting one’s development as a writer) is not particularly surprising, given that
students’ primary concern would be to pass the course. In this sense, this data
reinforces Harbord’s assessment that evaluation of mentoring programmes also has
to consider the practical effects of writing mentoring (2003). It also demonstrates
that a large subset of students perceive the mentoring programme to be beneficial to
their writing, and thus, efficacious, at least at guiding students towards improving
their writing for this particular assignment.
Aside from the short-term practical effects of a mentoring session, a significant
number of students also noted short-term psychological effects of a mentoring
session. 14.41% explicitly indicated they felt more confident after the mentoring
session, in addition to another 11.35% who asserted they had a better understanding
of either the task at hand or the underlying theoretical framework following the
mentoring session. Similarly, mentors’ approachability and non-directive approach
were also mentioned (5.68% and 10.04%, respectively) as important elements
contributing to making a student feel a mentoring session is a worthwhile part of
their writing process. Combined, these psychological elements comprise 41.48% of
students’ comments, thus eclipsing the “Practical Reasons” group. As such,
AR Cycle 3 - Student Reflections Thijs Gillioen
!69
psychological effects of mentoring sessions on students should not be underestimated
as contributors to the programme’s efficacy.
On top of the short-term effects of mentoring sessions on students themselves,
a group of students also indicated they would advise future students of English to
attend a mentoring session for (any of) their writing assignments, even if it was not
mandatory for them to do so. Although it can be argued that most comments in the
categories above can be read as implicit recommendations of mentors to future
students, 8.30% of students’ comments explicitly mention this advice. Although this
number, on its own, is not extremely high, it is significant considering that many
first-year students are overwhelmed with the amount of work they have to do - which
has been established in literature (e.g. Chraif 2015; Ruiz-Gallardo et al. 2016) and in
the reflective comments: “the obligated mentoring session we had to attend was –
despite the loss of time that we need sooooooo bad – actually not a bad
thing.” (A4S12, p. 219 [emphasis added]). Thus, these already busy students
recommending taking up an additional, non-obligatory activity, can be seen as an
indicator of at least some level of student satisfaction and of programme efficacy.
The group “Attending Multiple Mentoring Sessions” received 6.55% of the
total comments gathered during the analysis. The fact that some students voluntarily
attended multiple mentoring sessions, or would recommend seeing a mentor
multiple times for a given assignment, again serves as an indicator that these
students perceive the mentoring programme as efficient in its ability to guide
students in their writing process.
However, a small subgroup (3.06%) of students indicated their experience
during the mentoring session was not positive, or, at least, that the mentoring session
did not help them as much as they had expected. Although this group is, statistically
AR Cycle 3 - Student Reflections Thijs Gillioen
!70
speaking, too small to be considered relevant, this dissertation will discuss this group
below, in section 4.4.2.8, in order to understand why these students, in these specific
cases, did not feel helped. Given that the large majority of students indicated having a
positive experience during their mentoring session, it is possible that elements
extrinsic to the mentoring session itself (such as faulty expectations or a
misunderstanding of the workings of the mentoring process) are tied to the
dissatisfaction expressed by these students.
Finally, sixteen comments were gathered that relate to the long-term effects of
the mentoring session, representing 6.99% of the gathered data. The fact that any
comments related to long-term writer development are present, is another indication
the programme has at least partly succeeded in moving beyond the ‘here-and-now’ of
assisting students’ writing, instead having had an impact on students’ long-term
development as writers. These comments will be discussed in detail in section 4.4.2.9.
4.4.2 Discussion of Reflections per Category
Having given an overview of the results of the categorisation process and the
frequency with which each category appears, the following section will now look at
each of the categories defined above and attempt to establish what, in each category,
is a contributor to the programme’s efficacy, or students’ perceptions of its efficacy.
4.4.2.1 Practical Reasons
As noted above, this “Practical Reasons” category consists of student comments
indicating they perceived the mentoring session to be useful because it helped them,
in the short term, to refine and finish their essay. One example of such a comment is
the following: “the session with my mentor gave me the right directions to find the
AR Cycle 3 - Student Reflections Thijs Gillioen
!71
real focus [of my text]” (A4S1, p. 214). Students’ appreciation of such support is not
surprising, on its own, since this support will (indirectly) help them gain a higher
grade.
It is interesting to note, however, that students found practical benefits to
mentoring sessions in different stages of their writing process. Some students
indicated they had difficulty finding a focus for their assignment, but were able to
find a “specific focus on [their] topic during the mentoring session” (A4S13, p. 219).
Furthermore, students who came in with a limited draft found their session with a
mentor useful because it helped them develop their ideas, as is indicated by: “I have
to admit, [the mentor] was a great help in figuring out all of this. What [the mentor]
said, the ideas [the mentor] had on my text were in line with what I thought. Only I
couldn’t really put it down on paper, whereas [the mentor] did” (A4S14, p. 220).
Likewise, another student found “the mentor [they] had was very helpful and gave a
lot of advice to help [them] to finish and perfect [their] scribing at home” (A4S15, p.
220). For these students, the mentoring programme’s efficacy lies in its ability to
guide students’ thoughts as they process their ideas into text.
For students who had an advanced or complete draft, support from mentors
came in the form of help in refining their current texts. This refinement could pertain
to the structure, which “mentor[s] could really [help them] get right” (A4S16, p. 220),
or to the focus of the students’ texts. Pertaining to focus, some students indicated
they were helped in “keeping [their] focus more narrow” (A4S17, p. 220), in making it
“explicitly stated” (A4S18, p. 221), or even in finding a new focus that “suited the text
much better than the one [they] initially had” (A4S19, p. 221). Aside from focus-
related support, students also noted mentors could help them find issues they “saw,
but could not define” (A4S20, p. 221) or issues students just “really hadn’t seen
AR Cycle 3 - Student Reflections Thijs Gillioen
!72
before” (A4S21, p. 222). Students posed this form of support is made possible thanks
to the mentors’ ability to “deconstruct the parts of [students’] own writing” (A4S22, p.
222): thus, part of the programme’s efficacy lies in mentors offering students the
opportunity for their drafts to be read by readers who have been trained to offer them
support.
It is somewhat surprising to find students this receptive to near-peer support:
teachers often note students either simply do not care about feedback, or “do not act
on [it]” (Hepplestone et al. 2011: 118). Peer support, specifically, has often been
found to evoke “mixed, and often negative reactions” (Best et al. 2015: 332), in
addition to being “a process that is laden with strong negative emotions” (Best et al.
2015: 344) for students. Best et al. have argued the peer support process is often
negative for students, not because of the process itself, but because of group
dynamics, wherein a dedicated student might be ‘matched’ for support with a less-
dedicated student, thus resulting in an “unfair” situation (2015: 345). It can be
argued, however, that this group dynamics issue is rendered void in the current
programme. This is the case partly because all mentors are volunteers (thus, are
dedicated by definition), and partly because they have all received identical training
and (as such) have near-identical expertise. One student summed it up as follows:
I was afraid for my mentoring session […]. I was just thinking about the
fact that it would be a waste of time and that I'd come outside even
more puzzled. My worries were unauthorized. I had an appointment
with [a mentor], who was very talkative. I got rid of my stress
immediately. [The mentor] was very nice and when I told [the mentor]
my name, [they] said: 'Once upon a time, isn't it?' I was so so positively
AR Cycle 3 - Student Reflections Thijs Gillioen
!73
surprised with that answer, because it seemed to me that [the mentor]
was very well informed and put a lot of effort in [their] part of the deal.
(A4S23, p. 222)
As well as being dedicated, students’ perception of mentors as ‘experts’ is
widespread: they indicated that “writing mentors clearly have a lot more
experience” (A4S24, p. 223), “aren’t hesitant and have more experience” (A4S25, p.
223) and “clearly know what they’re talking about” (A4S26, p. 223), which made at
least one student note they took “the advice/feedback from a mentor way more
serious[…] than from a peer” (A4S27, p. 224). An additional issue for engaging with
support on written assignments, noted by Best et al., are “vague comments[, which]
students find confusing and discouraging” (2015: 343). Because of the one-to-one
nature of a mentoring session, anything that is not immediately clear to a student can
be cleared up ‘on the spot’ by the mentor, thus again eliminating the issue. One
student phrased it as follows: “I also had some questions about my focus, if it was
near enough, and the structure of my text. It made thin[g]s clear for me and
afterwards, I knew exactly how I wanted to change my text” (A4S28, p. 224). As such,
mentors’ position as near-peers, but also relative ‘experts’, combined with the
possibility for students to ask them questions during at stages of the writing process,
are the main contributors to the programme’s efficacy in this category.
AR Cycle 3 - Student Reflections Thijs Gillioen
!74
4.4.2.2 Confidence and/or Motivation Boost
In addition to finding practical benefits to a mentoring session, many students
indicated the mentoring session had a positive psychological effect on them. One of
these effects was an increase in confidence, or a boost in motivation. Again, students
indicated they could find psychological support in the mentoring programme
throughout different stages of the writing process.
For students who had not yet completed their draft, a mentoring session was,
quite often, a powerful motivator to keep writing. One student noted that “when
[they] got home afterwards, [they] felt a lot more motivated to start again than [they]
did when [they] left their apartment” (A4S4, p. 215). Similar reflections noted
mentors could help their peers get “[their] enthusiasm back” (A4S29, p. 225), feel
“much more secure about [their] work” (A4S30, p. 225), “relieved” (A4S31, p. 225),
“reinvigorated” (A4S22, p. 222) and “confident […] because [their] writing mentors
had made clear that [they] [were] doing well” (A4S32, p. 226). Finally, one student
felt like they were given “a bit of control back”, which gave them “a bit of reason to
keep writing” (A4S33, p. 226).
Students who had been able to finalise a draft were similarly encouraged.
Although sessions with mentors often result in extra work for students, students
indicated they were “very positive about [their] writing again” (A4S20, p. 221), in
addition to feeling “sure and a bit more confident” (A4S34, p. 226). Even students
who did not have a lot of changes to make following their mentoring sessions noted
feeling “a bit more confident” (A4S32, p. 226) and “incredibly proud […] like I’m
getting the hang of it” (A4S35, p. 227).
The source of the programme’s efficacy in this field can be found in mentors’
positions as ‘students-who-have-been-there’, and through this, their ability to reduce
AR Cycle 3 - Student Reflections Thijs Gillioen
!75
student anxiety. Research supports the perception that peer mentors can be
significant in reducing such anxiety (Rodger & Tremblay 2003: 13), which is
corroborated by students in the current setting indicating mentors are not just
mentors, “but also […] still students. […] They understand our difficulties - regardless
how petty - better in some way, because, not that long ago, they were sitting exactly
where we are sitting now” (A4S6, p. 216). One student even indicated mentors “make
the transition from high school to university easier” (A4S36, p. 227), which again
underscores the point made above with regards to decreasing student anxiety levels.
As such, although the effects of this category on students may not be immediately
visible, they are important to student wellbeing, ultimately resulting in increased
retention rates and increased academic gains (Campbell & Campbell 1997; Salinitri
2005), thus indicating long-term programme efficacy.
4.4.2.3 Extra Explanation Offered and/or Helps to Understand Theory
Another psychological area where the mentoring programme appears to have
achieved some efficacy is in offering a limited form of peer tutoring. During
mentoring sessions, it became clear some students were still struggling to grasp some
elements of Haas’ framework for writing, which they had been taught over the course
of the first semester. The mandatory mentoring session offered these students
another opportunity to receive additional explanation, thus indirectly resulting in
textual improvements, writer development, and psychological wellbeing (by feeling
more secure in understanding the theory). One example of such a comment is the
following: “in the session they explained the concept of focus really good, so I finally
got it” (A4S37, p. 227).
AR Cycle 3 - Student Reflections Thijs Gillioen
!76
Mentors’ individual and personal styles are an important contributor to
programme efficacy in this area. Although all mentors are capable of explaining Haas’
framework thanks to having been given extensive training with it, they are able to
bring in their own perspectives on the theoretical framework, thus offering new ways
of explaining it to students. One student noted a metaphor used by a mentor: “a little
scheme” of her own that “was veeeeeery helpful” (A4S38, p. 228), whereas another
student was helped by “the example of the dog” (A4S39, p. 228). In addition,
mentors’ personal approach made one student feel like they had “finally learned
something” (A4S40, p. 228). In this sense, these comments from students support
the assertion of Cho & MacArthur (discussed above) that noted students’ close
proximity to each other allows for them to “detect problems from their own
perspective”, thus “generating solutions to the problems” (2010: 329). On top of this,
in their own meta-study of mentoring research, Orland-Barak & Hasin noted “the
ability to talk about teaching in ways that connect between theory, practice, and the
particular context of the mentee” to be a characteristic of good mentoring (2010:
429), thus underlining the importance of this category for programme efficacy.
4.4.2.4 Appreciation of Non-Directive Approach
Another short-term trend indicated as positive by many comments (10.04%) was
mentors’ non-directive approach. Instead of offering immediate ways to ‘fix’ issues
spotted in student texts, mentors guide their mentees towards solving the problem on
their own. One student summarised it as follows:
Even though I thought it was hard to answer [the mentor’s] questions
about my text, I did like the way [the mentor] did it. [They] let me think
about my own text, which is very hard, but refreshing when you see
AR Cycle 3 - Student Reflections Thijs Gillioen
!77
where you went wrong. [The mentor] didn't just say what [they] did and
didn't like, but [they] let me think and search myself, which I didn't
expect, but liked very much. This way, I have the feeling I did learn
something and above all, I understand [the mentor’s] remarks, because
eventually I saw them myself.
(A4S41, p. 229)
This appreciation may, at first glance, appear contradictory. Students’ priority
is generally to pass their assignments, with learning becoming a secondary goal.
However, in this specific context, several comments indicated the non-directive
approach was appreciated because students were given the “chance to think for
[them]self” (A4S42, p. 229), instead of explicitly being told what to do. Similar
comments noted they appreciated that “it was up to [them] to decide what was going
to be [their] focus and how [their] text would look like” (A4S43, p. 229) and that by
“the questions that were asked to [the student], [they] found out [for themself] what
was wrong with what [they] had written and that was actually the nicest thing”.
(A4S44, p. 230). Or, as one student put it, “it’s way more educational when [they] are
set on a path to discover for [them]self” (A4S45, p. 230). Therefore, the programme’s
main form of efficacy for these students is that it empowers them, allowing to resolve
issues themselves, rather than giving them a straightforward solution. This is
supported by research conducted by Zepke & Leach, who note “enabling students to
work autonomously” is an important factor in student engagement (2010: 170).
AR Cycle 3 - Student Reflections Thijs Gillioen
!78
4.4.2.5 Student Advice on Mentoring (Programme)
In their reflective scribbling, students could also give advice to their successors
regarding the writing course. A number of students elected to give advice pertaining
to the mentoring programme. From this, two trends emerged: firstly, students
advised their future peers to go see a mentor (sooner); secondly, students advised the
course teacher to make mentoring sessions mandatory for earlier assignments.
The first recurring idea was that students gave the advice to see a mentor, or to
go see a mentor before the last assignment. Some students expressed this by saying
they “wish[ed] [they] had used them sooner” (A4S46, p. 231), or “sincerely regret not
having seen a writing mentor” (A4S47, p. 231) for earlier writing assignments. Others
explicitly “advise[d] next year’s students to use [mentors] from the start” (A4S46, p.
231), since “asking for mentor help is not a shame or a nerdy thing to do, neither is it
a waste of your time” (A4S48, p. 232), but it “really helps[,] even if you are too
shy” (A4S3, p. 215), and, after all, “[the mentors] don’t bite” (A4S49, p. 232).
Aside from pointing out the usefulness of a mentoring session to future
students, some of the current mentees also advised the teacher of writing to make
mentoring compulsory for earlier writing assignments, as is demonstrated by a
certain student commenting that “it might be a good idea to make this [mentoring]
obligatory for the first two essays too” (A4S50, p. 232). This advice is, in nearly all
cases, motivated by students’ initial surprise to the usefulness of a session: “if I knew
that a mentoring session would be like this, I would have done it for my other essays
too” (A4S51, p. 232). Despite the fact that students were informed multiple times as
to how a mentoring session works, several students noted its usefulness only became
clear when having an actual conversation with a mentor.
AR Cycle 3 - Student Reflections Thijs Gillioen
!79
The “Student Advice on Mentoring” category does not provide any explicit
indications of the efficacy of the mentoring programme. However, by inversion, the
fact that a significant number of students advise their future peers to make use of its
services, and the fact that they propose to have more mandatory sessions (and thus,
more work), indicates these students perceive the programme to have a high efficacy
at what it does. These students’ recommendations can be seen as significant
suggestions for programme improvement, which will be discussed further below.
4.4.2.6 Attending Multiple Mentoring Sessions
One of the recommendations outlined in section 3.6, based on the analysis of student
texts, and existing literature, was to encourage mentors to guide their students over
multiple mentoring sessions. A number of students referred to this in their reflective
writing: some by indicating they actually did attend multiple sessions, and some by
indirectly indicating how a second session could have been useful to them.
A small number of students voluntarily attended more than one mentoring
session. These students indicated the second session was useful as well, partly
because they received confirmation of the improvement over their first draft, and
partly because mentors could guide them in resolving any remaining (or new) issues:
I have seen my mentor twice because I wrote a completely different text
after my first mentoring session and it was way easier to write that
second draft, I even enjoyed writing it. I did also feel a lot more
confident about my second draft. My second mentoring session went
good as well. My text had improved a lot and there were only a couple of
linking problems. Those problems were a bit harder to fix because I
AR Cycle 3 - Student Reflections Thijs Gillioen
!80
didn’t want to change too much. I do think I managed to link them
without writing something else entirely.
(A4S52, p. 233)
Another comment that evidenced the effect of confirmation and further help is the
following:
I met up with the [second] writing mentor who was actually a really
kind person. [The mentor] gave me some helpful advice as well as
several compliments. It has kind of cheered me up and gave me some
more motivation.
(A4S53, p. 233)
In one case, a student attended a mentoring session where the mentor could
“point out nothing that would make [the student’s] text better”. Although, in such a
case, the practical usefulness of a second session is quite low, the student still
experienced some benefit from the session, psychologically, since they “couldn’t be
happier […] and […] so incredibly proud” (A4S54, p. 233). This underscores the
potential, both practically and psychologically, of encouraging students to attend
multiple mentoring sessions. The fact that these students voluntarily returned for a
second mentoring session can be seen as an indicator that these students perceived
the programme to be efficient in its goal of supporting them during the writing
process.
In a number of other cases, comments indicated how a second mentoring
session could have been helpful to certain students. One comment revealed a student
was still unsure about their writing, being afraid that “after rewriting, […] [they]
made the same mistakes again and failed again” (A4S66, p. 239). Given the data
analysed in section 3.5.2.1 (p. 44), indicating some students do retain HOC problems
AR Cycle 3 - Student Reflections Thijs Gillioen
!81
(or create new ones), it is not unfair to argue this student could have benefitted
significantly from a second mentoring session. This student felt they had to “rewrite
[their] whole text”(A4S66, p. 239): if mentors are aware of such cases, urging such
students to return for a second session could prove beneficial to the student.
Although instances like these do not directly support programme efficacy, they do
support the suggestion made above (section 3.6) with regards to multiple mentoring
sessions.
4.4.2.7 Appreciation of Approachability
A final contributor to short-term programme efficacy was the mentors’
approachability, which was noted in 5.68% of the reflective comments. Although
approachability is not an immediate contributor to programme efficacy, it allows for
students to feel at ease when conversing with a mentor (who is ultimately still a
stranger), thus making the support process easier. In addition, an approachable
attitude makes it more likely that students will voluntarily return for another session
afterwards. As such, this category represents an indirect but important long-term
contributor to programme efficacy and sustainability.
A significant element in maintaining an approachable atmosphere was
mentors’ status as peers. To students, this enables mentors to “understand our
difficulties - regardless how petty - better in some way, because, not that long ago,
they were sitting exactly where we are sitting now” (A4S6, p. 216). The fact that
mentors ‘have been there’ allows them to tap into a shared pool of experiences that
decreases the uneasiness of the situation, as indicated by this comment: “we spent
[some time] talking about Latin, secondary school, study choices and other
interesting stuff. I loved that we could do that (…) I simply think it was amazing that
AR Cycle 3 - Student Reflections Thijs Gillioen
!82
we could just chat like that.” (A4S55, p. 234). This also made the sessions a “lot less
scary than […] expected” (A4S56, p. 235). Mentors’ position as peers also created a
sense that sessions were “relaxed and actually pretty fun” (A4S57, p. 235).
Another element that contributed to approachability was mentors’
preparedness for the session. As noted by Best et al., students often feel negatively
about feedback from unprepared peers (2015: 345), but students do assert peer
feedback can be valuable when the peer is prepared. The following comment indicates
a student’s positive surprise to the peer mentor belonging to the latter category:
I was just thinking about the fact that it would be a waste of me and that
I'd come outside even more puzzled. My worries were unauthorized. I
had an appointment with [a mentor], who was very talkative. I got rid of
my stress immediately. [The mentor] was very nice and when I told
[them] my name, [they] said: 'Once upon a me, isn't it?' I was so so
positively surprised with that answer, because it seemed to me that [the
mentor] was very well informed and put a lot of effort in [their] part of
the deal.
(A4S23, p. 222)
A final contributor to mentors’ approachability was the mentoring room where
sessions took place. Students noted that the “cozy atmosphere” (A4S58, p. 235) made
it easier for them to feel at ease when speaking with a mentor, which in term allowed
for students to focus on developing their writing skills.
These elements contribute towards an “interpersonal relationship” (Orland-
Barak & Hasin 2010: 429) being created between mentors and students, which they
note as another important characteristic of good mentoring. This relationship allows
for mentors to be approachable sources of support, rather than another
AR Cycle 3 - Student Reflections Thijs Gillioen
!83
institutionalised (and thus, distant) form of help students would be less eager to
access.
4.4.2.8 Students Who Did not Feel Helped
Having discussed the students who experienced the mentoring session in a positive
way, the current section now turns to the minority of students (3.06%) who did not
feel helped by the mentoring session, or whose expectations were not entirely met.
Understanding why these students did not perceive the mentoring session to be
helpful or successful can lead to interesting data points for further programme
improvement.
An explanation for these students’ disappointment may lie in the fact that it
appears these students expected the mentor to be directive, or to provide explicit
instruction on how these students’ texts could be ‘fixed’. One example of a comment
indicating as much is the following:
[…] Don’t get me wrong, my mentor was the nicest [mentor] in the
world, but I expected to get more specific feedback. […] Then [the
mentor] asked me what my focus/topic was. I had no idea myself so I
couldn’t quite answer [their] question. […] The concept of ‘beetles’ was
too wide and I should narrow it down. I didn’t agree with that because,
in my opinion, I had already narrowed it down. […] I asked [the
mentor] to give me an example of a good focus for my text. For some
reason [they] refused to give me one... I was pretty disappointed once I
left the mentor room...
(A4S59, p. 236)
AR Cycle 3 - Student Reflections Thijs Gillioen
!84
Thus, in this comment, the student’s disappointment stems from the mentor’s refusal
to provide an adequate focus for the student’s text. Since mentors are explicitly
instructed to maintain a non-directive posture, the mentor was simply following
programme guidelines. However, this resulted in a less-than ideal experience for this
student, who was hoping to have their text ‘fixed’. A similar situation is presented in
the following comment:
[The mentor] said what my peers said, but I already told [them] I know
what’s wrong. [They] said there was too much new given information,
but how I could avoid this hasn’t [they] mentioned. When I left the
mentoring room, I was so confused: my peers said to me that the stuff
I’m working with for my text is great, but I have to work on it. [The
mentor] said the same things, but I had the feeling [they] didn’t like the
topic or focus I had in mind.
(A4S60, p. 236)
Again, the issue here appears to be that the student expected the mentor to be
directive, i.e. to tell them how they could fix their specific issue (here: a problematic
“Given/New” HOC problem).
On the other hand, the comment above also indicates the student did not
entirely comply with the assignment details: students were instructed to re-write
their text based on their peers’ feedback, before seeing a writing mentor (S3
Assignment, Appendix 1, p. 112). A different student also noted that “everything [the
mentor] had to say, my peer had already written down” (A4S61, p. 237), which again
could have been avoided had the student complied with the assignment details.
Despite this, dismissing these students’ issues entirely would be to dismiss
these students as a whole; since the programme aims to reach all students,
AR Cycle 3 - Student Reflections Thijs Gillioen
!85
considerations should be made to ensure that these students, too, can be assisted
during their writing process. The strict application of the non-directive approach has
resulted in a (small) number of students leaving the mentoring session dissatisfied;
this underlines the argument of Harbord (2003) and Crasborn et al. (2011) for a
flexible approach based on each student’s needs.
4.4.2.9 Writer Development
In addition to the (relatively) short-term benefits to mentoring discussed in the
sections above, a small group (6.99%) of comments signal a long(er)-term advantage
to attending a mentoring session. Since the comments analysed for the purposes of
this research were collected after a single mentoring session, one has to be careful
about drawing long-term conclusions of its effects. Despite this, a few students
explicitly noted the mentoring session aided them in their development as writers. In
what follows, two groups of these comments will be discussed; they are representative
of the other comments in this category.
The first group contains comments related to students’ ability to give their
peers support during their writing process. One student said “[they] already feel like
[they] developed some kind of skill that [they] didn’t have before [they] went to
university” (A4S62, p. 237). Another student similarly commented:
[A fellow student] had already gone to see a writing mentor, so [they]
[were] able to tell me things about my text that the writing mentor had
also said about [their] text, which was of course very helpful, because
[they] already knew what [they] had to improve and [the fellow student]
saw the kind of same defaults in my text. [That student] told me that I
should state my focus more explicitly in every paragraph. But [they]
AR Cycle 3 - Student Reflections Thijs Gillioen
!86
also told me that my text already had a pretty clear focus and that it was
structured well, so that was nice to hear. Those three things are also
kind of exactly what the writing mentor ([…]) told me about my text.
(A4S63, p. 238)
Given that this student indicates their peer was able to find several issues in their text
due to having had a mentoring session, it can be argued that the mentoring session
not only helped this peer improve her own writing, but also her ability to help others
during their writing process. This indicates a possible long(er)-term developmental
effect of the mentoring session.
The second group contains comments related to students’ ability to write.
These comments indicate that, following the mentoring session, students felt like they
would be able to write better texts for any future assignment. An example of such a
comment is: “After the mentoring session I am now convinced that if I had to write
another focus-based story, I would find a good focus all by myself!” (A4S2, p. 215).
These comments provide some evidence that the programme was not only able
to help students in the short term, but also in the long term, despite the fact that the
data analysed here only pertain to one specific assignment. This might serve as
another indirect indicator of (long-term) programme efficacy.
AR Cycle 3 - Student Reflections Thijs Gillioen
!87
4.5 Recommendations for Programme Improvements
Having elaborated upon each category of students’ reflective comments, this section
aims to suggest how these data can be used to improve the mentoring programme’s
practices, and its efficacy. Although only a small subset (3.06%) of students posited
not feeling helped by the mentoring session at all, there are a number of ways in
which the programme could improve to both reduce this number, and further assist
students who did feel helped during their writing process.
Firstly, the programme’s strict adherence to the non-directive approach to
mentoring may be in need of reassessment. Although this dissertation has not
specifically focussed upon this element, it became clear at several points during the
discussion above that this approach limited the programme’s ability to aid some
students in an optimal manner. Although many students declared that they
appreciated the non-directive approach, if the programme aims to reach all students,
the reality that this mentoring style does not work for some mentees has to be
considered and adapted to. This situation in the programme’s particular context is
similar to claims made in the literature discussed above (e.g. Kullman 1998; Harbord
2003); likewise, Crasborn et al. argued that “mentoring is a multifaceted
phenomenon and that no straightforward description or guidelines for success can be
given” (2011: 328).
Such adaptations have to account for the two points of view that inform the
programme. The first of these perspectives is that of the mentors: for them, the non-
directive approach offers the advantage of avoiding regressing into editors, in
addition to shielding them from making content- or analysis-related comments,
which could pose the possible problem of the mentor being wrong, or the assignment
instructor not appreciating such an intervention. However, in addition to mentors,
AR Cycle 3 - Student Reflections Thijs Gillioen
!88
students are the second group to which the programme is beholden: for them, the
non-directive approach enables them to reflect on their own work, thus enhancing
their opportunity for learning. Despite this, for students, passing the course is often a
primary concern, to which learning benefits (created by the non-directive approach)
have become secondary. Facing this divide, if the programme is to increase its
efficacy, both sides have to be accounted for.
Since the majority of students did indicate feeling helped, and since there are
some indications of writer development following a mentoring session, abandoning
the non-directive approach entirely would be too drastic. However, since some
students do need more directive support, one possibility would be to allow ‘senior’
mentors (i.e. mentors who have two or more years of mentoring experience) to go
beyond non-directive support and give students directive assistance. Although this
does limit mentors’ ‘shielding’ (discussed above), containing this directive advice to
an essay’s structure only (thus not commenting upon content), combined with senior
mentors’ experience, should limit their ‘risk’. For less experienced mentors, this
offers a double benefit: by referring students in need of directive support to senior
team members, ‘junior’ mentors can continue practising non-directive skills with the
majority of students for whom the non-directive approach does yield positive results.
Additionally, this removes any doubt or ‘uneasiness’ (junior) mentors may experience
when a student does not respond positively to the non-directive approach. By
opening up options for directive support, the programme could reach a larger
audience, and extend its usefulness to the current audience, thus improving its
efficacy.
The second and third recommendations pertain to introducing the mentoring
programme (and its practices) to students sooner in the semester. As discussed
AR Cycle 3 - Student Reflections Thijs Gillioen
!89
above, several students regretted not seeing a mentor earlier, thus advising their
future peers to approach a writing mentor at a less advanced stage of the semester.
Several students students noted their initial apprehension to see a mentor stemmed
from being “shy” (A4S3, p. 215) or finding the idea of a mentoring session
“scary” (A4S3, p. 215; A4S15, p. 220; A4S56, p. 235). Other students indicated they
“didn’t quite know what to expect of it” (A4S64, p. 238), and that the session was
“nothing like [they] expected” (A4S57, p. 235). Clearly, students’ apprehension is, in
part, caused by their unfamiliarity with mentoring (as a concept, and as a practical
instrument). Therefore, the following two recommendations aim to provide
suggestions to improve this situation.
A first way to make students more familiar with the concept of mentoring, and
the writing mentors themselves, is to implement a full mentoring session in one of
the writing classes. The familiar environment of the classroom, coupled with the
presence of a teacher as an ‘official’, could significantly reduce students’ anxiety when
talking to (at that point) a ‘stranger’ about their writing. Furthermore, by introducing
mentoring in class, students would get a clear view on what can (and cannot) be
expected from a mentoring session, thus clearing up the ‘unease’ indicated by the
students quoted above. Given the mostly positive post-mentoring experiences
discussed above, it is not unfair to pose that, once students are more familiar with
mentoring, they would be more inclined to visit a mentor, either for follow-up
mentoring sessions or for other assignments.
A second way to introduce students to mentoring is by making mentoring
mandatory earlier on in the semester, either by introducing it in-class, as suggested
above, or by making attendance of a mentoring session compulsory for earlier writing
assignments. This suggestion was spurred by comments from students:
AR Cycle 3 - Student Reflections Thijs Gillioen
!90
Something I would change about the writing class is a mandatory
writing mentor session for the first of second assignment already.
Because to me, it was really useful and I wish I had done it for my
previous assignments as well.
(A4S65, p. 239)
Mandating mentoring earlier in the semester serves a double purpose: firstly, it
would allow students to ‘grasp’ the concept of mentoring early on, thus making it
more accessible for future assignments. Secondly, it would allow mentors to guide
students from an early stage in the semester (rather than merely at the end of it), thus
increasing the potential for mentors to (through cognitive apprenticeship) have an
impact on students’ development as writers. The first suggestion could lead to short-
term improvements in programme efficacy; the second could lead to (increased)
long-term efficacy.
AR Cycle 3 - Student Reflections Thijs Gillioen
!91
IV Limitations, Conclusion, and Future Research
Having analysed and discussed the efficacy of the UGent peer writing mentoring
programme from two perspectives, and having made suggestions for programme
improvements based on these analyses, the current section now turns to discuss the
limitations of the current research. Following that, a conclusion of this research is
presented, before suggesting some avenues for future inquiry.
1 Limitations of the Current Research
Throughout the current AR cycle, the aim has been to conduct a thorough and
founded evaluation of the programme’s practices, and to use this evaluation to
propose grounded suggestions for improvements. However, as with any research, this
project was constrained by time and scope; therefore, it faced a number of
limitations.
A first limitation is that the essays analysed as part of the pre- and post text
analysis were written by students who had seen the same mentor (who is the author
of this dissertation). In the context of this research, this was required to avoid bias,
and because of practical limitations: it was impossible, within the short timeframe, to
account for personal differences in mentoring style and mentors’ differing personal
relations to students influencing the mentoring process. However, if due recognition
is given to these potential differences, considering the fact that all mentors are
volunteers (and thus more or less equally dedicated) and the fact that all mentors
have received the same training, this paper has argued these results can be
reasonably extrapolated to the programme as a whole. However, a more thorough
Conclusion Thijs Gillioen
!92
evaluation could include texts mentored by different (and, ideally, all) mentors
participating in the programme.
A second limitation is the possibility of researcher bias in evaluating the texts.
Even though attempts were made, by conducting inter-rater reliability tests, to
mitigate possible influences, the likelihood of some bias remains. Each of the three
raters was, to a certain extent, personally invested in the programme’s success: the
author and the undergraduate student have been writing mentors for several years;
the framework developer, who was the third rater, is also the programme’s
administrator. Although, while rating, each rater independently found the same
issues with every text, one cannot entirely eliminate the possibility of some common
bias. Completely eliminating any bias was not practically possible, and may, by
nature, not be possible at all. To avoid bias, raters from outside the programme would
have to be employed. Thus, one would have to ask either the other teachers involved
in the ETTV course (who may not have time to do so, and, given their familiarity with
the framework and the mentoring programme, can arguably also be considered
biased), or external raters (who would have to be taught the framework, and would
have to be paid for their time). As such, although some bias could not be completely
avoided, measures were taken to mitigate this bias.
Thirdly, the texts and reflection used as data for this analysis were made as
part of a single assignment, because of which this evaluation of the programme’s
efficacy was based on a limited point in time: the final weeks of the first semester of
the academic year 2015 - 2016. Thus, objective claims can only be made about this
period. More, and ideally longitudinal, research is required to more fully assess the
programme’s efficacy and the effects of suggestions such as those proposed above
upon said efficacy.
Conclusion Thijs Gillioen
!93
Finally, this paper has focussed uniquely on the student perspective on
programme efficacy (by analysing their written product, and their reflections). The
points of view of other ‘stakeholders’ in the programme, such as the other mentors,
the teachers responsible for teaching and assessing writing, and the English Studies
Group as a whole, could not be included. Future investigations into these perspectives
could reveal further areas where programme efficacy could be improved.
2 Conclusion of the Current Research
During the academic year 2013 - 2014, a peer mentoring programme was established
in the English Studies Group at UGent to assist the teaching of writing by offering
students additional opportunities for support on their written product, as well as at
least some opportunities for cognitive apprenticeship, both of which were defined as
essential by Kellogg (2008) in his research on writer development.
The current Action Research-based applied linguistics paper aimed to conduct
(within the limitations outlined above) a thorough evaluation of the programme’s
efficacy. Based on literature on the evaluation of similar programmes and writing
centres, this evaluation was conducted from two perspectives. The first perspective
was that of a text analysis, comparing pre-mentoring texts to post-mentoring texts.
The second perspective looked at programme efficacy as perceived by students,
through their reflective writing.
To conduct an analysis of pre- and post-mentoring texts, a reliable rating scale,
based on Haas’ framework (2016), had to be developed first. To do so, the author and
two other raters conducted a series of pilot tests and developed such a scale: it was
designed to be applicable to both the number of essays studied in this dissertation,
and large(r) volumes of essays, such as essays submitted for (other) courses. Based on
Conclusion Thijs Gillioen
!94
these pilot tests and post-test discussions, a number of points, such as how to deal
with spelling and/or grammar mistakes (later called L-LOCs), and how to assess
essays that did not meet the assignment’s minimum requirements, were discussed
and ultimately agreed upon.
The rating scale was then applied by the three raters to the corpus of essays
collected. Each rater independently read each essay to identify places of
disengagement, which were then analysed using Haas’ framework. After this, a
general mark was agreed upon, based on the rating scale. This process revealed that
mentoring has a positive effect on the written product of students, with an average
improvement of 27.29%. L-LOC problems in post-mentoring essays are reduced by
42.86%, whereas the frequency of LOC issues was found to decrease by 66.67%. HOC
problems are 42.86% less frequent. This decrease rises to 67.24% if one discounts the
new HOC issues created by students while rewriting their essays after the mentoring
session. In 20% of the cases, all HOC problems of the original draft were resolved;
53.33% of the essays studied show a HOC resolution rate of equal to or higher than
75% following the mentoring session. These numbers indicate the programme
achieves a relatively high efficacy rate, considering the limited one-off timespan of
thirty minutes for a mentoring session.
The analysis also revealed a number of specific areas where the programme
can (further) increase its efficacy. Whereas some HOC problems, such as “Precision”
problems or “Self-Containment” issues have a 100% improvement rate (thus,
generally speaking, solving any such issues that were present in the pre-mentoring
text), other HOC problems showed lower improvement rates. Examples of these are
“Focus too Wide” (50% improvement), “Relevancy” problem (44.44%) and “Complex
Issues” (25%).
Conclusion Thijs Gillioen
!95
To advance the programme’s efficacy in general, and specifically in the areas
described above, a number of suggestions were made that could lead to programme
improvements. The first of these was to encourage mentors to engage with their
student mentees multiple times by asking them to return for follow-up mentoring
sessions. This recommendation is supported by research noting that three mentoring
sessions is an important threshold (Irvin 2014). Doing so could allow for further
guidance of the students, and could give way to resolving the HOC issues created by
students following the mentoring session.
The second recommendation then pertained to mentor instruction. It was
argued that additional instruction, specifically in the areas that saw little HOC
improvements, could lead to additional programme improvements. To do so, the
research suggested employing mentors in the marking process for ETTVI
assignments. This would allow mentors to practise the identification of HOCs and
thus gain a deeper understanding of those cognitive issues, in addition to facilitating
a number of long(er)-term benefits such as increased problem-solving skills. Aside
from that, such involvement could reduce the work load on the course teachers, a
benefit which is backed by research confirming the validity and usefulness of peer
assessment (e.g. Iraji et al. 2016).
The second perspective from which programme efficacy was evaluated was
that of the students. Their written reflections were analysed for elements that indicate
where they perceive the programme to be efficacious, and where further
improvements to efficacy can be made. This analysis of 175 pieces of reflective writing
yielded 229 relevant comments, which were then categorised in eight categories. Only
a small subset of students indicated they did not feel helped; the other seven
categories each pertained to a different reason why students found their mentoring
Conclusion Thijs Gillioen
!96
session to be useful. Aside from practical reasons, such as helping students find a
focus, students indicated the mentoring sessions was helpful to them because they
were (re)motivated to write, because the mentor could explain some concepts they
had not yet fully grasped or because they felt empowered by mentors’ non-directive
approach. Other students offered indirect indicators of programme efficacy by
voluntarily attending multiple mentoring sessions, or by giving their future peers the
advice to attend a mentoring session. In addition to these short-term effects, some
signs of long(er)-term writer development were identified.
Based on these data, a number of additional recommendations for programme
improvements were suggested. The first of these posed that the strict adherence to
the non-directive approach may need to be abandoned in favour of an approach that
mixes the non-directive and directive approaches (as suggested by, for example,
Crasborn et al. (2011)). By, for example, allowing ‘experienced’ mentors to go beyond
non-directive support, more students could be reached, and more students would feel
‘helped’ by their mentor. The second recommendation was to incorporate a
mentoring session into the regular writing classes. This could have the effect of
introducing students to the concept and workings of a mentoring session in a (more)
familiar environment, thus alleviating the anxiety felt by many students when first
approaching a writing mentor (who is, ultimately, a stranger); this could, ideally,
result in students feeling more inclined to schedule a session with a mentor of their
own accord. Finally, the third recommendation suggested mandating attendance of a
mentoring session earlier on in the semester. This would, again, allow for mentors to
be introduced to students earlier, in addition to opening up opportunities for long-
term guidance of students by mentors.
Conclusion Thijs Gillioen
!97
Based on the data analysed, it is argued that, by implementing the suggestions
detailed in this paper, fundamental improvements to both short-term and long-term
programme efficacy can be made, as such increasing the programme’s ability to guide
students during their evolution as writers. Through doing so, this project aspires to
have provided a meaningful contribution to the evolution of the UGent writing
mentoring programme, which, although no longer entirely new, is still growing and
learning how it can optimally meet students’ (and mentors’) needs. It also hopes to
have made a valuable contribution to the field of research concerning peer and
writing mentoring, a field that is still open to much potential research.
3 Suggestions for Future Research
Having summarised the current research, this final section will suggest a number of
avenues for future research in the field of peer mentoring. The UGent writing
mentoring programme, like the ETTV courses within which it largely functions,
constantly evolves and adapts to meet both the changing circumstances as well as the
student audiences it tries to reach. Therefore, it would be improvident to assume that
the programme can reach its end goal merely by implementing the suggestions made
above. More research, more AR cycles, will be needed to re-evaluate and reassess the
programme’s efficacy, and make proposals for further improvements. The
suggestions for such research, below, are not intended to be exhaustive: they are
merely indications of where future research projects might lead.
During the literature overview, this paper summarised the ongoing discussion
between proponents of the non-directive approach, and proponents of a ‘mixed
approach’ (that incorporates both non-directive and directive methodologies). This
discussion has been referred to at several points, and could be an interesting avenue
Conclusion Thijs Gillioen
!98
for future research. Such research could examine audio recordings of mentoring
sessions, and establish if both directive and non-directive mentoring are happening,
and, if so, when and why mentors decide to transition from one approach to the
other. It would be interesting, then, to see which approach is appreciated more by
students, and if a combined approach results in more text improvements. Data for
this point have already been collected, but analysis could not be pursued due to
limitations of time and scope. However, future research projects could pick up this
thread and provide further investigation, thus adding to the ongoing debate in this
field, and potentially uncovering further ways to improve the programme’s efficacy.
A second option unexplored in the current setting is, as noted above, the
perspective of other stakeholders in the mentoring programme. Analysing the points
of view of mentors could reveal ways for further programme optimisations. Do
mentors believe themselves to be efficacious? Do mentors feel like the extra training,
mentioned above, is warranted? And do mentors have other suggestions that could
facilitate the programme’s workings? All of these aspects could be the subject of
further inquiry. Additionally, other stakeholders such as teachers who use essays for
assessment, or the English Studies Group as a whole, could offer additional
perspectives on the programme. These “outsider perspectives” could reveal valuable
information or suggestions that “insiders” may not have thought of; likewise, their
opinions on the programme may help integrate the writing mentoring programme
into more courses, thus again allowing mentors to continue guiding students beyond
the confines of the ETTV courses.
Thirdly, future examination could investigate the applicability of the
mentoring programme’s style and methodology to other languages offered in its
current setting. In other words, this investigation could assess the cross-language
Conclusion Thijs Gillioen
!99
applicability of mentoring, examining if writing mentors can be of help to students
studying languages other than English, and, if so, what adaptations would have to be
made to conform to other languages’ writing standards or needs. Such research would
not require additional mentors, since most members of the current group are already
enrolled in another language (in addition to English). As an added benefit, this could
help integrate the programme in non-English language courses.
Finally, this paper will advocate the need for so-called “slow research” (Lillis
2010). The current research has conducted an evaluation of the programme’s efficacy
at one point in time. For the programme to evolve, conducting a thorough
investigation that includes the perspectives listed above, a longitudinal study
spanning multiple AR cycles would be ideal.
Such a longitudinal timeframe opens up a number of possibilities for research
that are, otherwise, not possible. The first pertains to writer development, which is,
by nature, a slow process that spans decades (Kellogg 2008: 2). Therefore, while this
paper has found limited evidence of writer development, any claims about students’
development as writers can only be truly substantiated by extensive data gathered
over the course of at least several writing assignments, if not multiple years.
Secondly, such long(er)-term inquiries enable the Action Researcher to conduct
repeated reflective cycles. This would allow for more fine-grained analysis and
suggestions than can be achieved within one AR cycle. Thirdly, if other stakeholders
would find it beneficial to expand the programme to languages beyond English,
investigations into its progress and efficacy would require multiple AR cycles. Finally,
it is not unimportant to note that the opportunity to conduct such longitudinal
research, within the Belgian context, is unique to UGent: there are currently no other
institutes (of higher education) that make use of peer mentoring (and peer tutoring)
Conclusion Thijs Gillioen
!100
to the extent that the current mentoring programme does. This offers Ghent
University a unique position, on the one hand, to conduct research that can
potentially fill (a) gap(s) within this field; on the other hand, to maintain its forefront
position in Belgium in this regard by advancing the programme through further
research.
For these reasons, the current paper argues that longitudinal research is
optimal for the continuing improvement and refinement of the peer writing
mentoring programme. Regardless of whether or not such longitudinal research is
possible, the programme will continue to require investigation and adaptation, in
order for it, and its mentors, to continue succeeding in their goal to guide students in
their development as writers.
Conclusion Thijs Gillioen
!101
V References
Archer, Arlene. 2008. Investigating the effect of writing centre interventions on
student writing. Reprinted in Arlene Archer & Rose Richards (eds.), 2011.
Changing Spaces. Writing Centres and Access to Higher Education, 131-146.
Stellenbosch: AFRICAN SUN MeDIA.
Banerjee, Jayanti, Xun Yan, Mark Chapman & Heather Elliott. 2015. Keeping up with
the times: Revising and refreshing a rating scale. Assessing Writing 26. 5-19.
Berckmoes, Dirk & Hilde Rombouts. 2009. Intern rapport verkennend onderzoek
naar knelpunten taalvaardigheid in het hoger onderwijs. Antwerpen:
Universiteit Antwerpen/Linguapolis.
Best, Karen, Laura Jones-Katz, Bailey Smolarek, Marie Stolzenburg & Derek
Williamson. 2015. Listening to our students: An exploratory practice study of
ESL writing students’ views of feedback. TESOL Journal 6(2). 332-357.
Bloom, B.S. (ed.), M.D. Engelhart,, E.J. Furst, W.H. Hill & D.R. Krathwohl. 1956.
Taxonomy of educational objectives, handbook I: The cognitive domain.
New York: David McKay Co Inc.
Bromley, Pamela, Eliana Schonberg & Kara Northway. 2015. Student perceptions of
intellectual engagement in the writing center: Cognitive challenge, tutor
involvement, and productive sessions. Writing Lab Newsletter 39(7-8). 1-6.
Brooks, Jeff. 1991. Minimalist tutoring: Making the student do all the work.
Writing Lab Newsletter 15(6). 1-4.
Brown, Ruth E. 2001. The process of community-building in distance learning
classes. The Journal of Asynchronous Learning Networks 5(2). 18-35.
References Thijs Gillioen
!102
Campbell, Toni A & David E. Campbell. 1997. Faculty/student mentor program:
Effects on academic performance and retention. Research in Higher
Education 38(6). 727-742.
Çevik, Yasemin Demiraslan. 2015. Assessor or assessee? Investigating the differential
effects of online peer assessment roles in the development of students’
problem-solving skills. Computers in Human Behavior 52. 250-258.
Cho, Kwangsu & Charles MacArthur. 2010. Student revision with peer and expert
reviewing. Learning and Instruction 20(4). 328-338.
Chraif, Michaela. 2015. Correlative study between academic satisfaction, workload
and level of academic stress at 3rd grade students at psychology. Procedia -
Social and Behavioral Sciences 203. 419-424.
Council of Europe. 2001. Common European Framework of reference for
languages: Learning, teaching, assessment. Cambridge University Press /
Council of Europe.
Crasborn, Frank, Paul Hennissen, Niels Brouwer, Fred Korthagen & Theo Bergen.
2011. Exploring a two-dimensional model of mentor teacher roles in
mentoring dialogues. Teaching and Teacher Education 27(2). 320-331.
Davies, Alan & Catherine Elder. 2008. Handbook of applied linguistics. New York:
John Wiley & Sons.
Dawson, Philip. 2014. Beyond a definition: Toward a framework for designing and
specifying mentoring models. Educational Researcher 43(3). 137-145.
Deane, Paul. 2013. On the relation between automated essay scoring and modern
views of the writing construct. Assessing Writing 18(1). 7-24.
References Thijs Gillioen
!103
De Soete, Alexander. 2016. Analyzing Reader Engagement in NPO Newsletters: Is
Haas’ Reader Engagement Framework Applicable to Dutch NPO
Newsletters?. Ghent: Ghent University BA paper.
De Wachter, Lieve & Jordi Heeren. 2012. TaalVaardig aan de STart. Gerichte
ondersteuning van academische taalvaardigheid aan de KU Leuven.
Proceedings of the 2012 “Van Schools tot Scriptie” Colloquium. 55-68.
Domínguez, César, Arturo Jaime, Ana Sánchez, José Miguel Blanco & Jónathan
Heras. 2016. A comparative analysis of the consistency and difference among
online self-, peer-, external- and instructor-assessments: The competitive
effect. Computers in Human Behavior 60. 112-120.
DuBois, David L., Bruce E. Holloway, Jeffrey C. Valentine & Harris Cooper. 2002.
Effectiveness of mentoring programs for youth: A meta-analytic review.
American Journal of Community Psychology 30(2). 157-197.
English, Rebecca & Jennifer Duncan-Howell. 2008. Facebook© goes to college:
using social networking tools to support students undertaking teaching
practicum. MERLOT Journal of Online Learning and Teaching 4(4). 596-601.
Ferris, Dana R. 2016. Promoting grammar and language development in the
writing class. Why, what, how and when. In Eli Hinkel (ed.), Teaching
english grammar to speakers of other languages, 222-246. New York:
Routledge.
Flemish Government. 2016. Secundair onderwijs - Derde graad ASO - Moderne
vreemde talen Frans-Engels - Eindtermen. http://www.ond.vlaanderen.be/
curriculum/secundair-onderwijs/derde-graad/aso/vakgebonden/moderne-
vreemde-talen-frans-engels/eindtermen.htm. (1 May, 2016.)
References Thijs Gillioen
!104
Gielen, Sarah, Elien Peeters, Filip Dochy, Patrick Onghena & Katrien Struyven. 2010.
Improving the effectiveness of peer feedback for learning. Learning and
Instruction 20(4). 304-315.
Girgensohn, Katrin. 2012. Mutual growing: How student experience can shape
writing centers. Journal of Academic Writing 2(1). 127-137.
GO! Onderwijs van de Vlaamse Gemeenschap. 2014. Leerplan ASO - 3e graad -
Basisvorming AV Engels. http://pro.g-o.be/blog/documents/2014-014.pdf. (1
May, 2016).
Gopee, Neil & Mary Deane. 2013. Strategies for successful academic writing —
Institutional and non-institutional support for students. Nurse Education
Today 33(12). 1624-1631.
Haas, Sarah. 2009. Writers’ groups for MA ESOL students: collaboratively
constructing a model of the writing process. English Language Teacher
Education and Development 12. 23-30.
Haas, Sarah. September 15, 2013. “Peer Mentoring for first year students”. Personal
e-mail correspondence with author and other potential mentors.
Haas, Sarah. 2014. ETTVI lecture slides [PowerPoint slides]. Retrieved from
http://minerva.ugent.be/main/course_home/course_home.php?
cidReq=A00286802014. (3 March, 2016).
Haas, Sarah. 2016. Understanding reader engagement: A framework for teaching
and assessing writing. [manuscript].
Harbord, John. 2003. Minimalist tutoring - An exportable model? The Writing Lab
Newsletter 28(4). 1-5.
References Thijs Gillioen
!105
Harrington, Kathy, Peter O’Neill & Savita Bakhshi. 2007. Writing Mentors and the
writing centre: producing integrated disciplinary writers. Investigations into
university teaching and learning 4(2). 26-32.
Hattie, John. 2009. Visible Learning. A synthesis of over 800 meta-analyses
relating to achievement. Milton Park, Oxon: Routledge.
Hepplestone, Stuart, Graham Holden, Brian Irwin, Helen J. Parkin & Louise Thorpe.
Using technology to encourage student engagement with feedback: a literature
review. Research in Learning Technology 19(2). 117-127.
Hoon, Tan Bee. 2009. Assessing the efficacy of writing centres: A review of selected
evaluation studies. Pertanika Journal of Social Sciences & Humanities 17(2).
47-54.
Hutchings, Cathy. 2006. Reaching students: lessons from a writing centre. Higher
Education and Development 25(3). 247-261.
Iraji, Hamid Reza, Mostafa Janebi Enayat & Mahyar Momeni. 2016. The effects of
self- and peer-assessment on Iranian EFL learners' argumentative writing
performance. Theory and Practice in Language Studies 6(4). 716-722.
Irvin, L. Lennie. 2014. What a difference three tutoring sessions make: Early
reports of efficacy from a young writing center. The Writing Lab Newsletter
39(1-2). 1-5.
Jones, Ian & Chris Wheadon. 2015. Peer assessment using comparative and absolute
judgement. Studies in Educational Evaluation 47. 93-101.
Karcher, Michael J., Gabriel P. Kuperminc, Sharon G. Portwood, Cynthia L. Sipe &
Andrea S. Taylor. 2006. Mentoring programs: A framework to inform program
development, research, and evaluation. Journal of Community Psychology
34(6). 709-725.
References Thijs Gillioen
!106
Kellogg, Ronald T. & Bascom A. Raulerson III. 2007. Improving the writing skills of
students. Psychonomic bulletin & review 14(2). 237-242.
Kellogg, Ronald T. 2008. Training writing skills: A cognitive developmental
perspective. Journal of Writing Research 1(1). 1-26.
Kullman, John. 1998. Mentoring and the development of reflective practice: concepts
and context. System 26(4). 471-484.
Lewin, Kurt. 1946. Action research and minority problems. Journal of Social Issues
2(4). 34-46.
Li Gwet, Kilem. 2012. Handbook of inter-rater reliability. Third edition. The
definitive guide to measuring the extent of agreement among raters.
Gaithersburg, MD: Advanced Analytics, LLC.
Lillis, Theresa. 2010. The value of local research for sustaining writing development
in higher education: the case of ‘academic literacies’. WDHE [Writer
Development in Higher Education] Conference June 2010. Retrieved from
http://www.writenow.ac.uk/news-events/wdhe-conference-2010/conference-
presentations/monday-28-june-2010/#Keynote1. (10 May, 2016).
Liu, Ngar-Fun & David Carless. 2006. Peer feedback: the learning element of peer
assessment. Teaching in Higher Education 11(3). 279-290.
Martinez, Christy Teranishi, Ned Kock & Jeffrey Cass. 2011. Pain and pleasure in
short essay writing: Factors predicting university students’ writing anxiety and
writing self-efficacy. Journal of Adolescent & Adult Literacy 54(5). 351-360.
Min, Hui-Tzu. 2006. The effects of trained peer review on EFL students’
revision types and writing quality. Journal of Second Language Writing 15(2).
118-141.
References Thijs Gillioen
!107
Murphy, Tim. 1998. Motivating with near peer role models. The Japan Association
for Language Teaching: On JALT’ 97: Trends & Transitions. 201-206.
Orland-Barak, Lily & Ronit Hasin. 2010. Exemplary mentors’ perspectives towards
mentoring across mentoring contexts: Lessons from collective case studies.
Teaching and Teacher Education 26(3). 427-437.
Panadero, Ernesto, Margarida Romero & Jan-Willem Strijbos. 2013. The impact of a
rubric and friendship on peer assessment: Effects on construct validity,
performance, and perceptions of fairness and comfort. Studies in Educational
Evaluation 39(4). 195-203.
Park, Soo-Kyung. 2015. The interplay of task, rating scale, and rater background in
the assessment of Korean EFL students’ writing. English Teaching 70(2).
55-82.
Pleasant, Scott. 2015. It’s not just beans anymore; It’s our bread and butter. Writing
Lab Newsletter 39(9-10). 7-10.
Reason, Peter & Hillary Bradbury (eds.). 2001. Handbook of action research.
London: Sage Publications.
Richards, Jack C. & Richard W. Schmidt. 2013. Longman dictionary of language
teaching and applied linguistics. London: Routledge.
Rodger, Susan & Paul F. Tremblay. 2003. The effects of a peer mentoring program
on academic success among first year university students. The Canadian
Journal of Higher Education 33(3). 1-18.
Ruiz-Gallardo, José Reyes, José L. González-Geraldo & Santiago Castaño. 2016. What
are our students doing? Workload, time allocation and time management in
PBL instruction. A case study in Science Education. Teaching and Teacher
Education 53. 51-62.
References Thijs Gillioen
!108
Sagor, Richard. 2000. Guiding school improvement with Action Research.
Alexandria: Association for Supervision and Curriculum Development.
Salinitri, Geri. 2005. The effects of formal mentoring on the retention rates for
first-year, low achieving students. Canadian Journal of Education 28(4).
853-873.
Saliu, Sokol. 2005. Constrained subjective assessment of student learning. Journal
of Science Education and Technology 14(3). 271-284.
Salomon, Gavriel. 1993. On the nature of pedagogic computer tools: The case of the
writing partner. In Susanne P. Lajoie & Sharon J. Derry (eds.), Computers as
cognitive tools. 179-196. Hillsdale, NJ: Lawrence Erlbaum Associates.
Schmidt, Katherine M. & Joel E. Alexander. 2012. The empirical development of an
instrument to measure writerly self-efficacy in writing centers. The Journal
of Writing Assessment 5(1). pp. n.a.
Shermis, Mark D. 2014. State-of-the-art automated essay scoring: Competition,
results, and future directions from a United States demonstration. Assessed
Writing 20. 53-76.
Stevenson, Marie & Aek Phakiti. 2014. The effects of computer-generated feedback on
the quality of writing. Assessing Writing 19. 51-65.
Topping, Keith & Stewart Ehly. 1998. Peer-assisted learning. New York: Routledge.
Vista, Alvin, Esther Care & Patrick Griffin. 2015. A new approach towards marking
large-scale complex assignments: Developing a distributed marking system
that uses an automatically scaffolding and rubric-targeted interface for guided
peer-review. Assessing Writing 24. 1-15.
Vlaams Verbond van het Katholiek Secundair Onderwijs (VVKSO). 2014. Engels -
Derde graad ASO - studierichtingen met component moderne talen: leerplan
References Thijs Gillioen
!109
secundair onderwijs. http://ond.vvkso-ict.com/leerplannen/doc/
Engels-2014-003.pdf. (1 May, 2016).
Warschauer, Mark & Douglas Grimes. 2008. Automated writing assessment in the
classroom. Pedagogies: An International Journal 3. 22-36.
Weigle, Sara Cushing. 2013. English language learners and automated scoring of
essays: Critical considerations. Assessing Writing 18(1). 85-99.
Zepke, Nick & Linda Leach. 2010. Student engagement: Ten proposals for action.
Active Learning in Higher Education 11(3). 167-177.
Zundert, Marjo van, Dominique Sluijsmans & Jeroen van Merriënboer. 2010.
Effective peer assessment processes: Research findings and future directions.
Learning and Interaction 20(4). 270-279.
References Thijs Gillioen
!110
VI Appendices
Overview
Appendix 1 ETTVI Scribing 3 Assignment pp. 111 - 116
Appendix 2 Overview of General Responsibilities pp. 117
Appendix 3 Analysed Student Essays pp. 118 - 213
Appendix 4 Samples from Reflective Writing pp. 214 - 239
Appendices: Overview Thijs Gillioen
Because the appendices to this paper contain samples of student writing and student
reflective writing, they were not made available for open access, so as to avoid any privacy
concerns.
The appendices can be requested for consultation at [email protected].