Roles, functions and evaluative language in PhD examiners’ reports Professor Brian PaltridgeA/Professor Sue StarfieldDr Robert McMurtrie

Outline • The project and our role

• Part 1

• Methods, resources, tools, frameworks and theories• SFL lens: APPRAISAL, TRANSITIVITY

• Part 2• 10 roles

• institution, supervisor, expert, reporter, peer, viva examiner, evaluator, editor, commentator and examiner

• not isomorphic v. clustering

• Final comments

Roles[T]he real role of the examiner is to judge whether the student has mastered appropriate indeterminate skills and displayed the right indeterminate qualities. (Delamont et al. 2000, 41, emphasis added)

Resources from FOCUS sharpen and soften the boundaries between things, grading something that is inherently non-gradable (Martin and Rose, 2007: 46)

Previous research : function/roles

• Previous research on the language of examiners’ reports• Praising, hedging (Hyland 1998), criticising, sugar

coating the pill (Hyland and Hyland 2001)

• Previous research on the roles examiners adopt• Responsibilities, duties, job descriptions • Teacher, proof-reader, facilitator, gatekeeper,

evaluator, knowledge provider and reader (Hyland and Hyland 2001)• “schizophrenic” role of the teacher when marking

assignments Leki (1990: 59)

Our contribution:

• Identifying various roles• Making the correlation between the linguistic features

and the roles• TRANSITIVITY = what is being evaluated • APPRAISAL framework to see how “the what” is being

appraised/evaluated • Synthesis of experiential and interpersonal

APPRAISAL

APPRAISAL

APPRAISAL framework • AFFECT: how we feel

• dis-satisfaction• un-happiness• in-security

• JUDGEMENT: how we evaluate the behaviour of another person• normalcy• capacity• tenacity• veracity• Propriety

• APPRECIATION: how we evaluate an object • reaction

• impact• quality

• composition• balance• complexity

• valuation

Transitivity frameworkThe student has performed an in-depth study.

Actor

Process:material Goal

The findings were evaluated critically.

Goal Process:material

Circumstance

The thesis is a fine example of a good PhD.

Carrier

Process:relational: attributive:intensive

Attribute

I like the engagement with Popper.

Senser Process:mental

Phenomenon

Role of the PhD thesis

A successful PhD thesis will demonstrate the candidate’s ability to conduct original research and to present the findings of that research to a professional standard

Token Process:relational:intensive

Value

Criteria• Does the thesis comprise a coherent investigation of the chosen topic?

(APPRECIATION:composition:balance)

• Does the thesis deal with a topic of sufficient range and depth to meet the requirements of the degree? (APPRECIATION:valuation:standard:depth)

• Does the thesis make an original contribution to knowledge in its field (APPRECIATION:valuation:standard:original) and contain material suitable for publication in an appropriate academic journal ? (APPRECIATION:valuation:standard:publishable)

• Does the thesis meet internationally recognised standards for the conduct and presentation of research in the field? (APPRECIATION:valuation:standard/propriety APPRECIATION:reaction:quality/composition:complexity)

• Does the thesis demonstrate both a thorough knowledge of the literature relevant to its subject and general field? (APPRECIATION:valuation:capacity:cognition) and the candidate's ability to exercise critical and analytical judgement of that literature? (covert JUDGEMENT:capacity)

• Does the thesis display mastery of appropriate methodology and/or theoretical material? (APPRECIATION:valuation:standard:appropriate)

APPRAISAL from criteria

• +ve/-ve APPRECIATION:valuation

• +ve/-ve APPRECIATION:composition:balance

• +ve/-ve APPRECIATION:composition:complexity

• +ve/-ve APPRECIATION:reaction:quality

• +ve/-ve APPRECIATION:reaction:impact

APPRAISAL

Transitivity

The candidate has presented the thesis generally well

Actor Process:material Goal Circumstance Circumstance

The thesis is generally presented well

Goal Circumstance Process:material

Circumstance

The candidate has presented the thesis generally well

Actor Process:material Goal Circumstance Circumstance

JUDGEMENT v. COVERT JUDGEMENT

• APPRECIATION JUDGEMENT COVERT JUDGEMENT

This study is well-presented (+ve APPRECIATION: reaction:quality)

The candidate has presented the thesis well (+ve JUDGEMENT:capacity).

The thesis is generally presented well (+ve COVERT JUDGEMENT:capacity)

This thesis fails to justify its claims (-ve APPRECIATION:valuation)

The candidate has failed to substantiate her claims (-ve JUDGEMENT:capacity)

The failure to substantiate her claims … (-ve COVERT JUDGEMENT:capacity)

APPRAISAL

ENGAGEMENT

• Monoglossic – no room for negotiation (prescriptive)• Heteroglossic – room for negotiation• Other voices – reporting, quoting and paraphrasing, modality • Related to stance, voice, position, hedging

Register (style)• Holbrook et al (2012, 3, emphasis added) ‘[e]xaminers write

their report in a sequence and style that reflects their discipline, their unique interests and expertise’

• Johnston (1997, 336, emphasis added) ‘showed a wide variation in style’

• Register • Field What is going on (participants and processes)• Tenor What are the social roles of the participants• Mode How is the discourse presented (as formal written or causal

conversation)

Unit of information: clause

• Overview: This is an original, well thought out, methodical, detailed and concise PhD thesis

• 5x APPRECIATION (one role – examiner)

• The nature of this colour variation is interesting and important

• 2x APPRECIATION (two roles – commentator, evaluator)

Method Clause Passiv

e Positive/ negative

Role APPRECIATION

JUDGEMENT

COVERT JUDGEMENT

AFFECT

COVERT AFFECT

Overview: This is an original, well thought out, methodical, detailed and concise PhD thesis [[that addresses a poorly understood topic]].

positive

examiner

√

The results are fascinating positive

commentator

√

√

and are a major contribution to our understanding of fragmentation in Surtseyan eruptions.

positive

examiner

√

X has clearly put a big effort into this positive

evaluator

√

and she should be congratulated for a super piece of work.

positive

evaluator

√

√

I recommend that the thesis be accepted and the degree awarded after amendments have been made

positive

examiner

√

Roles

Examiner

examiner

The “real” role of the examiner

• APPRECIATION:• Impact: quality• Composition: balance/complexity• Valuation

However, because the thesis and the candidate are in a Token/Valued relationship:

• COVERT JUDGEMENT • Which is APPRECIATION: normalcy/capacity/tenacity/

veracity/propriety • Passive voice• Nominalisation

• Field: recommendations, award, degree, emendations• I recommend that the thesis be accepted and the degree awarded after

amendments have been made

Roles

Institution• Institution: Does the thesis make an original contribution to

knowledge in the field and contain material suitable for publication in an appropriate academic journal?

• Examiner: Yes

Roles

Supervisor • “I felt throughout reading the thesis that my comments on the

work were more akin to those of a supervisor rather than that of an examiner.”

• The actual role of supervisor: • formative feedback (Holbrook et al 2004, 9)• Gap in knowledge and how to close the gap and improve the

thesis.

• Dialogic (Kumar and Stracke 2007, 467).

• Tenor: close, friendly

Supervisor• Why do you think this is the case? Cf. Why is this the case?• … but can you say a little more about why you prefer it and did

not use the EM algorithm?• You might also consider some of the earlier work of Valière• A useful start is his 2002 JBR paper (for an overview see

Dixon, Donald F. (2002) "Emerging Macromarketing Concepts: From Socrates to Alfred Marshall." Journal of Business Research, 55 (2), 87-95)

• You should consider stating here that this piece of research focuses on this subtype

• I would make (even) clearer where this emerges from, and take full credit.

Comparisons Examiner Supervisor

The candidate needs to explain exactly what these numbers represent

You might also consider some of the earlier work of Valière

Page 44. (Do) As with Chapter, voucher specimens should be lodged in a National Collection and voucher numbers given

Page 77. (Consider) Could there be some thermoregulation function to retention of wings but loss of flight?

Roles

Editor• Prescriptive• x should be y• P14 loose should be lose

• Usually at the end of the report • P60 "to small" should be "too small" • P64 expression in the sentence "In winter 2006..." • p71 "an person" should be "a person" • p73 expression "There have been maximum..." • P76 describe above should be described above • P78 128 input station should be stations • P89 expression "The outgoing solar ..." • P94 Fig 4.2 is referred to before Fig 4.1 • P97 expression "the ELA was with 1810..." • P97 expression 2000 msl has an massive

Roles

Reporter• Synoptic overview of• Thesis Chapters Sections

• This thesis reports the results of what is essentially a single study with boys with ADHD and a control group of normally developing boys

• Uses verbal processes, quotes, paraphrases• “These organic compounds are known as siderophores." This sentence suggests that all Fe

(III)-binding compounds in seawater are siderophores.• Monoglossic

• Several well-known motifs (such as the 5' UTR TATA box) are low complexity elements

• To position her or himself in relation to the content• Heteroglossic contracting – closing space down for alternatives

• The author has demonstrated that putative toxin/antitoxin pairs can act as such on when found on the chromosomes of M. smegmatis…

• Heteroglossic expanding – distancing formulation• In particular, it is claimed (p. 56) that the research in the thesis is based on the work of Tripp and Alsop

(1999, 2001), but the details of those studies and their theoretical impact is not fully explained (p.44).

Roles

Peer• Encouragement/improvement beyond the confines of the

thesis

• Field: publish, paper, articles

• Tenor: close relationship; modality: should, could

• X should consider publishing the introduction, in a shortened version, as a review paper

• Some of the suggestions listed are identified as not necessary for the thesis but desirable prior to publication.

Roles

Evaluator• Evaluation beyond the criteria:

• APPRECIATION and JUDGEMENT (overt)

• No intrusion on the message, i.e., “I” • She is extremely passionate about blogging (+ve

JUDGEMENT:normalcy) • This is an early study opening up a very complex system

Roles

Expert• “Examiners should be suitably qualified to undertake the task.

Suitably qualified examiners: preferably have a PhD (or if not, some other higher degree but with appropriate research experience at a high level.”

• Authoritative declaratives• Also direct mass balance measurements cannot a priori claim to

exactly catch the end of the ablation season.

• Call into doubt their expertise• The statistical analysis seems sound, although I am not an expert

in statistics.

Expert• Examiners exert their knowledge in the field in question and in

terms of the genre/conventions of a thesis.• Claims must be evidence based• A strong structure allows readers to know where to look for

particular information and makes the reader more confident that all important information has been included…

• Distinguishing between reporter and expert when there is no attribution:• Page 15 Free rotation of carbons adjacent to single bond in

lycopene polyene carbon chain occurs only in those bonds which are not conjugated…

Imperatives / roles Editor Examiner Expert

Pg. 19 - line 14— change 'the' to 'these' and 'to' to 'in’

… and write in full sentences, explaining these observations

17. (2.1.2, Page 11, section 3, last line): Add e.g. "weathering rates and geomorphological activity", as the amount of rock walls alone are not decisive

Roles

Commentator • According to the OED, a comment is ‘a remark expressing an opinion or

reaction […] especially of a critical nature’.

• Not prescriptive, formative or summative

• Explicit intrusion on the message “I”

• Formulaic expressions, attitude markers, personal musings • What on earth has Donald Rumsfeld to do with marketing (quoted twice).• I wonder if reciprocal blasts were used,…

• AFFECT, COVERT AFFECT • I worry (-ve AFFECT:insecurity) about your mention of analysing relations

among concepts• This is an interesting (+ve APPRECIATION:reaction:impact) […] thesis • This thesis interested me (+ve COVERT AFFECT:satisfaction)

Realised as grammar: Third conditional

3rd v. 2nd conditional/rolesCommentator supervisor

I assume that they are the extracted cells, this could have been explained in the methods (or perhaps I have missed it).

You could explain that exchange value is only a limited form of value creation

Also, more detail on how specific toxins act should have been given.

You should consider stating here that this piece of research focuses on this subtype;

It might have been useful here to compare the results obtained to those recently reported by Nauseef et al, and Palazzolo-Balance et al. ….

You might also consider some of the earlier work of Valière.

Other multimodal realisations

• From the perspective of future work – 3rd conditional is formative = peer role

• Marked punctuation• Very interesting!!• P55. What is DRE??• I'm curious -???

Roles

Viva examiner• Actual primary role: to ask questions at the viva.• The role ↘ interrogative (technical term F^S)• Prepare for the viva voce. • Many types of questions – not all viva examiners• Field: • Thus, these questions are largely unanswered in the thesis and

are exactly the questions I would like to discuss with the candidate during an oral exam.

Different types of questionsViva examiner Supervisor Commentator how was this controlled for? Normal human neonatal foreskin

or adult skin? Page 40, para 2: laboratory Page 51, para 1: is there really an apostrophe after its?

what difference could this make? 2.5, p48, end of para 1: Should it be 'bone marrow cells' rather than 'white blood cells'?

What on earth has Donald Rumsfeld to do with marketing (quoted twice).

Page 67. (Clarify) What is a geomorphological region? Why were cell types chosen? How likely is this to relate to real cancer? Were all arrays exposed on the same piece of x-ray film?

Are these separated by major rivers, faults or mountain ranges?

4. Can the candidate outline/ suggest from the studies and their findings some quite direct applications in the joint flight crew and flight attendant training activities that would contribute to safety and efficiencies?

Can you speculate on how a particular type of siderophore would respond to such a pH change based on their acid base chemistry?

…could you have used another site with a longer term record, such as Craigieburn, to test the hypothesis that the ERA dataset is better than the Tait?

FINAL REMARKS1. The various roles and linguistic features are not isomorphic; rather roles are ↘ by a

clustering of features.2. The complexity of interpreting the evaluation when there are a number of roles to

grapple3. As Sadler (2010: 537) states, ‘students cannot convert feedback statements into

actions for improvement without sufficient working knowledge of some fundamental concepts’

4. A candidate might ask: • Which comments do I need to act upon to get this thesis passed?• Which comments might I consider in order to improve the thesis?• Which comments can I return to later when considering publications?• Which comments can I simply disregard?”

5. Are all roles necessary/appropriate?6. Can some comments be reworded so that they are given by a more authoritative role

(examiner, editor)7. Should there be a feedback rubric that ties the comments in with APPRAISAL and

TRANSITIVITY. 8. Can discourse analysts collaborate with educators to create a metalanguage to share

amongst examiners, candidates, and institutions.

Is a more robust PhD assessment regime actually possible?

Reference List• Bednarek, M. (2008). Emotion Talk Across Corpora. New York: Palgrave MacMillan.• Delamont, S., Atkinson, P. and Parry, O. (2000). The Doctoral Experience: Success and Failure in

Graduate School. Rouledge-Falmer: London and New York • Hyland, K. 1998. Hedging in scientific research articles. Amsterdam: Benjamins.• Hyland, F. and Hyland, K. 2001. Sugaring the pill: Praise and criticism in written feedback. Journal of

Second Language Writing 10: 185—212. • Johnston, S. (1997a). Examining the examiners: An analysis of examiners’ reports on doctoral

theses, Studies in Higher Education, 22, 3: 333--347.• Kress, G. (2010). Multimodality: A Social Semiotic Approach to Contemporary Communication.

London and New York: Routledge. • Kumar V. and Stracke, E. (2007). An analysis of written feedback on PhD thesis. Teaching in Higher

Education 12(4): 461—470.• Kumar V. and Stracke, E. (2011). Examiners’ reports on theses: Feedback or assessment? Journal of

English for Academic Purposes 10: 21--222. • Leki, I. (1990). Coaching form the margins: Issues in written response. In B. Kroll (ed.), Second

Language Writing: Research Insights for the Classroom, 57—68. Cambridge: Cambridge University Press.

• Martin, J.R. and Rose, D. 2007. Working with Discourse: Meaning beyond the Clause. London: Continuum.

• Martin, J.R. and White, P.R.R. (2005). The language of evaluation: Appraisal in English. New York: Palgrave Macmillan.

Further research• Genre analysis• Stages• Roles in stages• Clustering of roles• Micro structures

• More transitivity analysis

AFFECT

JUDGEMENT

APPRECIATION

Our role • Project• Examination of examiners’ reports• Linguistic perspective • Language of evaluation, roles and function