1111

Assessment of theses at Assessment of theses at masters and PhD levelmasters and PhD level

Presentation 2Presentation 210 November 201710 November 2017Erasmus+ LOAF Project, Erasmus+ LOAF Project, Vilnius, Lithuania Vilnius, Lithuania

Dr Declan Kennedy, Dr Declan Kennedy, Department of Education, Department of Education, University College Cork, Ireland. University College Cork, Ireland.

2222

1.1. Introduction to Introduction to research theses.research theses.

2.2. Process for evaluating Process for evaluating research theses. research theses.

3. What do examiners look for when they assess a thesis?

4. Examples of rubrics to help us when assessing theses.

33

Introduction

“Thesis student” (“candidate”): any Research Higher Degree student writing a Master’s, PhD or other doctoral thesis or dissertation. “Supervisor”: an academic whose official role is to provide research advice and guidance for a thesis student. “Thesis examiner”: an academic who reads the

completed thesis and gives a report recommending a result.

44

There are now two basic types of doctoral degrees in addition to the degrees in medicine and law; the PhD (which is the research doctorate and the most common form), and the professional doctorate, of which there are more than 20 different degrees, such as the Doctor of Education (EdD) and the Doctor of Business Administration (DBA) (Nerad 2008). The procedures for assessment of these types of doctoral degrees are basically the same.

(Kyvik, 2014)

55

Assessment of thesesProcesses for the assessment of PhD theses and the examination of doctoral candidates differ considerably across countries (Powell and Green 2007) in terms of:The composition of the evaluation committeeWhether the examination and defence take place in public or private. Whether the committee can award the degree on the condition of amendments or revisions being made to the thesis. Whether pass or fail is the only alternative. (Powell and Green 2007, Svein, 2014)

66

The League of European Research Universities issued recommended guidelines for doctoral training in which the assessment of theses is mentioned (LERU 2007):

“The evaluation of the thesis manuscript should be the responsibility of a dissertation committee. The committee should where possible not include the supervisors and must have at least one member from another university, either national or foreign. Final examinations must consist of an in-depth oral discussion with a panel of reputed scholars in which the candidate ‘defends’ her/his thesis.”

77

The composition of the evaluation committee

The number of members varies between five (the USA), three (Sweden and Norway) and two (Ireland, UK).In the USA, all the members (which constitute the PhD committee) normally come from the same university as the PhD candidateIn the UK one of the members must be from another universityIn Sweden it is often the case, and in Norway it is normally the case, for one of the members to come from abroad.In Lithuania, “the Doctoral Studies Committee forms a Dissertation Defence Board composed of the chairman appointed by the Committee and 4 members.”

Note: Examination normally consists of examination of thesis and viva voce (oral examination) . However, in Australia, a thesis examination typically consists of two or three examiners reading the thesis and writing a report recommending a result. In New Zealand, a viva voce or oral examination is used in addition to examiners reading the thesis.

88

“The status of the thesis when submitted for assessment also varies across countries. While in the USA and the UK the examiners may require the candidate to undertake smaller amendments or larger revisions of his or her thesis before its finalapproval (and publication), in Sweden and Norway the committee must either pass or fail the thesis.”

(Kyvik, 2014)

99

Examination of thesesMost thesis students are mystified by the process of examination (e.g. Johnston, 1997). This is contrary to good assessment practice (e.g. Biggs and Tang, 2011, Brown and Knight, 1994)Often thesis students have no idea what examiners look for in a research masters or doctoral (PhD) theses. Thesis is submitted and then some time later comes back

“As supervisors and examiners, we are less mystified by how a written thesis is examined, but much of the examination process is still hidden behind closed doors and I do not have access to what actually happens. My knowledge of what examiners do, like that of most supervisors, is based on personal experience of examining theses, observing how my thesis students were examined, anecdotes about the examination process, and ideal versions of what a thesis should be and what an examiner should do” (Golding et al, 2014)

1010

“This sort of personal knowledge tends to be partial and incomplete (Holbrook et al. 2007). I have sent thesis students through the examination gauntlet, but is their experience typical? I know how I examine a thesis, but am I typical? I know how a thesis should be examined, but are they actually examined this way? As Denicolo (2003) pointed out, there can be a disturbing disparity between what supervisors expect and what examiners implement. So, what do examiners tend to do as they examine a thesis? Are there any general trends?” - (Golding et al, 2014)

1111

“Defending” the PhD!

The PhD originated in Germany (Simpson, 1983) – the doctoral thesis represented an independent scientific contribution which had to be defended publicly.

1212

Purpose of viva voce (oral examination)Survey of views (Tinkler and Jackson (2004)

The viva allows the examiners to check the candidates’understanding and ability to produce and present research to PhD standard (36%);It clarifies areas of weakness (32%);It ensures authenticity (31%);It allows the examiners to further develop the candidate’s ideas and to pro- vide advice on publication (25%);It checks that the candidate can ‘defend’ her/his thesis (24%);It enables the examiners to test the candidate on his/her knowledge of the broader literature (22%);It allows the examiners to test the candidates’ oral skills (11%);It can be a site of final decision-making in borderline cases (11%); andIt acts as a ‘rite of passage’ (6%).

1313

The viva (oral examination)

Research shows that even when a viva is part of the assessment, most examiners have already made their judgement before the oral presentation on the basis of reading the thesis, and the oral is merely to confirm their assessment (Denicolo, 2003; Jackson and Tinkler 2001, Kyvik 2013; Trafford 2003). Hence, the key aspect of the assessment is what examiners do as they read a thesis (Golding et al., 2014)

1414

What do examiners look for when they assess a thesis?

1515

1. Examiners tend to be broadly consistent in assessment of theses

Examiner practice is broadly consistent despite differences in institutional instructions (Holbrook et al. 2007), discipline (Bourke, Hattie, and Anderson 2004; Lovat, Holbrook, and Bourke 2008), nationality (Pitkethly and Prosser 1995), level of degree (Bourke and Holbrook 2013), or the experience of the examiner(Kiley and Mullins 2004).

1616

Where a number of examiners are involved in assessing a thesis, Holbrook et al. (2008) found examiners gave consistent recommendations for 96% of the 804 theses they studied. An inconsistency was defined as one or more examiners recommending accept or accept with minor revisions, while one or more examiners recommended revise and resubmit or fail. Even when the recommendations were inconsistent, the content of their reports, what the examiners wrote about, tended to be the same (Holbrook et al. 2008).Inconsistent examiners tend to offer consistent recommendations if they have a chance to negotiate in a thesis examination committee or after viewing the oral presentation of a thesis student.

1717

“Because it is rare for examiners to give wildly inconsistent recommendations, you also need not worry about ‘the examiner from hell’ who gives a radically divergent judgement from the other examiners, or who recommends your thesis fail for a seemingly crazy reason. These are rare exceptions, memorable only because they are extreme. All your examiners are likely to be looking for similar qualities in your thesis, and they are all likely to give a consistent recommendation to pass (though they may disagree about whether minor or major changes are needed) (Holbrook, Bourke, Lovat, & Fairbairn, 2008; Lovitts, 2007).”

(Golding, 2017)

1818

2. Examiners expect a thesis to pass

Examiners begin reading with curiosity and enthusiasm, expecting a thesis to be good and ‘hoping to find their task rewarding and enjoyable’ (Johnston 1997). They know years of effort has gone into a thesis, and it has been judged worthy by supervisors (or at least passable); so, they anticipate it will pass, and even want it to pass (Kiley and Mullins 2004).

1919

Examiners are reluctant to fail a thesis. They consider a thesis a pass until it is proven a fail, and will only consider failing a thesis if there are significant errors or omissions that threaten the credibility of the research (Holbrook et al. 2004).Even then, examiners can get upset if they have to recommend a fail or even a resubmit.Less than 1% of examiners recommend a fail (Lovat, Holbrook, and Bourke 2008).

2020

3. Examiners judge a thesis by the end of the first or second chapter

Examiners make an initial judgement about the quality of a thesis early in their reading, at least in the first two chapters and sometimes while they skim the abstract, table of contents, introduction and conclusion.They quickly decide whether the thesis is likely to be hard work or an enjoyable read, a ‘treat’ or an ‘endurance test’ (Carter, 2008).The introduction and the literature review are particularly important in a thesis - does the candidate knows what they are doing?If examiners have a good first impression, they feel they can relax and enjoy the thesis; if not, then they read more critically, looking for problems.

2121

4. Examiners read a thesis as an academic reader and also as a normal reader

A thesis should be an enjoyable read in order to sustain an examiner’s good impression. Examiners take a lot of time and effort to assess a thesis, and if they are not enjoying the read, they are more likely to judge it to be poor quality. Examiners may read in the evenings when they are tired (Carter 2008), and can take from two to four full days across two to three weeks (Carter 2008; Mullins and Kiley 2002). Some examiners even read a thesis several times.For a thesis to be a good read, it must meet examiner’s academic expectations and their expectations as ‘normal’ readers (Johnston 1997). They expect to read a thesis that is academically sound – for example, engages with the literature, and has an appropriate methodology and a logical structureThey also expect a thesis to be clear, interesting, polished andeasy to follow.

2222

Examiners expect reader-friendly writing; they want the candidate to make the thesis clear for the examiner, to give arguments that will convince the examiner and to explain the approach, conclusions and significance of the research in ways that the examiner will understand.(Golding et al., 2014)

2323

5. Examiners are irritated and distracted by presentation errors

Examiners express a ‘sense of relief and pleasure, or even surprise, when a well-presented thesis was encountered’(Johnston 1997).Examiners become distracted, irritated, frustrated and agitated by frequent presentation errors that could have been fixed easilyA few errors are inevitable.Examiners take into account if English is not the first languageof the candidate. Examiners will be annoyed by normal presentation errors such as typographical, grammatical, spelling and formatting errors or inconsistencies, as well as unclear writing; Examiners irritated by academic presentation errors such as an incorrect or inconsistent bibliography or citations, references that are omitted or misreported, incorrect publication dates or misspelled names, and citations and quotations that are misused.

2424

Thesis students might think that presentation errors are unimportant, merely editorial, because it is only the research that counts.If there are frequent typographical or spelling errors, or if the formatting is unclear and inconsistent, examiners lose confidence in the thesis, doubt the quality of the research and conclusions, and start to question the overall quality of the research.If the candidate makes errors with something easy like spelling, then what sorts of errors were they making in their data collection, analysis, synthesis, conclusions, etc.?Sloppy presentation is seen as a sign of deeper problems, a sign of a poor thesis and can cause examiners to dig deeper – the presentation problems can distract from the ideas presented in the thesis.

2525

6. Examiners like a coherent thesisCoherent writing has focus, flow, and a logical and explicit structure that integrates and connects the various parts of the thesis and gives clear directions for the reader. Coherent writing often includes signposts and summaries that indicate what is coming and what has been done. Examiners reading a thesis in chunks over a few weeks when they may have forgotten Chapter 1 by the time they read Chapter 4.

2626

Tools to create a coherent thesisMeta-text is where you write about what you are writing (Mauranen 1993).- ‘This chapter will build on the previous chapter by …’- or ‘I will start with discussing x, then show how this links

to y …’Meta-text can function like signposts in your writing: words and phrases that indicate where you have been, where you are and where you are going next.- At the chapter level signposts indicate the relationship between different sections and different chapters. - include phrases like: ‘My conclusion is … ’-, ‘There are three main reasons …’- ‘When we combine the conclusions from chapter 4 with these findings, we can now conclude …’

(Golding 2017)

2727

Other signposts in a thesisSummaries and previews: Provide previews, overviews, summaries and reviews at the beginning and the end of sections and chapters. These help to remind the thesis examiner of their place in the overall thesis.Referring back and forwards: Another way to help orientate the reader is to refer backwards and forward in your thesis. - ‘I will expand on this description in chapter 7’, - Building on the example I introduced in chapter 3 …’Repetition of main ideas help to create a coherent thesis. - Word-for-word copying would be tedious, but your reader may also have forgotten important points raised several chapters earlier, so it may be useful to paraphrase them in the current chapter. - Emphasise your main point by telling the same story several times throughout your thesis at different levels of elaboration –the very short version in the abstract, a longer version in the introduction, the very long version throughout the thesis and then a different, shorter summary in the conclusion. (Golding 2017)

2828

Thesis examiners also equate coherence with a convincing argument. A thesis is crafted into a coherent whole by threading an argument through it for the examiners to follow.Connecting the research question with an answer.Connecting various subsidiary conclusions and connecting these conclusions with the supporting data, evidence and with the background literature.Without coherence, a thesis appears badly written and unstructured and can cause confusion to the examiners.

2929

In line with their academic expectations, thesis examiners also equate coherence with a convincing argument.A thesis is crafted into a coherent whole by threading an argument through it for the examiners to follow – connecting the research question with an answer, connecting various subsidiary conclusions and connecting these conclusions with the supporting data, evidence and reasons, and with the background literature.

3030

7. Examiners favour a thesis that engages with the literature

Examiners look for a thesis that is situated in a relevant body of literatureDemonstrates an accurate and comprehensive understanding of this literature. The thesis student must explain what has already been written on their topic but it is not enough to merely describe, list or cover the relevant literature. Examiners want a thesis candidate to analyse, interpret, categorise, order or apply the literature in order to draw new insights and conclusions.

3131

Students must “tell the story” - the student engages with the literature by giving his/her version of this story, telling it in his/her own words in the literature review.Display critical engagement with the literature, rather than a list of who said what. Signs of critical engagement - interpret, conceptualise, analyse or evaluate what has been written, supported with appropriate references and identify controversy, disagreement and ambiguity.

3232

Making maximum use of the literature

Examiners want to read a thesis where the candidate has used the literature to argue that: Their interpretation of the field is accurate.Their research question and approach are worthwhile and appropriate. Address a significant gap in the literature.Their results and conclusions make an important contribution to this literature.

3333

Holbrook et al. suggest that examiners tend to use the literature review as a ‘litmus test’ for the quality of a thesis as a whole. A pass requires coverage and a working understanding of the literature; but, for a good thesis, a thesis candidate must critically engage with the literature (Holbrook et al. 2004b).

3434

8. Examiners favour a thesis with a convincing research design

Examiners, especially experienced examiners, tend to be open to different research approaches, (Mullins and Kiley 2002).The candidate must convince the examiners that the approach they adopt is appropriate. A well-explained and justified approach, with clear acknowledgements of strengths and possible limitations, is very important.

3535

Examiners need to see a clear research question and a method or research design that is appropriate for addressing this.In some disciplines (e.g. physics, chemistry or

other sciences) there is a standard, assumed approach, and a candidate only needs to explain their approach if it deviates from this norm.In other disciplines where a variety of

approaches are available, the candidate always has to explain and justify their choice of research methodology.The research approach described must match with what was actually done by the candidate.

3636

9. Examiners favour a thesis that engages with the findings

The thesis must engage with the findings rather than merely listing what was found. The candidate must :Interpret, analyse and critically appraise their findings.Draw conclusions from them and show the implicationsfor the research question or problem (Holbrook et al. 2004c).

Examiners look for:Overstated, misrepresented or misreported findings,Unconvincing analyses or interpretations

Conclusions that go beyond what the findings can support.

3737

Examiners want candidates to critically discuss how their findings connect with the literature. Which studies do the findings support ?Which do they discredit or undermine? How do the findings contribute to or advance the literature? What further research is now warranted?

3838

10. Examiners require a thesis be publishable

Examiners require a passable thesis to be publishable research. This is the key criterion that examiners use to assess theses regardless of the institution, country or discipline (Holbrook et al. 2004a). A passable thesis must be original, have ample scope and significance, and make a contribution or have an impact on the field, discipline or practice.Although publishable work is often equated with original work, original research does not have to be ground breaking, unprecedented or paradigm shifting.Examiners expect a thesis to add something to a field, but they do not require the definitive contribution that transforms the field.

3939

What is an “original contribution”?

An original contribution might be:Opening up a new area of research.Reframing an old issue, e.g. introducing a new method, theoretical framework or concept, Applying established methods, theories and concepts in new areas;Gathering new data which lead to new findingsand conclusions, Providing a novel interpretation or synthesis of established data, theories or conclusions (Winter, Griffith, and Green 2000).

4040

Examiners judge how original and publishable a thesis is by comparing it to the research that has already been done in the field. This is one main reason why examiners expect a thesis to engage with the literature so that the candidate can show that their thesis contributes to a fieldand is publishable in that field.Mullins and Kiley (2002) found that half of the 30 examiners that they interviewed were favourably influenced when the thesis they were examining included material already published Some examiners said they were not swayed by the inclusion of published material in a thesis, because an examiner should make their own judgements without influence (Bourke, Holbrook, and Lovat2005; Johnston 1997).

4141

11. Examiners give summative and formative feedback

Examiner main job is to assess the quality of a thesis, Examiners offer constructive, formative feedback about how to improve.Examiners tend to see themselves as gatekeepers, upholding the standards for a master’s or doctorate.As such, they give summative comments, like a finalgrade or evaluation: ‘This is a fine thesis’, ‘This is a well-executed literature review’, ‘the analysis is deficient’, “your main conclusions are not clearly stated”, (Holbrook et al. 2004a) If a thesis meets their expectations, they give praise; but if it does not, examiners offer advice on how to improve it.

4242

Examiners also see themselves as teachers, supervisors or mentors, aiming to assist a candidate by offering formative comments such as instruction, advice and guidance. Examiners treat a written thesis as a work-in-progress, and give instruction and advice regardless of the quality of the thesis.If the thesis has met their expectations, they offer advice about future publications and research. Examiners offer lots of advice about how a candidate can progress and develop as a researcher, even if the thesis is of the highest quality (Hansford and Maxwell 1993; Lovat et al. 2008). Examiners are more eager to offer this constructive feedback than they are to condemn the inadequacies of a thesis (Lovat et al. 2008). More than a quarter of every examiner report is devoted to formative or instructive comments (Holbrook et al. 2004c; Holbrook et al. 2007).

4343

What is the difference between assessment of a masters thesis and a PhD thesis?

Although originality tends to be an explicit criterion for examining a PhD thesis, it is not a stated criterion for passing a master’s thesis.Nevertheless, examiners still expect a master’s thesis to

make some lesser contribution to the literature: “this lack of an expectation of originality in masters theses has been maintained institutionally …… while masters’theses might contribute to knowledge in the discipline, there were limitations in that these new researchers needed to align themselves with the research community and achieve a level of ‘research-mindedness’. However, the view was also put that the best masters dissertations would be original but that originality was not essential at the masters level.” (Bourke and Holbrook 2013).

4444

4545

“With respect to the importance of the 12 quality indicators, it is clear that examiners generally adopted the same criteria for PhD and research masters theses, in that all indicators were considered to be important at both levels.”

(Bourke and Holbrook 2013)

4646

Examples of rubrics to help us when assessing theses

4747

https://cms.bsu.edu/-/media/www/departmentalcontent/fcs/pdfs/rubicforthesis.pdf?la=en

48484848

Rubric for Assessing Masters Thesis

(continued on next slide)

49494949

Rubric Masters thesis (continued)

http://uwf.edu/media/university-of-west-florida/offices/cutla/documents/MS-Thesis-Rubric.pdf

5050

https://www.ufs.ac.za/docs/default-source/regulations-documents/rubric-masters-dissertation-1004-eng.pdf

5151

5252

5353

Rubric for evaluating progress on doctoral work

http://sph.umd.edu/sites/default/files//files/EPIB/EPIB%20PhD%20Rubric%20comps_prop%20def_defense%208_18_14.pdf

5454(continued)

5555(continued)

5656

http://cee.umd.edu/sites/default/files/documents/phd-ms_rubric-form_may2013.pdf

5757(continued)

5858www.ciis.edu/Documents/Administration/AVP/.../PhD%20Dissertation%20Rubric.pdf

5959

6060

6161

6262

6363

6464

6565https://pharmacy.osu.edu/sites/default/files/forms/future-students/grad/Rubric%20for%20Evaluating%20PhD%20Dissertation%20and%20Defense.pdf

6666

6767

6868

Learning Outcomes

Discuss the areas in which examiners tend to be broadly consistent when assessing a thesis.Plan how we can make these criteria explicit to our students to help them approach the examination in a more informed way.Evaluate modify and develop rubrics to help in the assessment of research theses.

6969

REFERENCESBiggs, J., and C. Tang. 2011. Teaching for Quality Learning at University. 4th ed. New York: McGraw Hill.Bourke, S. 2007. “PhD Thesis Quality.” South African Journal of Higher Education 21 (8): 1042–1052.Bourke, S., and A. Holbrook. 2013. “Examining PhD and Research Masters Theses.” Assessment & Evaluation in Higher Education 38 (4): 407–416.Bourke, S., A. Holbrook, and T. Lovat. 2005. “Using examiner reports to identify quality in PhD theses.” AARE Conference: Quality in Educational Research, Cairns.Brown, S., and P. Knight. 1994. Assessing Learners in Higher Education. London: Kogan Page.Carter, S. 2008. “Examining the Doctoral Thesis.” Innovations in Education and Teaching International 45 (4): 365–374.Denicolo, P. 2003. “Assessing the PhD.” Quality Assurance in Education 11 (2): 84–91.Golding, C, et al. (2014) What examiners do: what thesis students should know, Assessment & Evaluation in Higher Education, 39:5, 563-576

7070

Golding, C (2017) Advice for writing a thesis (based on what examiners do), Open Review of Educational Research, 4:1, 46-60 Holbrook, A. 2007. “Levels of Success in the Use of the Literature in a Doctorate.”South African Journal of Higher Education 21 (8): 1020–1041.Holbrook, A., S. Bourke, H. Fairbairn, and T. Lovat. 2007. “Examiner Comment on the Literature Review in PhD Theses.” Studies in Higher Education 32 (3): 337–356.Holbrook, A., S. Bourke, H. Fairbairn, and T. Lovat. 2012. “The Focus and Substance of Formative Comment Provided by PhD Examiners.” Studies in Higher Education: 1–18 (ifirst). doi:10.1080/03075079.2012.750289.Holbrook, A., S. Bourke, T. Lovat, and K. Dally. 2004a. “Qualities and Characteristics in the Written Reports of Doctoral Thesis Examiners.” Australian Journal of Educational and Developmental Psychology 4: 126–152.Holbrook, A., S. Bourke, T. Lovat, and K. Dally. 2004b. “PhD Theses at the Margin.”Melbourne Studies in Education 45 (1): 89–115.Holbrook, A., S. Bourke, T. Lovat, and K. Dally.2004c. “Investigating PhD Thesis Examination Reports.” International Journal of Educational Research 41 (2): 98–120.Holbrook, A., S. Bourke, T. Lovat, and H. Fairbairn. 2008. “Consistency and Inconsistency in PhD Thesis Examination.” Australian Journal of Education 52 (1): 36–48.Holbrook, A., J. St George, L. Ashburn, A. Graham, and M. Lawry. 2006. “Assessment Practice in Fine Art Higher Degrees.” Media International Australia incorporating Culture and Policy 118: 86–97.

7171

Jackson, C., and P. Tinkler. 2001. “Back to Basics: A Consideration of the Purposes of the PhD.” Assessment & Evaluation in Higher Education 26 (4): 355–366.*Johnston, S. 1997. “Examining the Examiners: An Analysis of Examiners’Reports on Doctoral Theses.” Studies in Higher Education 22 (3): 333–347.*Kiley, M., and G. Mullins. 2004. “Examining the Examiners: How Inexperienced Examin- ers Approach the Assessment of Research Theses.”International Journal of Educational Research 41 (2): 121–135.Kyvik, S. 2013. “Assessment Procedures of Norwegian PhD Theses as Viewed by Examiners from the USA, the UK and Sweden.” Assessment & Evaluation in Higher Education: 1– 14 (pre-published online). doi:10.1080/02602938.2013.798395.LERU. 2007. Doctoral Studies in Europe: Excellence in Researcher Training. Leuven: LERU.*Lovat, T., A. Holbrook, and S. Bourke. 2008. “Ways of Knowing in Doctoral Examination: How Well is the Doctoral Regime?” Educational Research Review 3 (1): 66–76.*Lovat, T., M. Monfries, and K. Morrison. 2004. “Ways of Knowing and Power Discourse in Doctoral Examination.” International Journal of Educational Research 41 (2): 163–177.Lovitts, B. (2007). Making the implicit explicit: Creating performance expectations for the dissertation. Sterling, VA: Stylus.

7272

Mauranen, A. (1993). Contrastive ESP rhetoric. English for Specific Purposes, 12, 3–22.Pitkethly, A., and M. Prosser. 1995. “Examiners’ Comments on the International Context of PhD Theses.” In Research into Higher Education, edited by Beattie, McNaught, 129–136. Melbourne: HERDSA.Powell, S., and H. Green, eds. 2007. The Doctorate Worldwide. Maidenhead: Open Univer- sity Press.Simpson, R. 1983. How the PhD Came to Britain. A Century of Struggle for Postgraduate Education. Guildford: Society for Research into Higher Education.Svein, Kyrik (2014), Assessment procedures of Norweigan Phdtheses as viewed by examiners from the USA, the UK and Sweden. Assessment and Evaluation in Higher Education, 39, 2 140 – 153. Trafford, V. 2003. “Questions in Doctoral Vivas.” Quality Assurance in Education 11 (2): 113–121.Winter, R., M. Griffith, and K. Green. 2000. “The ‘Academic’Qualities of Practice.” Studies in Higher Education 25 (1): 25–37.