Assessment and Examinations

John KirbyGraduate School

Faculty of Medical Sciences

University of Newcastle

Prof. John Kirby, the faculty postgraduate tutor, is responsible for the assessment process of the faculty’s postgraduate students, looking at all of the assessment reports that come into the graduate school.This presentation will firstly cover assessment and then the examination process.

Assessment– Why?

• Monitor student progress• Identify problems with

– Project– Supervisor(s)-student relationship

• Completion rate– ‘exit’ strategy for struggling students– Student re-registration

Assessments are required by funding bodies but even without this requirement it is essential that students progress is monitored. This allows the identification of problems hopefully at an early stage. This can help identify problems with projects, as although projects are assessed at the start this is not a guarantee of success. Problems with students and their relationships with their supervisors can also be identified. A major concern for this and all graduate schools is the completion rate, students submitting a thesis on time. Late submission (over 4 years for a usual PhD student) is classed as a fail as far as the University is concerned.

The assessment process allows an exit strategy for students who are struggling with the academic rigour of a research degree. In the instance of a PhD this will allow students with difficulties to write up for an MPhil, though students can at this point leave with nothing and that could well be the best thing for that person. Few assessors are prepared to tick the assessment boxes that will allow for an exit, greater use of these boxes may help avoid problems later in the project. This allows for student registration, students register for a year at a time.

Assessment

• When?

Focussing on the 3 year PhD studentships. The assessment is done at 9 months in order that a positive outcome can be achieved, before students are registered for their second year. It is necessary to know how a student is shaping up before the end of the first year.The second year assessment is less academically rigid, but shows the project is on time, but the final year assessment, ideally at 36 months, should give an idea of how the thesis is shaping. There is, of course the option for a fourth year for writing up.

Assessment

• What?– First assessment

Concentrating now on the first year assessment, the students are expected to produce a document – as shown, including a table of contents, a comprehensive literature review (it is probably now will they will discover what their project actually is). The students need to have a clear idea of the aims – their aims, not the supervisors - a summary of the results to date, which may not be much in the way of solid output, but they should be showing the potential to create methodologies and to be able to talk coherently about their research tools. They crucially then need to discuss their strengths and weaknesses of their research and where they are going.The limit is 7,500 words, which we hope students will be able to do in combination with their day-to-day work.

Assessment

• What?– Subsequent assessments

There has to be a positive response to each assessment for the student to re-register for the following year.Subsequent assessments are shorter, only a few pages in length, and should be used to monitor the student and project, ensuring that everything is developing as expected by student and supervisor. The final assessment should be a comprehensive thesis plan, giving the scope of results, or some guidance of when the experiments are likely to be completed, to the assessment panel.

–Final assessment to include a detailed thesis plan

The assessment report form was designed for the entire University so may not be entirely appropriate for this faculty, but is recognised throughout.The student is expected to complete a progress report, in their own words, and the supervisor is allowed to make comments on the student’s progress, but can also indicate how well the student is doing. For training, the Graduate School provides the infrastructure for research skills and training.Comments can be made on the use of English by international students.The progress recommendation is important, as it informs the Graduate School of the ability of the student to progress to the next year.

Please use these panels - a tick in a box is poor returnfor the student’s effort.

If the student has an unsatisfactory progress, a further assessment may be done within 2 months of the assessment – this 2 months is crucial, as the student needs to register for the following year. It may be necessary to recommend for an MPhil or even a termination of the studentship.Please fill in the panels as indicated – it means that the student gets relevant feedback in return for their efforts

Using this checklist may not be applicable to this faculty, but gives some guidance to the assessors Using this checklist may not be applicable to this faculty, but gives some guidance to the assessors

Purpose of an examination

• For the University– To assess and maintain quality– To mark ‘completion’ of the degree programme

• For the Student– Potentially leads to award of a degree– Is an important and memorable life event

• can be a real emotional roller coaster (for everyone involved)

The focus here is mainly on PhD examination, but it is very similar with MD or MPhil.Why an examination? For the university it is an assessment and maintenance of the quality of the research students, and to mark the completion of the research programme – whether or not the student is going to get the degree they have applied for.For the student it potentially leads to the award of a degree, and on the human side of things, it is a memorable life event. It can be an emotional time for both students and supervisors!

Who needs an oral examination?• All PhD and MD candidates

– Students and full-time RAs need one internal and one external examiner

– Staff candidates require two external examiners and an internal ‘moderator’

• Not all MPhil candidates– Same criteria for examiners as above– Oral examination held at examiners request

• Not only for weak students • With a good candidate can be fun for everyone!

Bearing in mind that regulations vary from University to University, rules may differ as internal and external examiners. All PhD and MD candidates require an oral examination. Students and full time Research Assistants both need one internal and one external examiner, whereas staff candidates require two external examiners, plus an internal ‘moderator’, who doesn’t need to know anything about the submitted thesis, but must ensure that the external examiners are familiarised with how Newcastle proceeds with examinations.Not all MPhil candidates require an oral examination, the thesis may be submitted alone, but examiners may want to meet the candidate. An MPhil must not be treated as a failed PhD student – they may have had considerable time restraints and therefore may be quite a strong candidate.

Appointment of examiners

• See nomination forms on Graduate School http://medical.faculty.ncl.ac.uk/postgraduate/internal/staff.html– Completed by supervisor and Head of school/Institute

• Examiners must:– be cognisant of standards– have subject knowledge (need CV)– command authority– not have played a role in the research– (if external) not have been a member of Newcastle staff for

at least 3 years– be able and willing to examine

The nomination forms are available from the Graduate School Office, or at the website -http://medical.faculty.ncl.ac.uk/postgraduate/internal/staff.html, and must be completed by the supervisor and Head of School or Institute.Examiners must be aware of the standard required of the exam, must have detailed knowledge of the subject i.e. be an expert in their field, research active, publishing good research. They must command authority, and must not have played a role in the research, or co-authored papers. External examiners must not have been a member of Newcastle University staff for at least 3 years.Finally, the examiner must be willing and able to examine – it is not necessary to take on board every thesis.

Internal examiners

• Liaise with supervisor, student (and sometimes external) to arrange examination

• Ensure the examination complies with Newcastle University protocol

• Ensure appropriate report forms are completed and submitted in a timely manner

• Provide balance, fairness and ensure good examination conduct

• Potentially play a role during any appeals process

Internal examiners for a standard PhD have to liaise with supervisors (who play a major role), student, and sometimes the external examiner, to arrange the examination. They have to ensure that the exam complies with the Newcastle University protocol, and must ensure that the report forms are completed and submitted, particularly the joint report form. They must ensure that the exam is completed fairly and in an appropriate manner. They also potentially play a role in any appeals process, so the consequences have to be considered.

External examiners

• Usually is the scientific specialist• Is an experienced examiner

– Chicken and egg….• Often though to have the ‘casting vote’

– But there are procedures for disagreement• Maintains inter-university quality

– Report provides important feedback to Graduate School (and QA etc)

• May not be completely ‘up to speed’ with local regulations!

External examiners are usually the specific scientific specialists, and are generally experienced, typically examining previously as internal examiner. Almost always the examiners will agree, with the external examiner having perhaps the more important view, but there are procedures in place for any disagreement. The external examiner hopefully maintains inter-university standards, and provides feedback on the training that the candidate has had.They may not be totally aware of the local regulations of the University, and it is helpful to be aware that each university may have totally different procedures for examination.

What if 2 external examiners?

• Need to appoint an internal moderator• This person need not read or understand the thesis• Role is to provide advice on Newcastle examination process

– May be required to present records of the exam if the result is questioned (appeal process).

The moderator is appointed where two external examiners are required. They do not need to read or understand the thesis, but are generally given the role of letting the external examiners know how Newcastle operates its examination policies, and may be required to provide details of the examination records in the appeals process.

Criteria – all theses

• Should be:– Authentic– Scholarly– Professional– Well-structured, written and presented

These criteria are from the regulations, and are self-evident.

MPhil candidates

• Should– Demonstrate advanced knowledge– Have good knowledge of literature

• Theses need not– Demonstrate consistent originality– Be worthy of publication

MPhil candidates should demonstrate an advanced knowledge of their subject, with a good comprehension of literature.The thesis does not need to demonstrate consistent originality, nor be worthy of publication, although quite often the MPhil thesis will contain considerable material suitable for submission for publication.

PhD/MD candidates

• Should – provide evidence of adequate industry– demonstrate ability for originality– understand relationship with wider field– thesis should contain material worthy of publication

PhD/MD candidates should prove that they have worked sufficiently well to be able to provide an in-depth knowledge of their subject, demonstrating their originality, and it is up to the examiner to find how wide their knowledge is of their given subject, which may not be apparent in the thesis. The thesis itself should contain material worthy of publication, giving the students a drive to produce published papers on their way to the end of their PhD, as no examiner can ever say no to that, prompting a drive to the publication of data.

Types of thesis

• ‘Standard’– Divided into chapters with results and

interpretations

• By publication– A review and a series of ~5 related papers– Can be difficult to examine as papers have already satisfied

external referees!– Staff candidates only

There are generally two types of thesis, the first of which is ‘standard’, a bound copy divided into chapters with results, etc, and the second, which is becoming more popular, by publication. Only staff candidates can submit their thesis by publication, which typically consists of a written review and a succession of around 5 related papers. It should be clear that the student has contributed significantly to work in the publications - something which the author must make clear. It is very difficult to examine this type of thesis, as external referees have already been satisfied that the papers are of a good standard, but one thing an examiner can do is make sure that the student fully understands the contents of the publication.

Useful WWW sites:

• ncl.ac.uk/spo/examhand.html– (examiners handbook)

• ncl.ac.uk/calendar/volume1/…– (conventions, regulations etc)

Any problems with these links can be resolved by going to the Graduate School Website, where all forms and documents mentioned can be downloaded.

There is no word limit for a thesis here at Newcastle, but the limit for the abstract should be around 300 words, ideally on one page.

Reading a Thesis

• Are you a proof reader or a scientist?• You will need to provide a list of corrections if you require them• I usually stick post-it notes to the margin to localise my

questions within the thesis – Be sensitive though; hundreds of these can look very scary!

Some academics are more critical of the spelling mistakes a student might have made, than of the concepts and opinions in the work, but clearly if it doesn’t make sense and the English is appalling, it is up to the examiner to recommend a rewrite. Where a typographical errors are made, it may be necessary to provide a list of corrections. A lot of examiners now use post-its to mark any points they’d like to raise , but remember it may be daunting for the student if you use a lot!

Preliminary Report

• Regulations vary between institutions -read them!• Many (but not all) institutions require examiners to

independently produce reports before the examination– Some need these to be submitted (well) before the

examination to flag up potential problems

The examiners should write a preliminary report before the oral examination. This does not involve ringing the other examiner, as this is a procedural irregularity- for your own protection, please follow the rules! In Newcastle, the report is requested 2 weeks before the exam, but it does not always follow – many submit only days before the exam.

How to conduct the examination - 1

• Arrange the room– Often good to have pencils and paper to draw on

• At the start candidates can be very nervous!– Put them at their ease if possible with a soft start (but don’t

anticipate the result!)– Remember to arrange refreshment breaks– Consider the candidate’s bladder

A lot of advice is given in the guidance notes, but in general it is a good idea to arrange the room and try to put the student at ease. A good opening question is to ask them what they are most proud of in their thesis – this is probably the piece they are happiest with and most comfortable to talk about. Do not anticipate the result of the examination. It is also recommended to take breaks for refreshment or a toilet visit.

How to conduct the examination - 2

• Agree a plan with your co-examiner• Remember:

– Oral examination of a good candidate can/should be one of the most pleasurable academic experiences for all involved

– Examination of a poor thesis/candidate can be truly awful!

• You need to devise different strategies for both situations

Before you start, agree a plan with your co-examiner to work out how you are going to proceed. Devise strategies for different situations, as the candidate may be very nervous. If the candidate gives a good answer, it is not a problem to say so, and if they run into difficulties it may be wise to have a break to give them a chance to recover their equilibrium.

RESEARCH DEGREE EXAMINATION JOINT REPORT OF EXAMINERS

Student Candidates for PhD

Name of Candidate:

Title of Thesis:

Name and Address of External Examiner:

Name of Internal Examiner:

Date of Oral Examination: SECTION A - RECOMMENDATIONS (tick the relevant box to indicate overall recommendation)

The Candidate be admitted to the degree

(a)(i) That the candidate be admitted immediately to the degree of Doctor of Philosophy.

(a)(ii) That the candidate be admitted to the degree of Doctor of Philosophy subject to minor corrections of the text made to the satisfaction of the internal examiner, normally within a period of one month of receiving formal notification of the corrections to be made.

(a)(iii) That the candidate be admitted to the degree of Doctor of Philosophy subject to minor revisions being made to the satisfaction of the internal examiner, normally within a period of six months of receiving formal notification of the revisions to be made.

The Candidate be permitted to resubmit for the degree

(b)(i) That the candidate’s thesis be deemed to be of a satisfactory standard, but that the candidate be adjudged to have failed to satisfy the examiners in the oral examination and that the candidate therefore be required to attend within six months either for a second oral examination or for a written examination, as the examiners shall determine in their written report.

(b)(ii) That the candidate be adjudged to have failed to satisfy the examiners and the candidate be permitted to revise and re-submit the thesis within twelve months without a further oral examination.

(b)(iii) That the candidate be adjudged to have failed to satisfy the examiners and the candidate be permitted to revise and re-submit the thesis within twelve months and be re-examined orally.

The Candidate be recommended for the Masters degree

(c)(i) That the candidate has reached the standard required for the appropriate Master’s Degree and should immediately be awarded that degree instead.

(c)(ii) That the candidate has reached the standard required for the appropriate Master’s Degree and should be awarded that degree instead subject to minor corrections of the text made to the satisfaction of the internal examiner, normally within a period of one month of receiving formal notification of the corrections to be made.

(c)(iii) That the candidate has reached the standard required for the appropriate Master’s Degree and should be awarded that degree instead subject to minor revisions being made to the satisfaction of the internal examiner, normally within a period of six months of receiving formal notification of the revisions to be made.

The Candidate be permitted to resubmit for the Masters degree

(d) That the candidate be permitted to revise and re-submit the thesis for the appropriate Master’s Degree within twelve months and be re-examined orally if the examiners so require by indication in their written report.

The Candidate be adjudged to have failed to satify the examiners

e) That no degree be awarded and that the candidate be adjudged to have failed.

This is the latest joint report form for Newcastle University. The top 3 boxes - a(i) – the degree is awarded, even if there are minor typographical corrections, though this isn’t clear from the for. a(ii) – minor corrections to be made to the satisfaction of the internal examiner a(iii) – minor revisions to be made to the satisfaction of the internal examiner. B becomes a fail – but it is not irretrievable. B(i) is typically chosen if the candidate was incapable of giving a satisfactory oral examination, particularly if they were nervous, the other 2 give varying options for resubmission.

University of London

A. Pass, subject to minor amendments to be completed and checked by one or both examiners within 3 months

B. Not pass, but candidate allowed to re-write the thesis and resubmit it within 18 months for examination by the same examiners (oral exam may not need to be held)

C. etc….

Different universities have different report forms, as you can see from this, so careful consideration should be given in your recommendations

At the end (if all has gone well)

• Ask the candidate to withdraw for a few minutes• REMEMBER you do not award the degree!!• Work out what you wish to say, then invite the candidate back• Tell the candidate what recommendation you will be making to

the higher degrees committee (or other appropriate authority).• Smile and shake hands

If all goes well, ask the candidate to leave for a few minutes, decide what you wish to say, as you must remember you are not personally awarding the degree, then invite the candidate back. You can then tell the candidate of your recommendations to the appropriate authority.

At the end (if it has NOT gone well) - 1

• Make absolutely sure you know what your options are (read the regulations)

• You are not obliged to tell the candidate anything (although you will feel some pressure to do so)– the candidate will receive written confirmation of the outcome in due

course

• You may wish to speak to the supervisor• You might need clarification of the regulations (Graduate

School)

If the examination has not gone well, make sure you know what your options are. You are not obliged to tell the candidate on the day, but can let them know that they will receive written information in due course. You may wish to contact the supervisor to find out where things have gone wrong, and you may need clarification of the regulations from the Graduate School so you know that what you propose is right.

At the end (if it has NOT gone well) - 2

• You will need to fill in the report form with very comprehensive details of any changes you require– This outcome results in much more effort in the future for the

examiners!• All report forms look different.

– Make sure you know the precise significance of a tick in every optional box!

– For example, ticking box 3 (pass) at Newcastle can produce a very different outcome from box 3 in London (fail)!

The report form, which again will be different from university to university, needs to be filled in with very comprehensive information of any changes you require and both examiners’ preliminary reports will be looked at.

The appeal process

www.ncl.ac.uk/spo/AppealsForm.pdf

If things go particularly badly, the candidate may follow the appeal process. It is very hard to fail a student, as Universities often come down on the side of the student.

Invitation to examine

• This is not necessarily an honour• Why you and not someone else?

– Are you a ‘soft touch’?– Are you the supervisor’s best friend?

• You will/should see the abstract of the thesis at the time of invitation

• You have the right to REFUSE!

To get back to your invitation to be an examiner, you will have to consider all reasons why you have been chosen, and remember you are not obliged to agree to the request.

Case study • X agreed to act as external examiner for Y’s PhD thesis.

• The thesis was of marginal quality but X’s preliminary report suggested that with modification and a satisfactory oral examination it was likely that a +ve recommendation could be made.

• Close scrutiny of the thesis immediately before the examination revealed references to an earlier MD thesis by the same candidate (at the same university). X contacted the internal to request that a copy of this thesis should also be available for scrutiny (it is reasonable to see all reference sources cited in a thesis).

• Before meeting the candidate on the day of the oral, X and the internal discovered that the earlier thesis reported about 70% of the data presented in the PhD thesis (about 45 pages of the introduction to both works was identical).

• X and the internal examiner commenced the examination.

Discuss the potential outcomes?