Peaks in sight: Research Master Review 2011

7/29/2019 Peaks in sight: Research Master Review 2011

1/84

Research Master Review 2011

Peaks in sight


2/84


3/84

Research Master Review 2011

Peaks in sight


4/84

4 Peaks in sight

Preace


5/84

NVAO Research Master Review 2011 5

This report ocuses on the quality o Dutch Research Masters

programmes. Accreditation assessments, the assessments

by oreign experts who have evaluated a wide selection o

Masters theses, student surveys, and the positions held by

alumni, are all used to sketch a picture o the quality o the

Research Masters programmes. The ndings in this report all

point to one conclusion: Dutch Masters programmes are o

a good quality, and the Research Masters programmes are

among the best in their elds. So the high plateau o Dutch

university education does appear to have real peaks oexcellence.

NVAO hopes this report will be welcomed. It is good or the

universities to know that Research Masters programmes are

held in high esteem internationally. It is good or students to

know that this type o Masters programme is a challenge or

anyone interested in research. It is good or society to know

that these programmes are perorming at the oreront o

their respective elds. Research Masters programmes

oer the possibility o increasing the number o doctorates

pursued; they oer the possibility o discovering and

educating talented people; and they provide a rm oundation

to urther strengthen already leading research achievements

in many disciplines and themes.

Karl Dittrich

Chairman, NVAO

Dutch universities are renowned the world over or their good

educational and research achievements. In all rankings,

universities in the Netherlands score high on the list. It can be

said, with good reason, that Dutch university education stands

on a high plateau. The Masters programmes, almost without

exception, have received high marks and are attractive to

oreign students. In addition, Dutch universities are highly

cherished partners in cooperative ventures understandably

so, in view o their level o achievement.

Despite this excellent track record, it is oten claimed that the

high plateau occupied by Dutch universities has no peaks o

excellence. In the eld o research, it has been proven that

this claim is unsubstantiated, but in the eld o education this

view continues to dog the universities. Not that consistently

staying on a high plateau is not an achievement in itsel,

yet obviously the existence o peaks (and perhaps also an

occasional valley) is necessary to put the high plateau into

relie. It is, in part, or this reason that some are arguing or

greater dierentiation and diversity between study

programmes.

A number o years ago, the so-called Research Masters

programmes were introduced in the Dutch university sector.

These two-year programmes are aimed at students looking

or a research programme, or at students who want to pursue

a proession in which research skills are essential.

The Research Masters programmes are taught by extremely

procient, experienced researchers. In these programmes,

emphasis is placed on learning research skills.

The combination o accomplished researchers and motivated

students, in particular, should lead to a study programme on

the cutting edge.


6/84

6 Peaks in sight

Contents


7/84


Summary 8

1 The Research Master Review 2011:

A birds eye view 14

1.1 Background and aims 15

1.2 Approach 17

2 Student enrolment, attrition,

and graduation 20

2.1 Student enrolment 212.2 Attrition rates 23

2.3 Graduation rates 24

2.4 Time to degree completion 24

2.5 Discussion 25

3 Thesis Assessment Study 28

3.1 Introduction 29

3.2 Preliminary analyses 30

3.3 Primary analyses 31

3.4 Discussion 32

4 Student satisaction in MA and RMA programmes 36

4.1 Overall ratings 37

4.2 Content o the programme o study 39

4.3 Acquisition o general skills 39

4.6 Teachers & lecturers in the programme 39

4.7 Assessment and grading 39

4.8 Study load 39

4.9 Academic guidance, support and supervision 39

4.10 Other aspects 41

4.11 Discussion 42

5 The alumni perspective 46

5.1 Selectiveness o the study programme 47

5.2 Satisaction with research related aspectso the programme 47

5.3 Satisaction with teachers and thesis

supervisors 49

5.4 Instruction time, workload and study results 49

5.5 Current employment 49

5.6 Satisaction with preparation or proessional

practice 49

5.7 Discussion 50

6 The accreditation o research masters

programmes 54

6.1 The accreditation o RMA programmes 55

6.2 KNAW committees nal conclusions 55

6.3 KNAW committees judgments

o themes 56

6.4 KNAW committees judgments

o standards 57

6.5 Discussion 61

7 Research masters: Crossing the borders

o science 64

7.1 The tul climate or science 65

7.3 The uture o research masters programmes 68

7.4 In closing 69

8 Synthesis 72

8.1 What did we nd? 73

8.2 What are the implications? 73

Reerences 78

RMR 2011 A collective eort 80

Interviews

Lucy van de Wiel Cultural Analysis 12

Jantine Boselie Psychopathology 18

Werner Raub Sociology and Social Research 26

Helena Cousijn Cognitive Neuroscience 34

Alexander Sack Cognitive and Clinical Neuroscience 44

Mirjam de Bruijn Arican Studies 52

Bojou Neecke Health Sciences 62

Thijs Porck History: Society and Institutions 70

Anne-France Pinget Linguistics 76


8/84

8 Peaks in sight

Summary


9/84


Student satisaction in MA and RMA programmesAlthough MA and RMA students both are generally quite

satised with their programme, satisaction is higher in RMA

students. RMA students are especially more positive about

the academic skills acquired through the programme, about

their teachers and lecturers, and about the academic

guidance, support and supervision oered by the programme.

On the other hand, RMA students are less positive about their

study load, the programme timetables, the extent to which

they learn how to work together with others, the availabilityand aordability o accommodation in the city where they are

studying, and about several acilities in their institution and

student-city environment.

The alumni perspectiveA large-scale survey among three cohorts o MA and RMA

alumni shows that RMAs are more selective than MA

programmes. Also, compared to MA alumni, RMA alumni

are more satised with research related aspects o their

study programme and with the quality and availability o

the teaching sta and thesis supervisors. They have more

instruction time, a higher workload, and higher study results.

And nally, they are more satised with the extent to which

their programme prepared them or proessional practice.

More than hal o all RMA alumni in the sample have enrolled

on a PhD programme.

Analysis o accreditation decisions orRMA programmesThe norms which RMA programmes have to satisy in order

to be accredited by NVAO clearly surpass those which have to

be met by regular MA programmes. Thereore by denition,

accredited RMAs demonstrate higher levels o quality than

regular MA programmes. Although the KNAW committees

who assess the RMAs maintain stern demands, they are quitepositive about the quality demonstrated by RMA. Between a

th and a third o all applications are judged as eithergood

or excellent on standards regarding (amongst others) the

intended learning outcomes, the curriculum and the quality

o the teaching sta.

What are the implications?

Educational policyThe RMA programmes t in perectly with the strategic

agenda initiated by the urgent call o the Veerman Committee.

RMAs main eatures include selection, sharper proling,

investments in research and higher qualied personnel the

same eatures which are highlighted in the recommendations

o the Veerman Committee. In doing so, the RMA contributes

What do we want to know?

The higher education system o the Netherlands is oten

characterised as a plateau without peaks: exhibiting high

quality across the board, yet virtually devoid o true excel-

lence. NVAO wishes to challenge this rather defated image,

being convinced that universities in the Netherlands certainly

do oer a reasonable amount o excellent study programmes.

More specically, we suppose that the research masters

programmes (RMAs) just might constitute such peaks onthe plateau.

NVAO initiated the Research Master Review (RMR 2011) with

one paramount research question in mind: What is the level o

educational quality realised in RMA programmes? A series o

studies was perormed to answer this question. We analysed

accreditation decisions, key statistics about enrolment,

attrition and graduation, and looked into the quality o RMA

theses, student satisaction and the alumni perspective.

What did we fnd?

Each o these studies provides valuable insights about

the quality and strengths o RMA programmes. They also

provide insights that go beyond the research question and

give an indication about how we can improve quality in

higher education.

Enrolment, attrition and graduationOur analyses show that the average number o entrants has

increased in recent years. However, the student intake o

many RMAs seems to be well below programmes capacity.

The average attrition rate increases rom 11% ater one year

to 17% ater our years. The average time to degree is 23.7

months. About 78% o all entrants graduate within our yearsater entering the programme. Enrolling on a PhD programme

prior to graduation eatures among the reasons reported to

explain students delay in nishing the programme.

International students make up a large proportion o the RMA

entrants: 36% in 2009.

Thesis assessment studyThis study involved a blind assessment o MA and RMA theses

by panels composed o oreign experts in our disciplines.

Our analyses indicated that all the theses assessed meet

international quality standards. However, RMA theses are

given signicantly higher ratings than MA theses. Moreover,

the dierences in quality between MA and RMA theses

appear to be considerably large. The assessment panels

considered the (vast) majority o theses assessed to be o

top international level.


10/84

10 Peaks in sight

to making the Dutch higher education system more uture-

proo. Thereore, the Dutch government and policy makers

would do well to cherish the RMA and assure the continuity,

accessibility and quality o RMA programmes.

Didactical approachRMA programmes derive part o their success rom their

characteristic didactical approach: the master/apprentice

model o instruction and academic socialisation.

Several distinctive attributes o RMAs are among the mostpowerul infuences on achievement:

> highly motivated students who are ready to put a

considerable eort into their studies;

> challenging learning goals;

> a captivating delivery by highly qualied and passionate

teachers;

> teachers and students orming a cohesive community

o learners.

Enhancing continuity through increasing studentenrolmentFor many RMAs, increasing student enrolment remains the

biggest challenge. RMR 2011 suggests a number o ways to

achieve this:

> Educational institutions could do more to inuse more

positive attitudes towards science and research in their

students. Challenging research seminars as part o the

bachelors programmes could be a good starting point to

oster a deeper and lasting interest in research and science.

> Scouting is a promising strategy: approach talented

students at an early stage and give them a taste o what

an RMA might bring them.

> The co-operation between universities could be

strengthened. Invigorating the inter-university

research schools seems to be a logical starting point.> To recruit more oreign research talent, Nuc could

eature the RMA more explicitly as a unique selling point

o the Dutch higher education system.


11/84



12/84

12 Peaks in sight

Interview


13/84


Lucy van de Wiel

Cultural Analysis

Lucy van de Wiel started her PhD in June 2011, ater nishing

the research masters programme Cultural Analysis at the

University o Amsterdam: It is an interdisciplinary programme,

ocused on analysing cultural objects. That could be anything,

or example, a radio programme, a policy text, or a literary

work. The programme teaches students how to analyse thoseobjects, using theories about culture stemming rom dierent

philosophical traditions. These analyses provide insight in

the meaning o cultural symbols and objects within a given

social context, while at the same time contributing to theory

building.

The course o my studies is a bit atypical. Ater nishing

my bachelors programme, I had the opportunity to study

abroad on a Fulbright scholarship. In that year, I participated

in a PhD programme at the University o Caliornia in Berkeley.

I was able to count this experience as an elective course or

the research masters programme, so I could nish in one year

instead o two. I wanted to do a research masters programme

anyhow, because it provides more challenge and because I

have always been interested in research.

I only did the compulsory courses, plus one optional course.

The general courses were interesting, although they were a bit

too broad in scope. Sometimes I would have liked more depth.

In general, the courses in the second year o the curriculum

have more depth, but I already did the second year in

Berkeley. However, I did like the degree o reedom we had in

choosing the topics or our projects. The thesis supervision

was very good, so in that respect I am absolutely satised.

The research or my PhD involves reezing egg-cells, making it

possible or women to have children at a later age. This has

been a hotly debated issue in recent years, or one, because

the Amsterdam Medical Center wanted to start a clinic where

women could have their egg-cells rozen in. At rst, that was

prohibited in the Netherlands, although it was allowed inBelgium and the UK. I study the societal debate about this

issue. Which kinds o arguments are used, which images and

conceptions about men, women, and parenting orm the basis

or these arguments? How do these ideas originate, and how

are they translated in dierent cultural expressions, such as

parliamentary debates, Victorian medical texts, and biblical

stories? What do they tell us about identity ormation, ideas

about the body, and temporality?

My primary aim is to prot as much as possible rom the next

our years. I want to prepare mysel ully or a career in

science, by working on my dissertation, giving lectures, and

writing articles. Ater completing my doctoral studies, I would

like to publish my dissertation and rewrite it into a book that

is more accessible to a broader audience.


14/84

14 Peaks in sight

Chapter 1 The ResearchMasterReview 2011:

A birds eyeview


15/84


For most universities, there also was a modest nancial

incentive or starting up RMA programmes. When the

bachelors-masters degree structure was implemented,

regular masters programmes received public unding or

one year (60 ECs). Only a number o disciplines, such as the

natural sciences, lie sciences, agriculture, and engineering,

were allowed masters programmes with a duration o and

unding or two years (120 ECs). Also, the students nancial

support by the government depends on the duration o

the programme. The RMA provided universities with theopportunity to oer masters programmes with a two-year

duration in other disciplines, such as the humanities, social

and behavioural sciences. The institutions do not receive

extra public unding or these programmes, but students

enrolled on an RMA programme are granted an extra year

o nancial support.

1.1.2 Research Master Review 2007In 2007 NVAO initiated a rst study into this new type o

programme. This study aimed at raising awareness o and

attention or the RMA, and identiying the distinctive prole

characteristics o the RMA. The study also explored the rst

experiences with and expectations or these programmes.

NVAO published the ndings in a report, titled Research

Master Review 2007: The exploration o a new domain

(RMR 2007).

RMR 2007 identied three prole characteristics which

distinguish RMA programmes rom regular MA programmes

oered by research-oriented universities in the Netherlands:

> Programme objectives and a curriculum specically ocused

on the acquisition o research competencies;

> An academic context o unquestionably high quality that

strives or excellence; and

> Selection o students with above-average academicperormances and ambitions.

NVAO received a large amount o applications or the initial

accreditation o RMAs: 156. This indicated a strong need or

this new type o masters programmes among most o the

research universities in the Netherlands. Throughout the

academic community, the RMA was very well received as a

successul and promising educational innovation.

However, RMR 2007 did also signal one serious drawback:

the disappointing student enrolment. Allegedly, this was

caused by the image o the RMA as only being o interest or

those aspiring a PhD. To increase student enrolment, the

added value o the RMA should be made clear to those who

do not aspire a career within academia, it was reasoned.

The higher education system o theNetherlands is oten characterisedas a plateau without peaks: exhibitinghigh quality across the board, yetvirtually devoid o true excellence.NVAO is convinced that universities inthe Netherlands certainly do oer a

number o excellent study programmes.We suppose that the research mastersprogrammes (RMAs) just mightconstitute such peaks on the plateau.

To support that assumption, NVAO took the initiative or the

Research Master Review 2011. The project consisted o

several studies, combining both quantitative and qualitative

research methods. This chapter discusses the backgrounds

and aims o the project, and provides an overview o the

chapters that ollow.

1.1 Background and aims

1.1.1 The introduction o the researchmasters programmeIn 2003, the RMA was introduced as a new type o study

programme in the Dutch higher education system. RMAs are

specically aimed at the acquisition o advanced research

abilities. Enrolment in an RMA is restricted to students with

above-average academic achievements and motivation.

RMAs are embedded in an academic environment ogood to

outstandingquality, and the core teaching sta consists o

highly qualied researchers. For many, an RMA is the natural

steppingstone towards a PhD trajectory.

The RMA was introduced or several reasons. For a long time,

Dutch policy makers have argued that the Netherlands can

only compete in a global perspective as a knowledge economy.

In the global competition between knowledge societies,

innovation and creativity are considered to be the most

important drivers o economic growth. In this context, the

RMA would answer to the nations demand or more and

better researchers.

As a beacon o high-level quality, the RMA would help attract

young talent rom outside the Netherlands. Also, RMAs were

supposed to increase the disappointing graduation rates o

Dutch PhD-trajectories and decrease the average time that

doctoral students take to nish their dissertations.


16/84

16 Peaks in sight

> A new arrangement or masters programmes: the oer o

masters programmes should be dierentiated, and the

proessional masters should be more structurally

embedded. Also, it should be explored how to enhance the

fexibility or lielong learning.

The committee also ormulated three recommendations to

the institutions:

> Choose a prole: institutions and their subdivisions should

opt or a distinctive prole on one or more dimensions othe European classication system, based on achieved or

desired strengths.

> Give more attention to education as a core task o higher

education institutions: be responsive to the dierent

learning styles and backgrounds o students, make study

programmes more fexible and better organised.

Institutions and students should work together at realising

this objective.

> Invest in higher qualied personnel: institutions should

provide teachers wider opportunities, and be sensitive to

career-planning with equivalent and coherent career

tracks in education and research.

On 1 July 2011, the current State Secretary or Education,

Culture and Science, Halbe Zijlstra, issued a memorandum

titled Quality in Diversity: Strategic Agenda or Higher Educa-

tion, Research, and Science. Many o the policy measures

which are suggested in this memorandum are inspired by

the Veerman Committees report. In our view, RMAs seem to

answer to many o the recommendations ormulated by the

Veerman Committee, and to many o the objectives stated in

the strategic agenda or higher education.

1.1.5 Global objectives

Against these backgrounds, NVAO initiated RMR 2011 withone paramount research question in mind: What is the level o

educational quality realised in RMA programmes? Answering

this question would contribute to attaining the ollowing

global objectives.

Evaluating the educational quality realised in

RMA programmes

NVAO collects a wealth o inormation about the quality o

individual RMA programmes in the orm o critical sel-

evaluations and external quality assessments. These sources

can be used to analyse the educational quality realised by

RMA programmes at a higher level o aggregation. This kind

o meta-evaluation could show important developments,

trends, good practices and structural challenges within the

domain o RMA programmes.

1.1.3 Accreditation o RMAs as existing studyprogrammesNearly all RMAs that were positively assessed as new study

programmes in 2003-2007, have undergone a second assess-

ment as existingstudy programmes. From March 2008 to

March 2011, NVAO received a total o 128 applications, pre-

dominantly or the accreditation o existing RMA programmes

(116), and some or the initial accreditation o new RMA

programmes (12). Furthermore, the so-called second phase

o the accreditation system was legally eectuated on 1January 2011. This involved some undamental changes to

the accreditation procedures which were ollowed in previous

years. Recently, the newAssessment rameworks or research

masters programmes have been established, and per

1 September 2011, RMAs are being assessed according

to the new regime.

1.1.4 Policy contextIn September 2009, the then Minister o Education, Culture,

and Science, Ronald Plasterk, assigned a committee chaired

by ormer Minister Cees Veerman. The committee was asked

to assess the extent to which the higher education system o

the Netherlands is uture-proo, compared to leading higher

education systems elsewhere. When NVAO initiated RMR

2011, the Veerman Committee had just published its advice

report, advocating a triple dierentiation (in the structure o

the system, in the proles o institutions, and in the oer o

study programmes) to make Dutch higher education more

uture-proo. The committees report caused a stir in the

academic community, uelling the national debate about some

undamental changes in the structure o the Dutch higher

education system.

The Veerman committee ormulated seven recommendations

to the Dutch Government. Five are relevant or RMAs:> Selection: each institution should be given the right to select

students.

> Stimulate institutional proling and reward accomplish-

ments: In order to stimulate distinctive institutional proling,

the European U-Map system or multidimensional classi-

cation should be urther developed and adapted to the

Dutch context.

> A smaller share o student-bound unding: a higher share o

mission-bound unding that rewards accomplishments in

the distinctive academic prole o institutions.

> Investment in research: in order to secure the competitive-

ness o the Netherlands, investments are needed to

stimulate the integration o research throughout the higher

education system.


17/84


The alumni perspective

Chapter 5 presents the results o a study perormed by

Jorrit Snijder, managing director o the Roosevelt Academy

and researcher at CHEPS (the Center or Higher Education

Policy Studies o the University o Twente). To learn more

about the eectiveness o the RMAs, he surveyed three

cohorts o alumni rom research masters (RMA) and regular

masters (MA) programmes rom 9 Dutch universities,

between 1.5 and 2 years ater their graduation.

Analysis o accreditation decisions or RMA programmes

Chapter 6 reports the outcomes o an analysis o the accredi-

tation decisions or RMAs. At the conerence, the preliminary

results o this study were presented by Karl Dittrich, chair o

NVAO. Specically, this study looks into the question which

quality criteria are the most important or distinguishing

between positively and negatively assessed applications.

Crossing the borders o science

Chapter 7 is an elaboration o the speech held by Robbert

Dijkgraa, President o the Royal Netherlands Academy o Arts

and Sciences (KNAW). The article sketches the current climate

or science, the challenges or higher education in the Nether-

lands, and the uture o RMA programmes.

Interviews

The chapters reporting the results o the various studies

are alternated with interviews with students, alumni and

sta rom RMAs. They oer a view rom inside and provide

valuable insight into the quality o RMAs.

Strengthening the continuity o RMA programmes

Although the academic community is generally enthusiastic

about the RMAs, RMR 2007 also pointed out one major

weakness: many RMAs appear to recruit (substantially) less

students than possible or desirable. Through RMR 2011, NVAO

hopes to collect ideas about how to increase the student

enrolment, in order to strengthen the continuity o RMAs.

Promoting quality improvement in higher education

Although accountability is being given ull attention in thecurrent accreditation system, the purpose o quality improve-

ment remains a very real part o the system. Through RMR

2011, NVAO aims to promote quality improvement in higher

education, which is, o course, primarily the responsibility o

the institutions.

1.2 Approach

RMR 2011 consisted o a series o studies, in which both

quantitative and qualitative research methods were

combined. To provide a look ahead, we discuss each

study briefy.

Enrolment, attrition and graduation

Chapter 2 provides some key statistics concerning the student

enrolment, attrition rates and graduation rates or RMA

programmes. These statistics are based on cohort datasets

provided by the Association o Universities in the Netherlands

(VSNU).

Thesis assessment study

Chapter 3 discusses a thesis assessment study (TAS), which

aims to answer the questions (a) to what extent theses written

by RMA students dier in quality rom theses written by MA

students, and (b) how the quality o theses written by DutchMA and RMA students relate to international standards.

Student satisaction in MA and RMA programmes

Each year, the Dutch National Student Survey(NSE) is held,

measuring student satisaction among almost all institutions

or higher education in the Netherlands. Chapter 4 reports the

results o an analysis o NSE data, identiying dierences in

satisaction between MA and RMA students.

Conerence

On Thursday 24 March 2011, NVAO organised the conerence

Research masters: Excellent profling la Veerman (or an

overview o the programme in Dutch, see http://www.nvao.net/

nvao-congressen). Three o the lectures held at the

conerence were converted into articles or this report,

included in chapter 5, chapter 6, and chapter 7.


18/84

18 Peaks in sight

Interview


19/84


The research masters programme absolutely meets my

expectations. Our teachers are leading specialists in their

elds, like our university proessors who are specialised in

anxiety disorders, and experts rom the USA. That interaction

is very stimulating. We are taught to think critically about

research. Besides knowing how to do scientic research,you should also be able to critically evaluate the quality o

a study.

I would like to continue with a PhD programme. Finding

grants, however, we have to do ourselves. Our department

oers some support, or example, by posting vacancies or

research positions on the website. But I havent yet ound

my bagul o money.

Jantine Boselie

Psychopathology

Jantine Boselie is enrolled on the second year o the research

masters programme Psychopathologyat Maastricht University:

The programme is ocused on mental disorders, such as

anxiety disorders, depression, and personality disorders.

For example, we examine whether a certain treatment is

eective. Or we investigate the mechanisms underlying adisorder, like depression: is that a biological disorder, or do

social and cognitive actors also have an infuence?

The Psychology department at Maastricht University has a

strong ocus on research. In the bachelors programme, I soon

discovered that I very much liked doing research. I wanted to

experience the whole process, rom ormulating the research

question, to executing the study, and nally writing about it

all. It certainly helps that I am extremely curious; I want to

know peoples drives and which actors infuence those drives.

An important motive or me to choose this masters

programme is its two-year duration. In the rst year, we get a

lot o theory. In the second year, we have a clinical internship

(a placement at a hospital, or example), as well as a research

internship. That combination makes it interesting to me. And it

oers also that little extra to mention on your resume, which

can make a large dierence in these times.


20/84

20 Peaks in sight

Chapter 2 Studentenrolment,attrition,andgraduation


21/84


Table 2.1 shows the number o entrants or the cohorts

2005-2009. Or more exactly, the total number o frst-time

enrolments:a small number o students are counted

twice, because they are registered in two dierent RMA

programmes. Over a third o all RMA students enrol on

programmes within the humanities. In contrast, RMA

programmes within the earth sciences had only 54 rst-time

enrolments. Within ve years, the total number o entrants

has almost doubled: rom 836 in 2005 to 1,528 in 2009.

However, there appear to be large dierences in growth rates

between the dierent discipline clusters: growth is most

prolic in the (bio)medical sciences and virtually nonexistent

in the earth sciences. The growth in the number o RMA

entrants parallels the growth in the whole population o

students entering a masters programme. Thereore, the

proportion o RMA entrants within that population has

remained at a steady 3-4%.

Growth in the number o RMA programmes

Table 2.2 displays the number o RMA programmes whichregistered at least one new entrant or the cohorts 2005-

2009. Over the years, a ew RMAs within the Humanities

and Social Sciences were discontinued and closed or new

enrolment. Still, the total number o RMAs with student

intake has steadily increased, rom 94 in 2005 to 123 in 2009.

About hal are RMAs within the humanities and about a

quarter are RMAs within the social sciences. Growth in the

number o RMAs is most salient in the (bio)medical sciences.

How many (international) students enrolon RMA programmes? How many othem drop out, and how many graduate?How long does it take RMA students tocomplete their programme?

Objective monitoring o perormance is essential or

evaluating and raising quality on all levels o analysis.

This chapter considers several measures which are widelyused as educational perormance indicators: student

enrolment, attrition rates, graduation rates, and time to

degree completion. All the results presented in this chapter

pertain to ve student cohorts, rom cohort 2005 onward.

VSNU provided and assisted with analysing the data.

Our analyses indicate that the number o RMA programmes

and the number o students enrolling on RMAs have been

growing steadily. About 78% o all entrants graduate within

our years ater entering the programme. The average time to

degree is 23.7 months, and 76% o all graduates nish within

the RMA programmes nominal duration o 24 months.

2.1 Student enrolment

Growth in the number o entrantsThe growing demand or more and better researchers, and

making Dutch higher education more attractive to oreign

talent, were among the main reasons or introducing the RMA.

To assess the extent to which the RMAs are meeting these

objectives, it is necessary to know exactly how many Dutch

and international students enrol on RMA programmes.

But keeping track o student enrolment is also important or

didactic purposes: group size has a direct infuence on the

teaching/learning process.

Table 2.1 Number o entrants in 2005-2009.

2005 2006 2007 2008 2009 Total

(Bio)medical Sciences 75 115 171 190 217 768

Behavioural Sciences 210 262 309 353 404 1,538

Earth Sciences 6 7 26 8 7 54

Humanities 343 371 414 507 522 2,157

Social Sciences 202 207 268 304 378 1,359

Total 836 962 1,188 1,362 1,528 5,876

% o MA population 4% 3% 3% 4% 4% 4%


22/84

22 Peaks in sight

Growth in the average number o entrantsThe average number o entrants per RMA has increased

rom 8.9 in 2005 to 12.4 in 2009. Growth is most salient in

RMAs within the behavioural and social sciences, and less

pronounced in RMAs within the humanities and (bio)medical

sciences. Over all disciplines and cohort years, the average

number o entrants per RMA programme was 10.7 (SD=8.7).

Table 2.3 Average number o entrants per RMA in

2005-2009.

Figure 2.1 Variability in the number o entrants or cohort

2009 (n=123).

High variability in the number o entrantsFigure 2.1 displays the high variability in the number o

entrants or the 123 RMA programmes with student intake

in 2009. Some RMAs are very small: 25 RMAs show an intake

o less than 5 students. On the other hand, 19 RMAs show

an intake o more than 20 students. More than hal o all

RMAs (63 programmes) had an intake o 10 students or less.

More than a quarter (32 programmes) had an intake o

5 students or less, and 5 programmes had an intake o only

1 student.

Objectives regarding student intakeRMA programmes, on average, aim or a maximum intake o

25 students per cohort, with a maximum o 10 students or

the smallest and 60 or the largest programmes. So with an

average intake o 12.4 students or cohort 2009, the student

enrolment seems to be well below the programmes capacity.

The KNAW Social Sciences Council (SWR, 2007, pp. 13-14)

reported that RMAs within the social and behavioural

sciences, on average, aimed or a maximum intake o

23 students per cohort (SD=10.8), and a minimum intake

o 15 students (SD=9.7). Since these RMAs, on average,

had an actual intake o 11 students in 2006 (SD=8.5), the

SWR concluded there was plenty room or growth. In 2009,

the average number o entrants or RMAs in the social and

behavioural sciences was 17 (SD=14.4). This is slightly

above these programmes objectives or the minimum

student intake, but still leaves room or growth.

A high proportion o international studentsTable 2.4 shows that international students make up 31% o

the RMA entrants. The largest proportions o international

students are ound in RMAs within the earth and social

sciences, the smallest proportion in RMAs within the (bio)

medical sciences. The proportion o international students

has increased rom 29% in 2005 to 36% in 2009.

Table 2.2 Number o RMAs with student intake in

2005-2009.

2005 2006 2007 2008 2009

(Bio)medical Sciences 6 8 10 12 15

Behavioural Sciences 13 15 17 17 17

Earth Sciences 1 1 2 2 2

Humanities 50 55 56 57 60

Social Sciences 24 26 27 29 29

Total 94 105 112 117 123

2005 2006 2007 2008 2009 Total

(Bio)medical Sciences 12.5 14.4 17.1 15.8 14.5 15.1

Behavioural Sciences 16.2 17.5 18.2 20.8 23.8 19.5

Earth Sciences 6.0 7.0 13.0 4.0 3.5 6.8

Humanities 6.9 6.7 7.4 8.9 8.7 7.8

Social Sciences 8.4 8.0 9.9 10.5 13.0 10.1

Total 8.9 9.2 10.6 11.6 12.4 10.7

0

2

4

6

8

10

12

14

0 5 10 15 20 25 30 35 40 45 50 55 60

Frequency

Number o entrants


23/84


Table 2.5 Attrition rates per cohort.

The most recent mobility statistics available or the Nether-

lands published by Nuc (Richter et al., 2011) indicate that

o all the students enrolled on masters programmes, 15%

are international students. The proportion o international

students among the RMA entrants is more than twice as high.

2.2 Attrition rates

In contrast to most other masters programmes, RMAs are

selective: only students with above-average motivation and

prior achievements are eligible or enrolment. Students who

are admitted may be expected to meet the high requirements

o the programme and to successully complete the programme

within the nominal programme duration o two years. In other

words, RMA programmes should have low attrition rates and

high graduation rates. Otherwise, their selectivity would not

be justied.

Table 2.5 shows the attrition rates (the number o

deregistered students as a percentage o the cohort size)1-4 years ater rst enrolment or the cohorts 2005-2009.

The percentages in this table are not cumulative: some

RMA students who deregister in a certain year return to

the programme in a later year. But the number o these

re-entrants is smaller than the number o deregistering

students: the average attrition rate increases rom 11%

ater 1 year (cohorts 2005-2009), to 17% ater 4 years

(cohorts 2005-2006).

Table 2.4 Intake o Dutch and international students in

2005-2009.

About a third o all students who let and did not return to

their RMA, at some time did complete a masters programme

within the same eld o science and at the same institution

as their RMA programme. About 60% let and did not enrol on

another masters programme within the same institution as

their RMA. Some RMA programmes sel-evaluation reports

mention several reasons or students leaving the programme:

(a) personal problems, (b) ollowing courses or doing an

internship at a university outside the Netherlands, (c) a job

oer, or (d) being admitted to a PhD programme. However,

there are no data on drop out rom regular masters

programmes. Thereore, we lack the necessary inormation

to comparatively assess these attrition rates.

The calculation o attrition rates (in this section) and

graduation rates (in the next section) is based on cohort

sizes which dier rom the number o entrants reported

in the previous section. For calculating the attrition and

graduation rates, we have excluded two groups oenrolments which would conound the results:

> 429 enrolments on our RMAs within the medical sciences

which have a nominal programme duration o more than

2 years; and

> 863 late enrolments (registered ater the reerence date

o 1 October).

Cohort Cohort

size

< 1yr < 2yr < 3yr < 4yr

2005 678 11% 14% 15% 16%

2006 719 11% 15% 16% 18%

2007 927 10% 14% 16% -

2008 1,073 11% 16% - -2009 1,187 11% - - -

Total - 11% 15% 16% 17%

Dutch International Total

(Bio)medical Sciences 625 18% 143 19% 768

Behavioural Sciences 1,067 69% 471 31% 1,538

Earth Sciences 32 59% 22 41% 54

Humanities 1,519 70% 638 30% 2,157

Social Sciences 820 60% 539 40% 1,359

Total 4,063 69% 1,813 31% 5,876


24/84

0%

20%

40%

60%

80%

100%

0 6 12 18 24 30 36 42 48 54 60

i

24 Peaks in sight

Table 2.7 shows that the average time to degree completion

was 23.7 months or all 2,788 RMA graduates (SD=7.5).

There seem to be no large dierences between students

within dierent disciplines. So on average, RMA graduates

seem to have nished exactly within the programmes nominal

duration o two years. However, the dataset also includes

the lateral entrants who are expected to complete the RMA

programme in less than two years. I these lateral entrants

could be excluded, the average time to degree would be higher.

Table 2.7 Average time to degree (in months).

2.3 Graduation rates

Table 2.6 displays the graduation rates (the number o

diplomas awarded as a percentage o the cohort size)

1-4 years ater rst enrolment. On average, 6% o all

entrants graduated within 1 year (cohorts 2005-2009),

52% graduated within 2 years (cohorts 2005-2008),

73% graduated within 3 years (cohorts 2005-2007),

78% graduated within 4 years (cohorts 2005-2006), and

79% graduated within 5 years (cohort 2005) ater rstenrolment on the RMA.

Table 2.6 Graduation rates per cohort.

Two actors conound these graduation rates. First, almost

all RMA programmes oer the possibility o lateral entry:

students who have completed a comparable regular masters

programme can enter in the second year o the RMA.

This explains the graduation rate o 6% ater 1 year.

Second, these graduation rates simply refect the number

o years between students rst enrolment and programme

completion. They have not been corrected or years thatstudents may have been deregistered prior to graduation.

This correction has been made when calculating the statistics

reported in the next section.

2.4 Time to degree completion

Graduation within the nominal programme duration

The time to degree completion is dened as the number o

months between frst enrolment and successul completion,

corrected or years in which the student was deregistered.

The results reported in this section pertain to all RMA

graduates o the cohorts 2005-2009. However, they do not

include the graduates o the our RMAs within the medical

sciences which have a nominal programme duration o more

than two years.

Cumulative percentage o graduates over time

Figure 2.2 shows how the percentage o graduates cumulates

over the number o months rom rst enrolment to successul

degree completion (corrected or years in which students were

deregistered, but including lateral entrants). O all RMA gradu-

ates, 11% completed the programme within 12 months, 16%

nished within 18 months, 76% nished within 24 months,

87% nished within 30 months, and 96% nished within 36

months. Only 112 graduates (4%) took more than 36 months

to complete their RMA programme.

Figure 2.2 Cumulative percentage o graduates over

1-60 months (n=2,788).

Cumulative percentage

Months

N M SD

(Bio)medical sciences 168 25.3 7.9

Behavioural sciences 881 23.4 5.6

Earth sciences 38 21.2 9.0

Humanities 996 24.5 8.6

Social sciences 705 22.8 7.4

Total 2,788 23.7 7.5

Cohort Cohort

size

< 1yr < 2yr < 3yr < 4yr

2005 678 6% 51% 72% 78%

2006 719 5% 54% 73% 78%

2007 927 6% 52% 73% -

2008 1073 5% 53% - -

2009 1187 6% - - -

Total - 6% 52 73% 78%


25/84


Some RMA programmes sel-evaluation reports mention

reasons or students delay in nishing the programme:

(a) personal problems, (b) ollowing courses or doing an

internship at a university outside the Netherlands, and

(c) choosing to spend extra time on nishing their masters

thesis: some students extend and elaborate their theses

into a component o a PhD programme they enrol on prior

to graduation.

2.5 DiscussionAlthough we have no data on attrition and graduation rates in

other masters programmes, we conclude that the attrition

and graduation rates or RMA programmes give no cause or

concern. Actually, our ndings sketch a airly positive picture

o the results o RMA programmes: out o every 100 students

enrolling on an RMA, approximately 11 students drop out in

the rst year, 8 students leave the programme in a later year,

60 students graduate within 24 months, 16 students graduate

within 25-36 months, and 5 students graduate within 37 or

more months ater rst enrolment.

Our ndings regarding student enrolment on RMA

programmes, however, sketch a less positive picture. Both

NVAO (2007, p. 60) and the SWR (2007, pp. 18-19) reported

that the low student enrolment was among the stakeholders

major concerns. A higher number o entrants would not only

be desirable or RMA programmes meeting the demand or

more researchers and PhD students. According to some,

increasing the student intake would also be necessary or

securing the viability and continuity o the RMAs.

The ambitious learning objectives which characterise RMA

programmes demand intensive modes o instruction. So when

student intake is too large, it becomes dicult or teachers toprovide the appropriate amount o challenge and guidance.

But when the intake is too small, students have ew opportu-

nities or learning through peer tutoring, or learning how to

work together in dierent groups, and or developing leader-

ship and confict resolution skills.

Our analyses show that the average number o entrants has

increased in recent years. However, the student intake o most

RMAs seems to be well below programmes capacity. Indeed,

in order to provide their students with the opportunity to reap

the benets o collaborative learning, many RMAs would do

well to increase the enrolment, while maintaining the current

level o admission requirements.

RMAs attract a relatively large proportion o international

students. International students make up 15% o all students

currently enrolled on masters programmes, whereas the

proportion o international students among the RMA entrants

o cohort 2009 was 36%. Generally, the recruitment o interna-

tional students is organised locally by the programmes and

universities individually. To urther increase the intake o

international students, the RMA programmes and institutions

could consider joining orces, and seeking co-operation with

Nuc.

But recruitment activities could also be aimed at increasing

the enrolment o Dutch students. To this end, urther research

should shed light on the reasons students have or enrolling,

or rather, not enrolling on an RMA programme. Is it because

they are unamiliar with this type o masters programme? Or

do they nd the programmes high demands and prolonged

duration unappealing? Do they think that RMAs are only or

students who aspire to enter a PhD programme? Or cant they

meet the admission requirements? Such knowledge about the

motivation o students or not enrolling on an RMA, can inorm

institutions about how to position the RMA more eectively in

the higher education market.


26/84

26 Peaks in sight

Interview


27/84


Werner Raub

Sociology andSocial Research

Pro Dr W. (Werner) Raub, proessor o theoretical sociology

at Utrecht University: I ounded the research masters

programme Sociology and Social Researchat Utrecht

University, and until recently I also led this programme.The programme attracts about 15 students each year.

That is exactly the number o students which we aim or.

Students start in September and are expected to successully

complete the programme within 22 months. There are no

study delays, all students nish in time. The attrition rates

are at a minimum, about 10%, o which the majority drops

out within the rst weeks o the curriculum. Those are

students who have made the wrong choice.

Important actors or the success o our research masters

programme are the relatively small groups, the intensive

guidance, and the adequate selection. Right rom the start,

students know what to expect. There are no re-examinations,

each course has to be successully completed the rst time

around. Students also compare themselves with each other:

weaker students see what the stronger ones do and dont

want to all behind.

Our research masters programme is a breeding ground or

the PhD trajectory. We co-operate with colleagues rom

Groningen and Nijmegen in an inter-university research

school. Annually, we select about 15 new PhD students

rom a total o 250-450 applicants. Each year, 7-10 research

masters graduates apply or a position, and 60-70%

successully so. This certainly indicates their quality.

We recruit intensively abroad, so we have around 50%

international students. That is also our target or the years

to come. Dutch universities should take into account their

competitiveness, when deciding on the tuition ees orinternational students. Students rom the European

Economic Area (EEA) have to pay the same registration

ees as Dutch students, but students rom outside the EEA

pay a lot more, about 15,000 euro per year. They pay less or

similar programmes in Germany.

The introduction o the research masters programmes has

been an excellent development. They contribute signicantly

to a better preparation o PhD students. However, many o

these programmes are too small. The main problem is the

policy o the Dutch universities: all institutions do their own

thing. Better co-operation between universities would

present much better opportunities or viable programmes.

For example, you could think o merging similar research

masters programmes rom two or three dierent universities.

With the small distances and well-unctioning inter-university

research schools, that would certainly be attainable.


28/84

28 Peaks in sight

Chapter 3 ThesisAssessmentStudy


29/84


To what extent is there a dierence inquality between theses written bystudents enrolled on research masters(RMA) programmes and those written bystudents enrolled on regular masters(MA) programmes? How does the qualityo Dutch (research) masters theses relate

to international standards?3.1 Introduction

I RMAs are truly excellent programmes, then this should be

expressed in the quality o the theses produced by their

students. Supposing that RMA theses are o high quality, not

only in comparison with regular masters theses, but also in

an international perspective, NVAO conducted a thesis

assessment study (TAS). The study was conducted in close

co-operation with masters programmes in our disciplines:

> Psychology and Neuroscience, Maastricht University (P&N);

> Media Studies, University o Amsterdam (MS);

> Communication Science, University o Amsterdam (CS); and

> Economics and Business, Tilburg University (E&B).

Foreign experts were invited to take part in the assessment

panels. The composition o the our panel was as ollows:

For all our study sites, samples were drawn rom the popula-

tion o recent theses in English (n=74). Hal o the sample was

written by RMA students, the other hal by students rom

adjacent regular MA programmes. For two o the study sites,

samples were drawn rom theses that centred on the mean

supervisors grades in the MA and RMA populations. The

theses in these samples represent the average MA and RMA

theses. For the two other study sites, we randomly selected

theses covering the ull range o grades (varying rom 6 to

10 on a 10-point scale). The theses in these samples refectthe ull variability in quality present in the two populations.

These panels were assisted by NVAO policy advisors, who

acted as secretaries. In order to enhance the consistency and

comparability o assessments between panels and panel

members, NVAO developed a thesis assessment ramework

including nine standards. Prior to presenting the 74 theses or

assessment by the panel members, they were made anony-

mous: the names o students and supervisor(s) were removed,

as well as inormation that would point to the programme in

which the student was enrolled. The assessments were

perormed double-blind: neither panel members nor secre-

taries knew which were the (R)MA theses. Nor did they know

how the theses were graded by supervisors.

Psychology and

Neuroscience

David Linden (Proessor o Translational Neuroscience, School o Medicine, Cardi University, UK), (Chair);

Miguel Castelo-Branco (Director o the Institute o Biomedical Research in Light and Image,

University o Coimbra, Portugal); and

Lars Muckli (Proessor o Psychology, University o Glasgow, UK).

Media Studies Astrid Sderbergh Widding (Proessor at the Institute o Cinema Studies, University o Stockholm, Sweden), (Chair);

Jostein Gripsrud (Proessor at the Department o Media Studies, University o Bergen, Norway); and Matthew Fuller (Proessor at the Centre or Cultural Studies, Goldsmiths, University o London, United Kingdom).

Communication

Science

Keith Roe (Proessor at the Leuven School or Mass Communication Research, University o Leuven, Belgium), (Chair);

Wolgang Donsbach (Proessor o Communication at the Department o Communication at the University o Dresden,

Germany); and

Ral Terlutter (Proessor o Marketing and International Management, University o Klagenurt, Austria).

Business & Economics Hannelore Weck-Hannemann (Proessor o Political Economy at the Institute or Public Finance at the University

o Innsbruck, Austria), (Chair);

Gordon Gemmill (Emeritus Proessor o Finance, Warwick Business School, University o Warwick, UK); and

Piet Sercu (Proessor o International Finance, Katholieke Universiteit Leuven, Belgium).


30/84

30 Peaks in sight

3.2.2 Assessment panels ratingsTable 3.2 presents the means and standard deviations or

the assessment panels ratings o both MA and RMA theses

on each o the nine standards o the assessment ramework.

The assessment panels ratings were calculated in the

ollowing way. First, the assessment panel members

individually rated each thesis on the nine standards, using

a 4-point scale (1=unsatisactory, 2=satisactory, 3=good,

4=excellent). Secondly, an average score or each standard

was calculated rom the individual ratings o the three panel

members. Thirdly, the individual panel members scores on

the nine criteria were averaged. Finally, these average panel

member scores were averaged in turn, to calculate a panel

total score or each thesis.

3.2 Preliminary analyses

3.2.1 Sample characteristicsTable 3.1 shows the dierences in supervisors grades

between the samples o MA and RMA theses. The 37 RMA

theses have signicantly higher grades than the 37 MA

theses included in the study. This indicates a strong eect o

the type o study programme (MA or RMA) on supervisors

grades. However, or the samples o theses o Media Studies

and those o Economics and Business, no signicantdierences in supervisors grades between MA and RMA

theses were ound.

Table 3.1 Dierences in supervisors grades between

MA and RMA theses.

3.2.3 Internal consistency o the measurementsWith Cronbachs ranging between .79 and .96, the

composite scale appears to be highly reliable or all

sub-samples. The reliability coecients ranging rom

.69 to .81 ound or the our assessment panels total

scores, indicate satisactory to good levels o inter-

rater agreement in the scores on the composite scale.

Table 3.2 Mean scores (M) and standard deviations (SD) in assessment panel ratings.

P&N (N=18) MS (N=18) CS (N=18) E&B (N=20)

Standards M SD M SD M SD M SD

Anchoring in current knowledge 3.39 .43 2.78 .67 2.71 .65 3.25 .55

Problem defnition and hypothesis 3.26 .48 2.69 .82 2.84 .60 2.90 .57 Research design 3.11 .60 2.39 .53 2.58 .77 2.84 .81

Data collection and analysis 3.17 .67 2.43 .64 2.66 .87 2.86 .54

Inormation and argumentation 2.94 .75 2.61 .56 2.82 .63 2.93 .65

Structure, wording, presentation 3.20 .54 2.52 .74 3.12 .70 2.85 .71

Relevance 3.15 .71 2.80 .81 2.89 .63 2.90 .59

Creativity and originality 3.04 .76 2.44 .49 2.64 .58 2.67 .76

Over-all quality 3.15 .68 2.67 .71 2.79 .63 2.92 .68

Panel total score 3.16 .55 2.59 .58 2.73 .58 2.90 .54

MA RMA

N M SD N M SD d

Psychology & Neuroscience 9 7.9 1.10 9 9.1 .60 1.41c

Media Studies 9 7.5 .35 9 7.6 .42 .26d

Communication Science 9 7.5 .54 9 8.6 .43 2.27a

Economics & Business 10 7.9 1.00 10 8.0 .83 11d

Total

37

7.7

.81

37

8.3

.82

.75b

a p


31/84


3.3 Primary analyses

3.3.1 Dierences in quality between MA andRMA thesesTwo series o analyses were perormed to test or dierences

in quality between the samples o MA and RMA theses.

Table 3.4 presents the results o a rst series o t-tests,

indicating that, on average, RMA theses were given

signicantly higher ratings than MA theses. These statistics

indicate a very powerul eect o the type o study programme

(MA or RMA) on the ratings in all our assessment panels.

That is, signicant dierences in panel total scores were also

ound or those programmes (Media Studies and Economics

& Business), in which there were no signicant dierences in

supervisors grades between MA and RMA theses.

3.2.4 Ranking the thesesEach thesis included in the study was ranked according to

its panel total score rating. Table 3.3 presents the results or

each o the our assessment panels. As would be expected,

RMA theses are predominantly ound in the top hal o the

rankings, and MA theses are predominantly ound in the

bottom hal o the rankings. For each o the our assessment

panels, the proportion o correctly ranked theses is expressed

as a p-value, ranging rom .67 or the Media Studies panel

to .90 or the Economics & Business panel. These resultsindicate that the MA and RMA theses could adequately be

distinguished based on panel total scores, suggesting that

the measurements have a satisactory level o external

validity.

Table 3.3 MA and RMA theses as ranked by the our

assessment panels.

Table 3.4 Dierences in panel total scores between MA and RMA theses or each o the our study sites.

MA RMAM SD min max M SD min max d

Psychology & Neuroscience 2.83 .48 1.89 3.52 3.49 .42 2.81 4.00 1.47b

Media Studies 2.31 .48 1.67 3.15 2.87 .55 2.26 3.78 1.08c

Communication Science 2.25 .34 1.83 2.83 3.20 .31 2.80 3.57 2.89a

Economics & Business 2.48 .30 2.07 3.07 3.33 .37 2.75 3.89 2.51a

Total 2.47 .45 1.67 3.52 3.22 .46 2.26 4.00 1.66a

a p


32/84

32 Peaks in sight

3.4 Discussion

3.4.1 ConclusionsTo what extent is there a dierence in quality between

theses written by students enrolled on research masters

programmes (RMA) and those written by students enrolled

on regular masters (MA) programmes?

We have seen a marked dierence in the quality between MA

and RMA theses. First, when theses were ranked according

to assessment panels ratings, RMA theses were in the top 3or all study sites, and in the top 5 or all but Media Studies.

Secondly, we ound that assessment panels ratings are,

on average, signicantly higher or RMA theses than or MA

theses. Thirdly, when comparing MA and RMA theses with

similar supervisors grades, assessment panels ratings were

consistently higher or RMA theses than or MA theses.

How does the quality o Dutch (research) masters theses

relate to international standards?

An answer to this question must be a tentative one. There is

no such thing as international standards in the sense o

an internationally accepted assessment ramework with

xed standards. However, by employing panel members

rom outside the Netherlands, we have made sure that inter-

national norms have been applied in assessing the theses.

All panels agree that the MA theses assessed are in line with

international standards. To this, the Communication Science

panel adds that most o the RMA theses assessed are

certainly above average in comparison to international

standards. The other three panels all state that a (vast)

3.3.2 Dierences in quality between theses with similarsupervisors gradesCurious about whether signicant dierences in the assess-

ment panels ratings would also be ound when comparing

theses with similar supervisors grades, we perormed a

second series o t-tests. To this end, we rst analysed the

distribution o supervisors grades in the total sample o

74 theses. Aiming to orm groups o roughly equal size, we

decided on a division o ve levels, subsequently allocating

each thesis to one o these ve levels: Level 1 (includingtheses with supervisors grades 6-7), Level 2 (including

theses with supervisors grade 7.5), Level 3 (including theses

with supervisors grade 8.0), Level 4 (including theses with

supervisors grade 8.5), and Level 5 (including supervisors

grades 9-10).

Table 3.5 presents the results o these analyses. Considering

the number o theses allocated to each level, it becomes clear

that there is a large dierence in the way that supervisors

grades are distributed in the MA and RMA samples. The large

majority (60%) o MA theses were allocated to Levels 1 and 2,

whereas hal o all RMA theses were allocated to Levels 4

and 5.

Signicant dierences were ound or three o the ve groups

(Levels 2, 3 and 4). For Level 1 and Level 5, the dierence in

sample sizes was too large to perorm a t-test. The results

o these analyses also point to a powerul eect o the type

o study programme on the assessment panels ratings.

Furthermore, these results suggest that supervisors apply

stricter norms or RMA theses than or MA theses.

Table 3.5 Dierences in panel total scores between MA and RMA theses or fve levels o supervisors grades.

MA RMA

Grade level N M SD min max N M SD min max d

Level 1 (6 7) 11 2.35 .38 1.83 2.85 4 2.77 .25 2.44 3.06 1.32d

Level 2 (7.5) 11 2.35 .45 1.67 3.15 5 3.24 .53 2.44 3.78 1.81b

Level 3 (8.0) 7 2.43 .42 1.93 3.11 9 3.14 .57 2.26 3.80 1.45c

Level 4 (8.5) 4 2.70 .38 2.26 3.19 6 3.14 .15 2.93 3.33 1.66c

Level 5 (9 10) 4 2.96 .52 2.27 3.52 13 3.45 .42 2.80 4.00 1.05d

Total 37 2.47 .45 1.67 3.52 37 3.22 .46 2.26 4.00 1.66a

a p


33/84


At least three interacting actors can be considered to

contribute to the quality o a thesis:

> the quality o the student writing the thesis,

> the quality o the curriculum in which the student is

enrolled, and

> the quality o the teacher acting as the thesis supervisor.

Some o the salient dierences between RMA and MA

programmes regarding these three actors seem obvious

candidates or explaining the dierence in thesis quality:> The quality o the student: most MA programmes have no

other admission requirement than a relevant BA-degree,

whereas enrolment into RMAs is reserved or students with

above-average academic achievements and motivation.

> The quality o the curriculum:the RMA programmes have a

considerably stronger ocus on the acquisition o research

abilities than regular MA programmes. Also, MAs have a

programme length o one year (60 EC), whereas RMAs have

a programme length o two years (120 EC). This allows RMA

students more time to deepen and widen their substantive

and methodological knowledge, and more time to work on

their thesis.

> The quality o the teacher:the core teaching sta in RMAs

consists o highly qualied and experienced researchers,

constituting academic environments, which oten are o a

considerably higher quality level than those in which regular

MA programmes are oered.

The underlying actor that seems to account or all the

dierences in the quality o students, curricula, and teachers

is: high expectations. In comparison with regular MA

programmes, substantially higher expectations are put on

the enrolling students, on the teachers, and on the intended

learning outcomes o RMA programmes. RMA research

internship assignments appear, on average, to be morechallenging than MA assignments. Consequently, RMA

assignments would have greater potential or developing

into top theses. Some o the actors discussed above may

contribute to enabling students to actually make a

challenging assignment into a top thesis. One o these actors

could well be the very much larger amount o courses and

skills training that RMA students can build rom by the time

they embark on their research assignment.

majority o theses assessed are o top international level.

In act, the Media Studies panel rated three RMA as excellent,

although the sample included only average theses (with

supervisors between 7 and 8) and did not include the best

theses available (with grades o 8.5 or higher). Thereore, it

would be sae to conclude that the quality o Dutch (research)

masters theses relate avourably to international standards.

3.4.2 Strengths and limitations o the study

StrengthsThe assessment panels agreed that the overall procedure

made sense and that blind assessment is a useul method

or objectively identiying dierences in thesis quality.

Neither panel members nor secretary knew beorehand which

were the (R)MA theses. However, one o the assessment

panels could identiy a small number o theses as RMA,

either by indications in the text, or because o the ormat. In

these instances, the panel decided to ignore the indications.

A second strength is the level o internal consistency in the

measurements o thesis quality. A third and last strength is

the apparent external validity o the TAS measurements o

thesis quality. When the theses were ranked according to the

assessment panels ratings, RMA theses were predominantly

ound in the top hal o the rankings.

Limitations

One limitation o TAS lies in the design o the assessment

ramework and the accompanying rating scale. Aiming or a

concise and practical ramework consisting o a limited set

o quality criteria, necessarily caused several criteria to be

multidimensional.

TAS results may have been infuenced by sampling arteacts.

First, given limited time and money, we could only select asmall number o theses. Larger numbers would certainly

contribute to the representativity o the sample. Second,

with the assessment panels being composed o international

experts, the sample could only include theses written in

English. Third, our sample design did not prevent large die-

rences in the distribution o supervisors grades between the

samples o MA and RMA theses. Ideally, the samples o MA

and RMA theses would have represented the ull variability

in quality present in the two populations, yet displaying no

signicant dierences in supervisors grades.

3.4.3 ImplicationsWhich actors are responsible or the dierence in quality

between RMA and MA theses? As our study did not

specically address this issue, we can only propose

tentative explanations.


34/84

34 Peaks in sight

Interview


35/84


Helena Cousijn

Cognitive Neuroscience

Two years ago, Helena Cousijn completed the research

masters programme Cognitive Neuroscience at Radboud

University Nijmegen. Currently, she is enrolled on a PhD

programme at the University o Oxord: My interest in

research was awakened well beore my studies started.

I did a bachelors programme in Cognitive Psychology,beore I started with Cognitive Neuroscience in Nijmegen.

Research on the brain and nervous system is the main ocus

o that programme. I wanted to elaborate on my oundational

knowledge o psychology by increasing my knowledge o

neurobiology. The rst year o the curriculum consists o a

broad range o dierent courses. The second year is ully

dedicated to the research internship.

The programme exceeded my expectations. It was hard work,

because the expectations are certainly high. But the quality

was excellent; the programme goes very deeply into the

subject matter. When I now compare it with courses I did as a

PhD student at the University o Oxord, I am still amazed at

the proundity o the courses I did in Nijmegen. The internship

was very good as well; I had a high degree o autonomy and

learned a lot. It also resulted in a ew publications.

During my internship, I studied a certain gene which we

supposed to be somehow related to stress. A group o

subjects was genotyped, put in a scanner, and then exposed

to some very unpleasant videos which were intended to

induce stress. We have published the results in PNAS,

Proceedings o the National Academy o Sciences o theUnited States o America, an infuential multidisciplinary

periodical. Someone else has now used the data rom that

video study or a publication in Science, which I co-authored.

Two years ago, I started in Oxord. I continue to work in the

same eld, still investigating how genes infuence brain

activity. The dierence is that I now study a more psychiatric

subject: schizophrenia. My main question is how a certain

gene called ZNF804a could lead to the development o

schizophrenia. To examine the eects o that gene, I combine

neuroimaging techniques with methods to study post-mortem

human brains on a molecular level.

Also in the uture, science will continue to be my avourite

sphere o activity. I dont know whether that will be in

the Netherlands, I have not made any specic plans yet.

Science is an international endeavour, so I can very well

imagine mysel ending up somewhere else.


36/84

36 Peaks in sight

Chapter 4 Studentsatisactionin MA and

RMAprogrammes


37/84


especially more positive about the academic skills acquired

through the programme, about their teachers and lecturers,

and about the academic guidance, support and supervision

oered by the programme. However, RMA students are less

positive about their study load, the programme timetables,

the extent to which they learn how to work together with

others, the availability and aordability o accommodation

in the city where they are studying, and about several acilities

in their institution and student-city environment.

4.1 Overall ratings

Table 4.1 indicates that the overall ratings o MA students

are quite positive. However, RMA students gave signicantly

higher ratings on most o these 15 items. For each item,

the standardised mean dierence (Cohens d) is calculated,

indicating the extent to which MA and RMA students dier in

their average ratings. When asked or an overall rating, RMA

students seem especially more positive about the general and

academic skills learned in the programme, the preparation

or a proessional career, the teachers and lecturers in the

programme, the academic guidance, support and supervision

oered, and about the extent to which they are encouraged to

help improve the programme.

To what extent do MA and RMA studentsdier in satisaction with their studyprogramme? Which aspects o the studyprogramme are RMA students moresatised about than MA students? Andwhich aspects are they less satisedabout?

Student satisaction is an important indicator o the quality

o study programmes. Using data rom the Dutch National

Student Survey (NSE; Muskens et al., 2010), we aimed to

identiy dierences in satisaction between MA and RMA

students. First o all, the respondents were asked about their

overall satisaction with their study programme. They were

asked to provide an overall rating or their study programme in

general. Subsequently, the respondents were asked to provide

a rating o 14 aspects o their study programme. For each o

these 14 aspects, several sub-questions ollowed. Each item

was rated on a 5-point scale (1 = very dissatised, 5 = very

satised). The ratings o the sub-questions were used to

calculate a theme score or each o the main aspects.

Our results indicate that MA and RMA students are generally

quite satised with their programme, although satisaction is

somewhat higher in RMA students. RMA students are

Table 4.1 Dierences between MA and RMA students overall ratings.

MA RMA

N M SD N M SD d

Study programme in general 35,793 3.90 .80 2,858 4.02 .85 .15a

Content o the programme 35,042 3.91 .80 2,823 4.00 .85 .12a

General skills 34,965 3.74 .84 2,817 3.97 .84 .28a

Academic skills 34,994 3.85 .85 2,822 4.19 .86 .40a

Preparation or a proessional career 34,430 3.20 1.03 2,762 3.56 1.05 .35a

Teachers/ lecturers 35,019 3.86 .80 2,822 4.07 .81 .26a

Inormation provided 34,954 3.48 .93 2,812 3.50 1.01 .03d

Study acilities 34,782 3.65 .93 2,811 3.80 .96 .16a

Assessment and grading 34,962 3.53 .86 2,812 3.65 .88 .13a

Programme timetables 34,854 3.70 .88 2,799 3.71 .90 .02d

Study load 34,976 3.59 .83 2,816 3.57 .87 -.03d

Academic guidance, support and supervision 34,247 3.36 1.01 2,797 3.69 1.05 .33a

Other acilities and student-city environment 34,336 3.65 .86 2,767 3.75 .88 .12a

Overall atmosphere 34,953 4.02 .82 2,818 4.12 .91 .12a

Encouragement to help improve the programme 34,229 3.40 .99 2,791 3.76 1.05 .37a

a p.05


38/84

38 Peaks in sight

Table 4.2 Satisaction with the content o the study programme.

Table 4.3 Satisaction with the general skills acquired in the study programme.

Table 4.4 Satisaction with the academic skills acquired in the study programme.

MA RMA

N M SD N M SD d

Content o the programme (theme score) 34,654 3.74 .62 2,811 3.90 .68 .27a

Level o the programme content 34,656 3.89 .83 2,809 4.12 .85 .28a

Meets the initial expectations 34,582 3.75 .92 2,806 3.80 1.04 .06b

Fits in with education 33,885 3.77 .90 2,780 3.83 .95 .06b

Content is stimulating 34,635 3.81 .89 2,811 4.03 .94 .25a

Content draws on current developments 34,441 3.87 .88 2,769 3.97 .91 .11a

Coherence o the programme 34,581 3.69 .87 2,804 3.68 .95 -.02d

Instructional ormats 34,534 3.67 .85 2,797 3.82 .87 .17a

Opportunity to shape your own programme 34,060 3.34 1.07 2,782 3.84 1.11 .46a

Quality o the study materials 34,560 3.82 .76 2,789 4.04 .78 .30a

a p


39/84


4.2 Content o the programme o study

Table 4.2 shows that, on average, MA students are quite

satised with the content o their programme. However, RMA

students are a little more positive, especially about the level

o the programme content, the extent to which the programme

is stimulating, the opportunity to shape their own programme

o study, and the quality o the study materials. No signicant

dierence was ound in MA and RMA students ratings o the

coherence o the programme.

4.3 Acquisition o general skills

Table 4.3 displays the dierences between MA and RMA

students satisaction with several general skills acquired in

the study programme. Again, although MA students appear

quite satised, RMA students gave signicantly higher ratings

or most o these items. However, RMA students are less

satised than MA students about the extent to which they

learn how to work together with others.

4.4 Acquisition o academic skills

As may be expected, RMA students are clearly more satised

than MA students with the academic skills acquired in the

programme. Table 4.4 shows that the standardised mean

dierences (Cohens d) between MA and RMA students are

larger or these items than or all the other items in the

questionnaire.

4.5 Preparation or a proessionalcareer

For both MA and RMA students, the preparation or a

proessional career is among the least positively ratedaspects o the study programme. However, as displayed

in Table 4.5, RMA students are moderately more satised

than MA students.

4.6 Teachers & lecturers inthe programme

For both MA and RMA students, the quality and availability

o teachers and lecturers is among the most positively rated

aspects o the study programme. However, the results

presented in Table 4.6 indicate that RMA students are a

little more positive about their teachers than MA students,

especially about their teachers engagement with students.

4.7 Assessment and grading

Table 4.7 shows that RMA students are more satised

than MA students with the extent to which assessments

and grades are aligned with programme content and the

turn-around time or assessments and grades. However,

compared to MA students, RMA students are less satised

with the clarity o the grading criteria. Although signicant,

the dierences between the two groups o students are small.

4.8 Study load

Table 4.8 indicates that both MA and RMA students are

quite positive about their study load. However, on average,

satisaction with study load is somewhat lower in RMA

students.

4.9 Academic guidance, support andsupervision

As may be expected, RMA students are more satised than

MA students with the quality and availability o academic

guidance, support, and supervision oered by the programme.

Table 4.9 indicates that the dierences in the average ratings

between the two groups o students are quite substantial.


40/84

40 Peaks in sight

Table 4.5 Satisaction with the study programmes preparation or a proessional career.

Table 4.6 Satisaction with the teachers/lecturers in the programme.

Table 4.7 Satisaction with the assessment and grading in the study programme.

MA RMA

N M SD N M SD d

Teachers/lecturers (theme score) 33,583 3.74 .68 2,754 3.99 .67 .36a

Teachers'/lecturers' subject area expertise 33,648 4.20 .73 2,757 4.42 .71 .31a

Quality o teaching methods 33,622 3.68 .85 2,756 3.80 .85 .14a

Teachers'/lecturers' availability outside o class hours 32,487 3.67 .97 2,723 3.97 .93 .32a

Teachers'/lecturers' engagement with students 33,558 3.69 .95 2,755 4.04 .92 .37a

Quality o supervision 33,425 3.68 .88 2,736 3.95 .88 .31a

Quality o eedback 33,471 3.55 .94 2,751 3.70 .98 .17a

Extent to which teachers/lecturers inspire you 33,613 3.66 .92 2,752 3.94 .92 .30a

Teachers'/lecturers' knowledge o proessional practice 33,038 3.83 .92 2,671 4.09 .89 .29a

a p


41/84


Table 4.8 Satisaction with the study load.

Table 4.9 Satisaction with the academic guidance, support and supervision.

4.10 Other aspects

The study also covered other aspects o students satisaction,

as shown in the overall ratings in Table 4.1. Since these

aspects are less central to a comparison o MA and RMA

programmes, we will cover them in less detail.

Inormation provided by the institution

RMA students give higher ratings than MA students or theinormation provided by the institution about the programme

and the students progress through the programme. However,

the dierences between MA and RMA students are small.

Study acilities and inormation acilities

MA and RMA students are equally positive about their library,

their digital learning environment, and their inormation desk.

RMA students give signicantly higher ratings or other study

acilities (teaching rooms, student work areas, learning

materials, ICT acilities) and their student administration unit

than MA students. However, the dierences between the two

groups o students are small.

Programme timetables

RMA students give slightly lower ratings than MA students

or the timely publication o the programme timetables and

changes in the programme timetables. However, MA and

RMA students are equally satised with the suitability o their

timetables.

Other acilities and the student-city environment

Compared to MA students, RMA students are less satisedwith the restaurant and canteen acilities at their institution,

the catering acilities in the city where they are studying, and

with its range o cultural acilities. RMA students do give

slightly higher ratings or the ease with which they can reach

their institution and the amount o green space in the city

where they are studying. MA and RMA students are equally

satised with the sports acilities at their institution.

Both MA and RMA students are somewhat negative about

the accommodation in the city where the

Documents

Peaks in sight: Research Master Review 2011