External reviews of

Quality assurance agencies

Perspective of the expert team

Thierry Malan

• I. THE CONTEXT

• II. THE REVIEW PROCESS

• III. COMPLIANCE WITH the ESG and ENQA MEMBERSHIP CRITERIA

• IV. Conclusions

• Part I. The Context

The Agency in the network of institutions in charge of HE in the country

• division of labour with relevant stakeholders in higher education, especially MHE and funding authorities, the higher education institutions, public/private HEI

• the cultural and historical context of the agency’s work : the importance of legal frameworks (two possible views : lack of independence or more strength/credibility towards HEI and the public)– number of factors beyond scope of the agency

– its decision-making powers and procedures

The Quality Assurance Institutional context

– Different situations in the HE System and different places in present internal HE debates

• a single national agency with exclusivity• several national public agencies :

specialized/general - overlaps• several, including private and/or foreign

agencies – with missions precised by law, with funding and

formal inclusion in national HE policy and decision-making system, or only minimal verification

• PART II

• THE REVIEW PROCESS

Preparation of the Review process on ENQA’s side

• The Guidelines for national reviews of ENQA member agencies (September 2006)

• Preparation of TOR and contract with the Agency

• Guidelines on the Review Process. Efficient preparation tools :– the briefing pack– the Review report template– a telefon conference between chairs and secretaries of

the ENQA panels and/or between a panel’s members

Composition of the Review Panels

– Four or five external reviewers and a review secretary

– two quality assurance experts, – representative(s) of higher education institutions, – A student member

– Diversity of professionnel origins and cultural backgrounds/biases within the team : getting together

– The experts have to sign a non-conflict of interest statement as regards the review

– Necessity at least one panel member knowledgeable on the country’s HE system, culture, language

Preparation of the process on the Agency’s side

• The Internal assessment process: the self evaluation report (SER) – First step and key piece of the process

• a detailed and useful periodical analysis, even if the agency was not undergoing an external review

– In itself a valuable indicator• reasonably self-critical if to be useful : presentation and

comments of weaknesses/strengths, ongoing improvements

• Additional documentation

Preparation of the process on the Review Panel’s side

• Making use of the SER, documentation and evidences provided by the Agency. Request of additional documents

• Internet communication between panel members : Developing lines of inquiry and outline report

• Meeting and distribution of roles within the panel

• Little time during the site-visit : Necessity of a strong advanced preparation

Preparation of the site visit

– Communication of the Agency with the panel : a member of the Agency with the panel’s secretary

– Identification, organization and timing of meetings, interviews

– Facilities– Additional information available during the site visit– Languages : use of own language and/or English– feed-back meeting– Internal panel coordination during the review

THE REVIEW REPORT

• The draft report : Secretary and Chair– consideration of the context of the Agency, of the

contents and structure of the self-evaluation report – compliance with ESG, – recommendations

• An interactive process within the review panel• An interactive process with the Agency :

– verification of factual accuracy and comments on the draft report

The final external assessment report

• Revision after the Agency’s comments

• Communication within the panel on eventual adaptations of the draft report

• Finalization

• Communication to the Agency and ENQA

• Feedback letter to ENQA board

PART III

COMPLIANCE WITH THE ESG AND ENQA MEMBERSHIP CRITERIA

Common understanding of the European Standards and Guidelines

1) Consider interpretation of the ESG by the Agency and the Review Panel

2) Contextual appraisal : taking in consideration the Agency’s environment and its specific missions

3) Qualitative and quantitative points of view : assessing the levels of compliance : partially/substantially/fully ?

4) Compliance with ESG and/or development and improvement : how much is enough ? measurement and/or qualitative appraisal ?

• Compliance on every ESG • Global appraisal on them altogether • Assessment of compliance with ESG

against appraisal on the efficient fulfilment of the Agency’s missions.

• Examples of issues of interpretation on some ESG :

ESG 3.2 Official status

The Agency should be formally recognised by competent public authorities in the European Higher Education Area as an Agency with responsibility for external quality assurance and should have an established legal basis. It should comply with any requirements of the legislative jurisdiction within which it operates.– Status often very extensive and detailed : including

mission statement, procedures, etc..

 

ESG 3.3 Activities

• The Agency undertakes external quality assurance activities (at institutional or programme level) on a regular basis. These may involve evaluation, review, audit, assessment, accreditation or other similar activities and should be part of the core functions of the Agency.

• In undertaking its activities, the Agency should take into account the presence and effectiveness of the external quality assurance processes described in the ESG.

• The Agency should have performed these core activities for at least two years

Diversity of Missions/Activities of agencies

• From one activity (for example accreditation of study programmes in a specific academic field) to an (almost) all encompassing evaluation mission– relative importance in status and missions,

agency’s actual policy, planning and operations, – balancing all activities load– If multiplicity of tasks, which are the agency’s core

functions ? What are the priorities ?

ESG 3.4 Resources

• The Agency should have adequate and proportional resources, both human and financial, to enable it to organise and run its external quality assurance process(es) in an effective and efficient manner, with appropriate provision for the development of its processes and procedures.

Budget, staffing, at head office and for the different types of activities depend on

• various parameters for each type of activities, set by law and/or decided upon by Agency : number of institutions/programmes to evaluate, set duration of evaluation cycles (4, 5, 6 years ?), importance of other activities (training, transversal system-wide surveys, international)

• average time and resources required, number of experts to be involved in one type of action : for a small/big HEI, a single study programme, a programme cluster, programme samples

ESG 3.5 Mission statement

• The Agency should have clear and explicit goals and objectives for its work, contained in a publicly available statement.

• This statement should describe the goals and objectives of the member’s quality assurance processes, the division of labour with relevant stakeholders in higher education, especially the higher education institutions, and the cultural and historical context of its work.

ESG 3.6 Independence

• The Agency should be independent to the extent both that it has autonomous responsibility for its operations and that the conclusions and recommendations made in its reports cannot be influenced by third parties such as higher education institutions, ministries or other stakeholders.

A precisely delimitated notion of independence (autonomous responsibility

for operations and conclusions)

• still subject to some interpretation and variations : documents and practices

• In particular to be viewed together with ESG 3.2. Official status and ESG 3.5 Mission statement,

• with consideration of the institutional, historical and cultural context of HE system and the network of relations and distribution of competencies of the agency with MHE and HEI

Conclusions : Still a learning and emergent process

– A great variety of external review reports (composition/backgrounds of panels, styles of reports, articulation between findings, consideration of evidence, assessment of compliance, recommendations)

– Principal purpose, backbone of the report, is assessment of compliance with ESG

– Additional functions/value of external reviews and recommendations for the Agency

– General appraisal on efficient fulfilment by the Agency of its missions

Questions for ENQA

• External reviews reports distinct from ENQA/Register decisions

• No uniformisation but harmonization• Ensure equality of treatment of agencies in

consideration of External Review reports• Bring more comparability/homogeneity of

criteria across reports and their proposals• Deepen the issue of compliance :

– associate qualitative and quantitative points of view – further discuss and precise the meaning and extent of

“substantial » and « full » compliance”