Embed Size (px)
Citation preview
FINNISH HIGHER EDUCATIONFINNISH HIGHER EDUCATIONFINNISH HIGHER EDUCATIONFINNISH HIGHER EDUCATIONFINNISH HIGHER EDUCATIONEVALUATION COUNCILEVALUATION COUNCILEVALUATION COUNCILEVALUATION COUNCILEVALUATION COUNCIL
Audits of Quality AssuranceSystems of Finnish HigherEducation InstitutionsAudit Manual for ����–����
ISBN 952-206-030-5 (paperbound)
ISBN 952-206-031-3 (pdf)
ISSN 1457-3121
Publisher: The Finnish Higher Education Evaluation Council
Cover: Juha Ilonen
Layout: Pikseri Julkaisupalvelut
Tammer-Paino Oy
Tampere 2006
Preface
Quality assurance of higher education is seen to constitute one of the keydevelopment areas in efforts to construct a European Higher Education Areaby 2010. European countries are developing their own national solutions forevaluating and demonstrating the quality of degrees in line with the objectivesof the Bologna process.
After consultations with the Finnish Ministry of Education and theuniversities and polytechnics, the Finnish Higher Education EvaluationCouncil (FINHEEC) launched a project for auditing the quality assurancesystems of higher education institutions (HEIs) and thereby supporting theHEIs in improving their quality assurance. The purpose of auditing is to ensurethat the HEIs have quality assurance systems in support of continuous andsystematic improvement of operations, and that such a system works accordingto stated objectives, brings about change and has international credibility.
In addition to auditing, FINHEEC disseminates related information,organises seminars on the auditing of quality assurance systems in co-operationwith the HEIs, and provides auditor training.
The Berlin Conference of Ministers of Education in 2003 invited theEuropean Network for Quality Assurance in Higher Education (ENQA)1 andEuropean higher education co-operation bodies to prepare Europeanstandards, procedures and guidelines for quality assurance in higher educationfor the Bergen Conference of European Ministers Responsible for HigherEducation. The report submitted to the ministerial conference puts forwardsproposals for European co-operation, principles of national quality assurance,best practices of HEI quality assurance audits or comparable procedures, andgeneral standards and guidelines for quality assurance.
The auditing and development of quality assurance systems are in the bestinterest of the HEIs. Quality assurance forms part of management, strategicwork and internal performance management in HEIs. Quality assurancegenerates quality-related information on education and degrees, whichbenefits society at large, taxpayers and employers, as well as HEI staff, studentsand applicants. Effective quality assurance, which comprises both qualitymanagement and quality enhancement, also contributes to students’ rights andtheir opportunities for participating in the development of education.
1 In November 2004 the General Assembly transformed the Network into the EuropeanAssociation for Quality Assurance in Higher Education (ENQA).
Analytical description and systematic evaluation of QA systems make it easierfor HEIs to convince their national and international partners about thequality of their education provision, also making it more attractive for students.
The audit model presented in this manual, which has been developed inco-operation with representatives of the Finnish universities and polytechnicsand their students, is founded in the principle of enhancement-led evaluation,which is the universally accepted approach in Finland. The planning of themodel has also been informed by the ENQA Standards and Guidelines forQuality Assurance in the European Higher Education Area.
In spring 2005, FINHEEC piloted the audit methods and criteria.Feedback from the pilot institutions and auditors has been used in thepreparation of the audit manual and audit procedure. Audits will be undertakenon the basis of applications from HEIs and within the scope of FINHEECresources. This audit manual will be effective until the end of 2007 unlessotherwise indicated by FINHEEC.
Ossi V. LindqvistChair of the Finnish Higher Education Evaluation Council
Contents
����� Background to the audits of quality assurance systemsBackground to the audits of quality assurance systemsBackground to the audits of quality assurance systemsBackground to the audits of quality assurance systemsBackground to the audits of quality assurance systemsin higher educationin higher educationin higher educationin higher educationin higher education ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ �����
��� European development ____________________________________________ ���� The Finnish model ________________________________________________ �
����� Audits of institutional quality assurance systemsAudits of institutional quality assurance systemsAudits of institutional quality assurance systemsAudits of institutional quality assurance systemsAudits of institutional quality assurance systems ______________________________________________________________________________________________________________ ������������� Premises _________________________________________________________ ����� Objectives ________________________________________________________ ��� Focus and criteria of the audit _____________________________________ �
Audit processAudit processAudit processAudit processAudit process ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ ������������ The HEI registers for an audit ______________________________________ ���� Audit agreement _________________________________________________ ��� Collecting the audit material _______________________________________ ��� Meeting to prepare the audit visit __________________________________ ���� Audit visit _______________________________________________________ ���� Audit group _____________________________________________________ ��
���� Composition and criteria _____________________________________ ������ Disqualification _____________________________________________ ����� Recruitment _______________________________________________ ����� Auditor training ____________________________________________ ������ Principles and ethical guidelines ______________________________ ������ Remuneration ______________________________________________ ��
�� Audit report ______________________________________________________ ���� Publication of results and feedback discussion ________________________ ��
Audit conclusions and follow�up Audit conclusions and follow�up Audit conclusions and follow�up Audit conclusions and follow�up Audit conclusions and follow�up _________________________________________________________________________________________________________________________________________________________________________________________ ����������
����� Development of the audit model Development of the audit model Development of the audit model Development of the audit model Development of the audit model _________________________________________________________________________________________________________________________________________________________________________________________ �����
AppendicesAppendicesAppendicesAppendicesAppendices�: Quality Assurance of Higher Education Institutions in Finland _________ ��: Auditing criteria _________________________________________________ ��: Audit concepts ___________________________________________________ �: The phases of the audit process and their chronological order __________ �: An example of an audit visit programme ____________________________ �
�
� Background to the auditsof quality assurance systemsin higher education
��� European development
International developments, particularly the Bologna process, have broughtnew challenges for higher education development. The European HigherEducation Area (EHEA) has to compete in the global market. Competitionand operation across national borders has led to a situation in which trust inthe standard of higher education within a country is no longer enough.Instead, higher education provided by each country should be transparent andcredible internationally. In particular, student and labour mobility hasheightened the need to substantiate the quality of education and degrees on aninternational scale.
The European Ministers of Education, convening in Bologna in 1999, seta target to realise a coherent and cohesive EHEA by 2010. Finland is involvedin this process along with 44 other countries. Held in Berlin in 2003, thesecond Bologna follow-up meeting set an intermediate development objectivefor 2005, with quality assurance of higher education as one of the priorities.According to the European Ministers of Education, the national qualityassurance systems of higher education should comprise the following by 2005:■ a definition of the responsibilities of the bodies and institutions involved;■ evaluation of programmes or institutions, including internal assessment,
external review, participation by students and the publication of theresults;
■ a system of accreditation, certification or comparable procedures;■ international participation, co-operation and networking.1
1 Berlin communiqué 2003. The original English text can be found at http://www.bologna-berlin2003.de/pdf/Communique1.pdf
�
Many European countries have developed a national quality assurance system.The nature of the higher education system, legislation and steeringmechanisms determine how a country organises its national quality assurancesystem as a part of the higher education system. Countries can be divided intothree categories based on the solutions made: those that audit, those thataccredit, and those that have a combination of the two or a different solutionaltogether. At the time of writing, England, Ireland, France and Scotland2 areexamples of countries that audit. They use different terms to describe theirauditing procedures: institutional audit, quality audit, evaluation of qualityassurance systems or enhancement-led institutional review. Some of thecountries have confined their auditing to quality assurance of education anddegrees, while others audit all quality assurance related to HEI operations.
Countries that accredit include the Netherlands and Germany, whichhave recently adopted a system of degree programme accreditation. In theinitial phase, accreditation applies to new degree programmes and will later beextended to all degree programmes. Norway and Switzerland are examples ofcountries that use a combination of auditing and accreditation. Examples ofanother approach are Denmark and Sweden, which systematically evaluate alldegree programmes at regular intervals3 . What all the European countriesmentioned above have in common is that they have reformed their nationalevaluation systems around the turn of the millennium.
European co-operation in higher education evaluation is co-ordinated bythe European Association for Quality Assurance in Higher Education(ENQA), which the Berlin Conference of Ministers of Education invited todevelop European standards and guidelines for quality assurance. The ENQAworking group also included representatives from the European UniversityAssociation (EUA), the European Association of Institutions in HigherEducation (EURASHE) and the National Unions of Students in Europe(ESIB). The planning and implementation of the pilot phase of auditing inFinland has been informed by the guidelines proposed by the ENQA workinggroup. The working group’s proposals can be divided into: (1) Europeanstandards and guidelines for internal quality assurance of HEIs, (2) Europeanstandards and guidelines for external quality assurance within HEIs, and (3)European standards and guidelines for national and regional bodies responsiblefor external quality assurance (such as FINHEEC in Finland). The most
2 Outside Europe, quality assurance is audited for instance in Australia, New Zealand, HongKong and South Africa.3 Examples of different evaluation and audit models can be found at: www.enqa.net/files/procedures.pdf.
�
relevant in terms of this manual are the standards and guidelines for internalquality assurance of HEIs.4
The ENQA working group defined the standards for internal qualityassurance of HEIs as follows:1. Institutions should have a policy and associated procedures for the
assurance of the quality and standards of their programmes and awards.They should also commit themselves explicitly to the development of aculture which recognises the importance of quality and quality assurancein their work. To achieve this, institutions should develop and implementa strategy for continuous quality enhancement. The strategy, policy andprocedures should have a formal status and be publicly available. Theyshould also include a role for students and other stakeholders.The policy statement is expected to determine:– the relationship between teaching and research in the institution;– the institution's strategy for quality and standards;– the organisation of the quality assurance system;– the responsibilities of departments, schools, faculties and other
units and individuals for the assurance of quality;– the involvement of students in quality assurance;– the ways in which the policy is implemented, monitored and revised.
2. Institutions should have formal mechanisms for the approval, periodicreview and monitoring of their programmes and awards.
3. Students should be assessed using published criteria, regulations andprocedures which are applied consistently.
4. Institutions should have ways of satisfying themselves that the staffinvolved in the teaching of students are qualified and competent.Information about procedures should be available to those undertakingexternal reviews.
5. Institutions should ensure that the resources available for the support ofstudent learning are adequate and appropriate for each degreeprogramme.
6. Institutions should ensure that they collect, analyse and use relevantinformation for the effective management of their degree programmesand other activities.
7. Institutions should regularly publish up-to-date, impartial and objectiveinformation, both quantitative and qualitative, about the degreeprogrammes and degrees they are offering.
4 The document “Standards and Guidelines for Quality Assurance in the European HigherEducation Area” is available at http://www.enqa.net/files/BergenReport210205.pdf.
�
The primary focus of quality assurance in these standards is education, inrelation to which other operations, such as services which support teachingand studying, are considered. Research is touched upon to the extent it has abearing on teaching.
The ENQA working group proposes that external quality assurance, suchas audits, focus on the effectiveness of internal quality assurance. Furthermore,the aims and objectives of quality assurance processes should be determinedbefore the processes themselves are developed and should be published with adescription of the procedures to be used. Any formal decisions made as a resultof external quality assurance should be based on explicit, published criteria.External quality assurance processes should be designed to ensure that theprocedures achieve the aims and objectives set for them. ENQA recommendsclear reporting; a predetermined follow-up procedure accompanyingrecommendations given to HEIs; external quality assurance on a cyclical basis;and summary reports describing and analysing salient findings of audits andreviews.
The ENQA working group also issued European standards concerningthe official status, resources and activities of quality assurance agencies.FINHEEC will use these standards in developing its own activities as part ofnational and European evaluation.
Meeting in Bergen in May 2005, European ministers responsible forhigher education in the Bologna process countries adopted the ENQAstandards and guidelines for quality assurance in the European HigherEducation Area.
��� The Finnish model
The Finnish response to the aims and objectives set in the Berlin communiquéwas deliberated by a committee on quality assurance5 representing HEIs, theirstudents, the Ministry of Education and FINHEEC.
The committee found that, as regards different components of qualityassurance set out in the communiqué, the Finnish system includes a division ofresponsibilities between various bodies and institutions; the evaluation of HEIsand degree programmes; student participation; and participation ininternational co-operation. On the other hand, Finland has not used
5 OPM 2004. Korkeakoulutuksen laadunvarmistus. (Ministry of Education. QA in highereducation) Opetusministeriön työryhmämuistioita ja selvityksiä 2004:6. Helsinki: Yliopistopaino.[The report is available in Finnish, with an English abstract, at www.minedu.fi/julkaisut/koulutus/2004/tr06/tr06.pdf]
��
accreditation as such, certification, or any other clearly demonstrable system asa quality assurance procedure6 .
In its 2004 report, the committee proposed that the universities andpolytechnics develop quality assurance systems covering all their operationsand that these be regularly evaluated by FINHEEC7 . With a view to a clearernational system, the committee recommended a system in which 1) the HEIsare responsible for their own quality assurance, 2) the Ministry of Educationdetermines the evaluation criteria and procedure used in instituting orterminating programmes, and, in special cases, evaluating existing education,and 3) FINHEEC is responsible for evaluating the quality of education andinstitutional performance. Appendix 1 contains a more detailed description ofthe reformed national quality assurance system.
The committee also recommended that Finland adopt the concepts‘quality assurance’ and ‘quality assurance system’. The term 'quality assurancesystem' has two uses: it may refer either to the quality assurance system of anindividual HEI or to the national quality assurance system as a whole. The firstcomprises the quality assurance organisation, the division of responsibilities,procedures, processes and resources within an individual institution. Thenational quality assurance system is composed of the institutional proceduresand processes, FINHEEC’s evaluation activities, the Ministry of Education andlegislation enacted to assure the quality of higher education.
The report of the quality assurance committee was sent on an extensiveconsultation round to universities and polytechnics and to major stakeholdersin education and higher education policy. The comments showed that theHEIs and other stakeholders were in favour of the proposals, which they foundfeasible.
In spring 2004, FINHEEC set up a taskforce to plan auditing and writean audit manual setting out the aims and objects of audits, as well as themethods, evaluation criteria, principles of signing up for audits, and follow-upprocedures. The members of the taskforce were representatives of HEIs,
6 However, a procedure comparable to accreditation was used in the evaluations conductedbefore polytechnics were granted permanent operating licences. These were carried out byFINHEEC from 1997 to 1999. In addition, professional courses have been evaluated on avoluntary basis since 1999, which is a procedure resembling accreditation.7 The committee arrived at this solution because it saw that, as a quality assurance procedure,auditing bolsters the autonomy of HEIs and their diversity. The committee also regardedauditing as an evaluation tool implying trust in HEIs (see e.g. Woodhouse, D. 2003. QualityImprovement through Quality Audit. Quality in Higher Education, Vol. 9, No. 2, July 2003, 133–139).
��
students, employers, and the FINHEEC Secretariat. In the preparatory phase,FINHEEC arranged seminars to introduce the audit model to highereducation institutions and other interested parties and to collect their feedbackon it with a view to refining the audit procedure and the audit manual.
��
� Audits of institutionalquality assurance systems
��� Premises
In higher education, quality assurance refers to the procedures, processes andsystems used by HEIs to manage and improve the quality of their educationand other activities. An audit is an independent external process for evaluatinga quality assurance system.
The main responsibility for quality assurance rests with the HEI. Thequality assurance system must cover all its activities. Each institutiondetermines independently the objectives, organisation, principles, methods anddevelopment of its own system. The aim of FINHEEC is not to harmonise thediverse quality assurance systems according to any particular predeterminedmodel. On the other hand, interaction between institutions is useful also in thismatter. Further, quality assurance and comparison are at the core of growinginternational interaction between HEIs.
Audits examine the quality assurance system at two levels: institutionalquality assurance as a whole and quality assurance relating to the HEI’s mainmission8 , with focus on procedures assuring the quality of degree education inline with ENQA standards.
The audit looks at the qualitative aims a HEI has set and at the processesand methods it uses to manage and enhance the quality of its education andother activities. The aim is to determine if the quality assurance systemfunctions as intended, produces information needed to improve operations andleads to effective improvement measures. The HEI is required to providedocumentation and otherwise substantiate the performance of its qualityassurance system. The audit addresses neither the objectives or the qualityassurance system as such nor the content or outcome of activities as such.Performance is evaluated by the HEI itself and, to a certain extent, within theperformance management process carried out by the Ministry of Education.
8 Under Finnish legislation, the mission of universities and polytechnics is three-fold: 1) degree-oriented and other education, 2) research/R&D and 3) interaction with and impact on societyand contribution to regional development.
�
��� Objectives
The national aim of FINHEEC audits is to support universities andpolytechnics in their quality management and performance enhancement.Auditing processes and public reporting on institutional quality assurancesystems stimulate debate on quality assurance and boost interaction betweenHEIs and their stakeholders. The aim is to share best practices in qualityassurance and disseminate them within the higher education system.
The development of quality assurance in Europe is built on severalimportant premises, which are reflected in our national quality assurancesystem, notably students’ vested interest in the quality of higher education andthe autonomy of HEIs, which in turn entails that they recognise and accepttheir responsibilities vis-à-vis society.
The principle of autonomy entails that quality assurance is reviewed bypeers. The peer review principle in turn means that HEIs also take on majorresponsibility as experts in national quality assurance reviews.
The point of departure in quality assurance audits is enhancement-ledevaluation, which has a robust tradition in Finland. The aim is that auditsconducted by FINHEEC can be used by the auditees as part of their ownquality assurance and that they cause as little extra work as possible for theauditees.
The goal of the audits of institutional quality assurance systems are:■ to evaluate how well the quality assurance system works as a tool for
quality management and enhancement– whether the HEI’s quality assurance system promotes the attainment
of national higher education policy objectives, as well as those set bythe HEI itself
– whether the HEI’s quality assurance system produces usefulinformation for the improvement of its operations and engendersimprovement measures
■ to evaluate the quality assurance system in terms of the audit criteria, tohighlight strengths and best practices, to put forward recommendations forthe development of quality assurance and to determine whether the HEIpasses the audit or needs to undergo a re-audit at a later date.
�
�� Focus and criteria of the audit
Auditing focuses on two levels: the HEI’s quality assurance as a whole andquality assurance relating to the HEI’s basic mission. Auditing assesses thecomprehensiveness, performance, transparency, and effectiveness of the systemand determines how the HEI monitors, evaluates and develops it.
Auditing targets:
1. Objectives, overall structure and internal coherence of the qualityassurance system
2. Documentation, including the formulation of quality policy and thedefinition of procedures, actors and responsibilities
3. Comprehensiveness of quality assurance:a) Degree education9
b) Research/R&Dc) Interaction with and impact on society, and contribution to
regional development10
d) Support and other services (such as library and information services,career and recruitment services, and international services)
e) Staff development4. Participation of staff, students and external stakeholders in quality
assurance5. Interface between the quality assurance system and management/steering6. Relevance of, and access to, quality assurance information within the HEI7. Relevance of, and access to, quality assurance information for external
stakeholders8. Efficiency of quality assurance procedures and structures and their effect
on the development of activities9. Use of information produced by the quality assurance system as a tool for
quality management and enhancement in education and other activities10. Monitoring, evaluation and continuous development of the quality
assurance system.
9 Degree education means the first, second and third cycles of education leading to a degree. Thefirst cycle comprises university and polytechnic Bachelor’s degrees, and the second cycleuniversity and polytechnic Master’s degrees. The third cycle is research training, i.e. in Finlandthe postgraduate Licentiate and Doctorate degrees.10 HEIs' societal and regional mission also includes continuing education, (such as professionalcourses) and open university and open polytechnic education.
��
Criteria
Audits of quality assurance systems employ a set of criteria, with different scalesfor four different stages of development. There are criteria for an absent,emerging, developing and advanced quality assurance system (see Appendix 2).
��
Audit process
The audit process has seven phases:1. The HEI registers for an audit.2. An audit agreement is signed between the HEI and FINHEEC.3. The HEI collects the audit data and other material.4. A meeting is arranged to prepare an audit visit.5. The audit group visits the HEI.6. An audit report is prepared.7. The results are published, followed by a feedback meeting.
Appendix 4 contains a table describing the phases of the audit process inchronological order.
�� The HEI registers for an audit
Each year, FINHEEC undertakes four to eight audits, depending on requestsand the resources available. FINHEEC works with the HEIs to build an overalltimetable for the audits, so that all parties know well in advance of anupcoming audit.
�� Audit agreement
FINHEEC and the HEI sign an audit agreement indicating:■ how the audit will be carried out■ in what timetable■ whether the audit group will be international or domestic and what
language will be used■ how long the audit visit will be (2–3 days depending on the size of the
HEI)■ how the audit will be financed■ the commitment of the auditee to a possible re-audit.
��
� Collecting the audit material
The HEI to be audited collects data and other material needed for the auditfrom its own quality assurance system. The material should provide a sufficientinformation base for the auditors to assess the comprehensiveness,performance, transparency and effectiveness of the quality assurance system.The audit material must also outline the HEI’s organisation, describe thestructure of the quality assurance system and its links with the managementsystem and provide evidence of the performance of the system.
The audit material should include the following two documents:■ basic material for the audit and■ material and samples chosen by the HEI in order to substantiate the
performance of the quality assurance system.
The audit materials should be in the language of the audit project; Swedish-speaking HEIs submit their material in Swedish. In an international audit, thematerial is in English.
Audit materials are primarily collected from existing sources at the HEI'sdiscretion. This preparation process is intended to support HEIs in qualitymanagement and enhancement.
�� Basic materials for an audit
■ a chart, a table or other brief description of the organisation of theinstitution and the number of students and staff (one page)
■ a brief description of the quality assurance system (max. three pages)■ the HEI’s quality manual or other quality documentation in full■ a short history of the quality assurance system (one page)■ a brief description of how the quality assurance system is linked to the
management system (one page)■ the HEI’s SWOT analysis of its quality assurance system (one page)■ the HEI’s summary of major development targets indicated by the quality
assurance system, and projected measures (one page).
�� Material substantiating the performanceof the quality assurance system
In substantiating the performance of its quality assurance system, the HEIshould provide proof concerning each of the ten auditing targets11 . Thematerial should indicate clearly what evidence relates to what target.
11 The auditing targets are listed in section 2.3.
��
The proof can be examples showing■ how review procedures, indicators or feedback systems are used to
monitor the quality of degree education and other activities■ how evaluation findings or indicator data are used to develop operations■ how the quality assurance system has evolved over time and how the
results are communicated within the HEI and to external stakeholders■ how the quality assurance system has influenced the development of
education and other activities.
The HEI submits ten copies of the audit material to FINHEEC approximatelyfour weeks before the audit visit.
The audit group may ask the HEI to provide any additional informationit deems necessary before the audit visit.
The HEI is also requested to give the audit group access to any electronicmaterial relevant to the audit.
� Meeting to prepare the audit visit
About three weeks before the audit visit, the chair of the audit group and theFINHEEC project manager co-ordinating the audit pay a visit to the HEI. Thepurpose is to give the HEI staff and students a chance to discuss the objectives,targets, criteria and implementation of the audit.
�� Audit visit
The aim of the audit visit is to verify and complement the informationprovided by the audit material. The visit is intended as an interactive eventcontributing to the development of the quality assurance system.
The duration of the visit is two or three days. On the first day the auditgroup interviews representatives of the management, teaching and other staff,students, and other stakeholders. The focus is on the quality assurance systemas a whole.
The second day centres on quality assurance in degree education andother activities of the departments and units. The audit group can visit facultiesor departments to verify how comprehensive, effective and transparent thequality assurance system is in practice at the operational level. The audit groupgenerally chooses the sites they visit on the basis of the audit material. Inaddition, they can arrange meetings with different staff and student groups todiscuss themes central to quality assurance.
��
If necessary, the visit can be extended to three days, but it always ends witha meeting with the HEI leadership and management.
Alongside interviews, the auditors acquaint themselves with writtenmaterial relating to the quality assurance system.
Appendix 5 gives the outline of an audit visit, which can be modifiedaccording to need.
�� Audit group
���� Composition and criteria
FINHEEC appoints each audit group separately. As a rule, the group comprisesfive members: three representing HEIs, one students and one externalstakeholders. In their selection, care is taken to ensure that the auditors alsorepresent different higher education sectors (universities and polytechnics) anddifferent staff groups (management and administration, teaching and researchand support services).
The auditors must meet the following criteria:1. solid knowledge of the higher education field2. experience of evaluation/auditing3. knowledge of quality management/quality assurance systems4. participation in auditor training organised by FINHEEC.
Before the appointment of the audit group, the HEI has an opportunity togive its opinion of its composition.
HEIs have the choice of a domestic or international audit group. The roleand number of international auditors is determined on a case by case basis.
���� Disqualification
A person is disqualified as an auditor if he or she is an interested party or ifconfidence in his or her impartiality towards the HEI under review is at issue.This may happen, for instance, if the person is employed by the HEIconcerned or has held a position of trust in its executive body. Auditors havean obligation to inform FINHEEC about anything that might disqualify them.
��� Recruitment
FINHEEC mainly invites experts suggested by HEIs and student organisationsto take part in an audit, but may also consult the FINHEEC database ofexperts, which contains the names and fields of expertise of all those who haveparticipated in FINHEEC evaluations. Recruitment entails participation in
��
auditor training organised by FINHEEC, unless otherwise decided inindividual cases.
The aim is that each auditor participates in at least two audit projects.
��� Auditor training
Special training is arranged for the auditors, with the focus on the objectivesand different phases of the audit process, the responsibilities of the audit group,audit methods and the situation in quality assurance in Finland and abroad.The course can be arranged for 10–15 persons and takes 1.5 working days.
The training focuses on the following matters:■ the Finnish higher education system and the current policy situation■ current situation in international quality assessment■ objectives, methods and criteria of audits■ the ethical and social dimensions of auditing■ the roles of the chair and members■ the implementation of the audit visit■ auditing techniques and questions■ analysis of audit materials and reporting.
Before the audit visit, the audit group meets at least once to discuss the auditagreement and the audit materials submitted by the HEI and to agree on theirrespective responsibilities regarding the visit and reporting.
���� Principles and ethical guidelines
The audit group should observe the following ethical guidelines:1. Auditing must be systematic and based on transparent and intelligible
methods.2. Auditors must have competence in auditing/evaluation and be willing to
improve it.3. Auditors must act impartially and objectively towards the HEI.4. The audit must be based on material accumulated during the audit
process and visit.5. An auditor must be aware of his or her own connections with different
interest and value systems.
���� Remuneration
Auditors are remunerated according to a schedule of fees approved byFINHEEC.
��
�� Audit report
The audit group writes a report based on the analyses made and materialscollected during the audit process. By and large, the reports should follow toa uniform structure, including■ a description of the audit process■ a description of the HEI and its quality assurance system■ audit findings, itemised by auditing targets■ the strengths and best practices of the quality assurance system■ recommendations for improvement■ an overall assessment of the quality assurance system.
Additionally, the report records FINHEEC’s decision that the HEI has passedthe audit or that a re-audit is needed.
The audit report, which should be approximately 50 pages, is published inthe language of the audit process.
�� Publication of resultsand feedback discussion
The audit reports are public and published in the FINHEEC publicationseries. In addition, FINHEEC may publish compilation reports whichsummarise audit findings and analyse audits in Finnish or English.
The results of an audit are published at a seminar jointly organised byFINHEEC and the HEI under review. This offers students and staff anopportunity to discuss the findings and conclusions with the auditors.
��
Audit conclusionsand follow�up
Based on the stated audit criteria and principles, the audit group appraises thefitness for purpose and performance of the quality assurance system, issuingrecommendations for its improvement and highlighting best practices.Moreover, the report records the auditors' conclusion:■ that the auditee has passed the audit, or■ that, measured against the audit criteria, the auditee's overall quality
assurance system or quality assurance relating to its basic mission had somemajor shortcomings, which necessitate development measures andsubsequent re-auditing. In this case, the report indicates the measuresneeded.
Based on the auditors' proposal, FINHEEC formally decides that the HEI’squality assurance system passes the audit or that a re-audit is needed. Asuccessful HEI is awarded an audit certificate indicating that its qualityassurance system has been audited. FINHEEC keeps a register of HEIs thathave signed up for an audit and those that have passed an audit.
As a rule, the audits are conducted at six-year intervals.If the audit group decides to propose a re-audit, the HEI is offered a
chance to discuss the proposed improvements and their order of priority withFINHEEC and the audit group. The re-audit, which will be conducted twoyears after the audit, will concentrate on the proposed improvement measures.
�
� Developmentof the audit model
The audit system was piloted at Kymenlaakso and Pirkanmaa Polytechnics inthe spring of 2005. Feedback collected during the pilot phase has been used inthe design of the audit model and in the drafting of this manual, which will bevalid until the end of 2007, unless otherwise decided before it.
The audit model will be under development up to 2007. From 2005 to2007, each auditor and participating HEI will be requested to give feedback onthe audit methods and criteria. FINHEEC will also discuss the audit modelwith its international partners. This feedback will inform the conclusions onthe further development of the audit model in 2007.
�
APPENDIX �:Quality Assurance of Higher EducationInstitutions in Finland
According to the quality assurance committee�� Finnish national quality assurance inhigher education has three components: national policy steering the institutions’own quality assurance and national auditing� Additionally national qualityassurance is increasingly influenced by a growing need to substantiate quality in theinternational market and by international policy lines�
Figure �� National system of quality assurance in higher education
�� National education policy steering
Within the Government the Ministry of Education prepares matters relating tonational education policy ensures appropriate administration and oversees thesector� Government steering in the higher education sector is based on legislation notably on the Universities Act and the Polytechnics Act and the relevant Decrees�
The factor determining the steering system and the status of HEIs is that both ofthe Finnish higher education sectors have been established by official decisions andaccording to uniform criteria respectively: the universities by legislation and thepolytechnics by means of operating licences granted by the Government� In order toobtain operating licences the polytechnics had to meet � statutory criteria� The pre�licensing evaluations in ����–���� can be seen to represent the accreditation model which has been gaining ground in Europe over the past few years�
The Government is responsible for the core funding of the HEIs� All universitiesare state�funded� Polytechnics are run by municipalities municipal consortia or
▼
▲INTER�NATIONALEDUCATIONMARKET
Government steering(MInistry of Education)
National evaluationFINHEEC
Quality assurance at HEIs
Establishment of institutions: legislation accreditationCriteria: new education termination of old programmes
– steering– decision�making– evaluation by authorities
– national responsibility– programme evaluations
thematic evaluations– audits of QA systems
– main responsibility for qualityand development of education
– institutional QA systems– participation in external
evaluation
��
foundations but the Government pays a significant part of their operatingexpenditure and they are largely steered according to policy outlined by theMinistry�
Finnish higher education policy is based on the Government Programme and onthe Development Plan for Education and Research which complements it and isadopted by the Government every four years� In practice national steering takes theform of performance agreements negotiated and signed by the Ministry and eachHEI every three years� The written agreements which govern the performancemanagement of polytechnics and universities determine major target outcomes andoutline the development of operations for the coming years� The agreements arereviewed annually as regards resources and student intakes� The university database(KOTA) and the polytechnic database (AMKOTA) play an important part in themonitoring included in performance�based steering�
Decisions concerning new education policy openings are taken by the Ministry ofEducation on the initiative of HEIs or important social partners� New universityprogrammes are established by Government Decrees� As regards polytechnics theMinistry approves their degree programmes and possible changes in operatinglicences such as teaching languages and changes of location� Other licence�relatedquestions are decided by the Government� The quality assurance committee alsoproposed that the Ministry of Education set out criteria and evaluation proceduresfor instituting new degree programmes and terminating old ones and for reviewingexisting education in special cases�
Performance management in higher education has evolved into a year�roundprocess with emphasis not only on quantitative objectives but increasingly also onqualitative and strategic objectives� For the Ministry the national objectivescommon to all the HEIs are an important tool for implementing national policy while the institutional objectives promote the strategic aims of an individualinstitution and its profilisation in its particular situation�
�� National evaluation
In Finland external evaluations of HEIs are primarily the duty of FINHEEC� Underthe Decree on the Finnish Higher Education Evaluation Council (���/���� asamended by ��/���� and ��/����) FINHEEC assists HEIs and the Ministry ofEducation in matters relating to evaluation� Assigning the responsibility for externalevaluation in higher education to FINHEEC the Decree clearly gives FINHEEC thestatus of a national quality assurance agency� Universities and polytechnics can alsoparticipate in evaluations organised by other parties if they so wish� Science policyand research are evaluated by the Academy of Finland which is also an importantsource of research funding� The Academy's mission is to improve the quality andprestige of basic research by means of competitive research funding�
�� OPM ���� Korkeakoulutuksen laadunvarmistus� Opetusministeriön työryhmämuistioita jaselvityksiä ���:�� Helsinki: Yliopistopaino [The report is available in Finnish with an Englishabstract at www�minedu�fi/julkaisut/koulutus/���/tro�/tro��pdf]
��
FINHEEC is an autonomous expert body which means that in deciding on andundertaking evaluations it is entirely independent of the HEIs and the Ministry ofEducation which finances its operations� At the beginning of each four�year term FINHEEC determines its evaluation targets objectives and methods� Its policy isnaturally informed by general development needs in Finnish society and in theinternational context�
In its operations FINHEEC has always emphasised the principle of enhancement�led evaluation� This means that the evaluations produce information about highereducation and its quality which can be used in institutional development� Theinformation is also used by the Ministry of Education for example in performancemanagement and decision�making�
FINHEEC evaluations fall into four categories: �) institutional evaluations;�) programme evaluations; ) evaluations relating to national higher educationpolicy objectives and other thematic evaluations; and ) accreditation ofprofessional courses offered by HEIs� In addition to these the Ministry of Educationcommissions evaluations before designating Centres of Excellence in Education�FINHEEC also provides fee�charging services�
Since ��� FINHEEC has prioritised quality assurance audits which havereplaced other institutional evaluations�
� Responsibilities of the HEIs
Under legislation all HEIs are autonomous and thereby also responsible for theeducational arrangements and for their quality� Under Section of the UniversitiesAct (��/��) the universities must organise their activities with a view to attaininga high international level in their research education and teaching and follow ethicalprinciples and good scientific practice in their activities� According to Section � universities must evaluate their education research and artistic activities and theirimpact� Universities must also take part in external evaluations of their operationsand publish the results� Similarly Section � of the Polytechnics Act (��/�) assignspolytechnics the responsibility for the quality and continuous development of theireducation or other operations and for regularly participating in external qualityassessment�
The premise in Finland is that each university and polytechnic can construct aquality assurance system that best meets its needs� Thus each HEI is responsible forits own quality assurance objectives methods and development� From theperspective of education quality it is vital that the HEIs define the objectives ofquality work the procedures and processes for achieving the objectives and themechanisms for assuring quality� This entails extensive public debate and a sharedunderstanding of education quality within each HEI�
��
APP
EN
DIX
�: A
udit
ing
crit
eria
The
crit
eria
are
use
d as
an
indi
cati
ve to
ol in
ass
essi
ng th
e st
age
of d
evel
opm
ent i
n in
stit
utio
nal q
ualit
y as
sura
nce�
AUDI
TING
TAR
GETS
CRIT
ERIA
ABSE
NTEM
ERGI
NGDE
VELO
PING
ADVA
NCED
1. O
bjec
tives
, ove
rall
stru
ctur
eTh
e HE
I has
no
QA p
roce
dure
s in
Ther
e ar
e QA
pro
cedu
res
rela
ting
QA c
over
s m
any
of th
e HE
I'sQA
cov
ers
all o
r nea
rly a
llan
d in
tern
al c
oher
ence
o ft
hepl
ace
rela
ting
to it
s ac
tiviti
es.
to s
ome
HEI a
ctivi
ties,
but
activ
ities
and
the
QA p
roce
dure
sac
tiviti
es, a
nd Q
A pr
oced
ures
qual
ity a
ssur
ance
sys
tem
they
are
nei
ther
sys
tem
atic
form
a fa
irly
effic
ient
sys
tem
.pr
oces
ses
form
a d
ynam
ic w
hole
.in
stru
ctur
e no
r int
erlin
ked.
2. D
ocum
enta
tion,
inclu
ding
the
Qual
ity p
olicy
, pro
cedu
res,
The
defin
ition
and
doc
umen
t-Pr
oced
ures
, act
ors
and
resp
ons-
The
proc
edur
es a
nd d
ivisio
n of
form
ulat
ion
of q
ualit
y po
licy
acto
rs a
nd re
spon
sibilit
ies
atio
n of
resp
onsib
ilitie
s an
dib
ilitie
s ar
e cle
arly,
com
preh
ensib
lyla
bour
man
ifest
ly co
mpl
y w
ith th
ean
d th
e de
finiti
on o
f pro
cedu
res,
have
not
bee
n de
fined
or
proc
edur
es in
clude
d in
the
QAan
d co
ncre
tely
defin
ed, a
nd th
edo
cum
ente
d pr
oces
ses.
The
QAac
tors
and
resp
onsib
ilitie
sdo
cum
ente
d.sy
stem
are
inad
equa
te, a
nddo
cum
enta
tion
is ea
sily
avai
labl
eor
gani
satio
n is
extre
mel
y w
ell-
the
QA
proc
edur
es a
reto
all.
The
QA o
rgan
isatio
nde
signe
d an
d re
info
rces
QA.
inad
equa
tely
org
anise
d.is
wel
l-des
igne
d.
3. C
ompr
ehen
siven
ess
of Q
ANo
QA
in th
e ac
tiviti
es a
ndTh
e sy
stem
cov
ers
isola
ted
activ
ities
The
syst
em c
over
s m
any
activ
ities
In th
e m
ain,
the
syst
em c
over
spr
oces
ses
rela
ting
to b
asic
miss
ion.
and
proc
esse
s m
ainl
y re
latin
gan
d pr
oces
ses
rela
ting
toac
tiviti
es a
nd p
roce
dure
s re
latin
gto
deg
ree
educ
atio
n.ba
sic m
issio
n.to
the
basic
miss
ion.
3 a)
Deg
ree
educ
atio
nNo
QA
in d
egre
e ed
ucat
ion.
QA c
over
s so
me
isola
ted
aspe
cts
ofQA
cov
ers
seve
ral a
spec
ts o
fQA
cov
ers
all s
alie
nt a
spec
tsth
e pl
anni
ng, i
mpl
emen
tatio
n an
dth
e pl
anni
ng, i
mpl
emen
tatio
n an
dof
the
plan
ning
, im
plem
enta
tion
and
eval
uatio
n of
deg
ree
educ
atio
n.ev
alua
tion
of d
egre
e ed
ucat
ion.
eval
uatio
n of
deg
ree
educ
atio
n.
3 b)
Res
earc
h / R
&DNo
QA
in re
sear
ch/R
&D.
QA c
over
s so
me
isola
ted
aspe
cts
QA c
over
s se
vera
l asp
ects
QA c
over
s al
l sal
ient
asp
ects
of re
sear
ch/R
&Dof
rese
arch
/R&D
of re
sear
ch/R
&D
3 c)
Inte
ract
ion
with
and
impa
ctNo
QA
rela
ting
to in
tera
ctio
nQA
cov
ers
som
e iso
late
d as
pect
sQA
cov
ers
seve
ral a
spec
ts o
fQA
cov
ers
all s
alie
nt a
spec
ts o
fon
soc
iety
; and
con
tribu
tion
tow
ith a
nd im
pact
on
socie
ty o
r to
of in
tera
ctio
n w
ith a
nd im
pact
on
inte
ract
ion
with
and
impa
ct o
nin
tera
ctio
n w
ith a
nd im
pact
on
regi
onal
dev
elop
men
tre
gion
al d
evel
opm
ent
socie
ty, a
nd o
f reg
iona
l dev
elop
men
tso
ciety,
and
of r
egio
nal d
evel
opm
ent
socie
ty, a
nd o
f reg
iona
l dev
elop
men
t.
��
AUDI
TING
TAR
GETS
CRIT
ERIA
ABSE
NTEM
ERGI
NGDE
VELO
PING
ADVA
NCED
3 d)
Sup
port
and
othe
r ser
vices
No Q
A in
sup
port
and
othe
r ser
vices
.QA
cov
ers
som
e iso
late
d as
pect
s of
QA c
over
s se
vera
l asp
ects
of s
uppo
rtQA
cov
ers
all s
alie
nt a
spec
ts o
f sup
port
(e.g
. lib
rary
and
info
rmat
ion
serv
ices,
supp
ort a
nd o
ther
ser
vices
.an
d ot
her s
ervic
es.
and
othe
r ser
vices
. c
aree
r and
recr
uitm
ent s
ervic
es,
and
inte
rnat
iona
l ser
vices
)
3 e)
Sta
ff de
velo
pmen
tNo
QA
in s
taff
deve
lopm
ent.
QA c
over
s so
me
isola
ted
QA c
over
s se
vera
l asp
ects
of
QA c
over
s al
l sal
ient
asp
ects
of
aspe
cts
of s
taff
deve
lopm
ent.
staf
f dev
elop
men
t.st
aff d
evel
opm
ent.
4. P
artic
ipat
ion
of s
taff,
stu
dent
sSt
aff,
stud
ents
and
ext
erna
l sta
ke-
Som
e of
the
follo
win
g gr
oups
are
The
afor
emen
tione
d st
aff g
roup
sEx
tern
al s
take
hold
ers
are
mea
ning
fully
exte
rnal
sta
keho
lder
s in
QA
hold
ers
are
not i
nvol
ved
in Q
A.ex
clude
d fro
m Q
A:an
d st
uden
ts p
lay
an a
ctive
par
tin
volve
d in
eva
luat
ion.
The
diff
eren
t–
stud
ents
in th
e QA
sys
tem
. Ext
erna
l sta
ke-
staf
f gro
ups
are
man
ifest
ly co
mm
itted
– te
ache
rsho
lder
s ar
e in
volve
d in
QA.
to a
nd a
ctive
in p
ract
ical Q
A. P
artic
-–
supp
ort s
ervic
es s
taff
ipat
ion
is su
ppor
ted
by c
omm
on v
alue
s–
rese
arch
ers
and
a cu
lture
bas
ed o
n tru
st a
nd–
adm
inist
rativ
e st
aff
equa
lity.
– m
anag
emen
t–
exte
rnal
sta
keho
lder
s
5. I
nter
face
bet
wee
n th
e QA
sys
tem
No li
nkag
es b
etw
een
QA a
ndQA
con
sists
of p
roce
dure
s an
dTh
e QA
sys
tem
is a
n in
trins
ic pa
rt of
QA is
a n
atur
al p
art o
f the
ope
ratio
nsan
d m
anag
emen
t/ste
erin
gth
e le
ader
ship
and
man
agem
ent.
proc
esse
s th
at a
re s
epar
ate
from
the
oper
atio
ns a
nd th
e di
rect
ion
ofan
d th
e di
rect
ion
of o
pera
tions
. The
othe
r ope
ratio
ns. T
he li
nkag
esop
erat
ions
. Inf
orm
atio
n pr
oduc
ed in
man
agem
ent a
re c
omm
itted
to th
ebe
twee
n th
e QA
sys
tem
and
QA is
use
d in
dev
elop
men
t, w
ithsy
stem
and
take
resp
onsib
ility
for i
t.m
anag
emen
t/ste
erin
g ar
e in
-m
anife
st li
nkag
es b
etw
een
the
QAIn
form
atio
n is
syst
emat
ically
use
dad
equa
te.
syst
em a
nd th
e m
anag
emen
t,an
d th
ere
is cle
ar a
nd c
onst
ant
perfo
rman
ce m
onito
ring
and
evid
ence
of i
ts e
ffect
ive u
se in
the
deve
lopm
ent.
man
agem
ent,
perfo
rman
ce m
onito
ring
and
deve
lopm
ent.I
nfor
mat
ion
prod
uced
by th
e QA
sys
tem
give
s an
ove
rall
pict
ure
of th
e qu
ality
of e
duca
tion
and
othe
r act
ivitie
s.
��
AUDI
TING
TAR
GETS
CRIT
ERIA
ABSE
NTEM
ERGI
NGDE
VELO
PING
ADVA
NCED
6. R
elev
ance
of,
and
acce
ss to
,QA
doe
s no
t cat
er fo
r int
erna
lTh
e pr
oduc
tion
of in
form
atio
n is
not
The
activ
ities
and
key
resu
lts o
f the
The
oper
atio
n of
the
QA s
yste
m is
ove
rtqu
ality
ass
uran
ce in
form
atio
nst
akeh
olde
rs a
nd in
form
atio
n is
not
syst
emat
ic an
d do
es n
ot s
uffic
ient
lyQA
sys
tem
are
kno
wn
to in
tern
alan
d tra
nspa
rent
. Int
erna
l com
mun
icatio
nco
mm
unica
ted
with
in th
e HE
I.ca
ter f
or in
tern
al s
take
hold
ers.
stak
ehol
ders
. The
QA
syst
em p
rodu
ces
rela
ting
to Q
A is
activ
e an
d in
form
atio
nin
form
atio
n re
leva
nt to
them
.is
syst
emat
ically
com
mun
icate
d an
dta
rget
ed to
diff
eren
t par
ties
with
inth
e HE
I. The
rele
vanc
e of
info
rmat
ion
to in
tern
al s
take
hold
ers
is an
impo
rtant
cons
ider
atio
n in
the
plan
ning
and
cont
inuo
us d
evel
opm
ent o
f the
QA
syst
em.
7. R
elev
ance
of,
and
acce
ss to
,QA
doe
s no
t cat
er fo
r ext
erna
lEx
tern
al s
take
hold
ers
are
not
Exte
rnal
inte
rest
gro
ups
are
defin
edEx
tern
al c
omm
unica
tion
rela
ting
toqu
ality
ass
uran
ce in
form
atio
nst
akeh
olde
rs. I
nfor
mat
ion
is no
tsu
fficie
ntly
take
n in
to a
ccou
nt in
and
thei
r inf
orm
atio
n ne
eds
clear
lyQA
is a
ctive
and
info
rmat
ion
is sy
s-fo
r ext
erna
l sta
keho
lder
sco
mm
unica
ted
to e
xter
nal
the
plan
ning
and
dev
elop
men
t of t
heta
ken
into
acc
ount
. Inf
orm
atio
n ab
out
tem
atica
lly c
omm
unica
ted
and
targ
eted
stak
ehol
ders
.QA
sys
tem
. Inf
orm
atio
n is
only
the
activ
ities
and
key
resu
lts o
f the
to d
iffer
ent e
xter
nal s
take
hold
ers.
spor
adica
lly c
omm
unica
ted
toQA
sys
tem
is a
cces
sible
to m
ajor
The
rele
vanc
e of
info
rma t
ion
to e
xter
nal
exte
rnal
sta
keho
lder
s.pa
rtner
s an
d st
akeh
olde
rs.
stak
ehol
ders
is a
n im
porta
nt c
onsid
er-
atio
n in
the
plan
ning
and
dev
elop
men
tof
the
QA s
yste
m. T
he s
yste
m p
rodu
ces
info
rmat
ion
whi
ch a
lso h
as re
leva
nce
toex
tern
al in
tere
st g
roup
s.
�
AUDI
TING
TAR
GETS
CRIT
ERIA
ABSE
NTEM
ERGI
NGDE
VELO
PING
ADVA
NCED
8. E
fficie
ncy
of Q
A pr
oced
ures
and
QA p
roce
dure
s ar
e un
able
to id
entif
yQA
aim
s at
mai
ntai
ning
the
pres
ent
QA p
roce
dure
s pr
omot
e th
eSp
ecia
l atte
ntio
n is
paid
to m
etho
dsst
ruct
ures
and
thei
r effe
ct o
n th
esu
b-st
anda
rd q
ualit
y.le
vel o
f qua
lity.
QA p
roce
dure
s ar
ede
velo
pmen
t of a
ctivi
ties
and
and
stru
ctur
es c
ondu
cive
to n
ew id
eas
deve
lopm
ent o
f act
ivitie
sab
le to
iden
tify
sub-
stan
dard
qua
lity
gene
rate
cha
nge.
Sub
-sta
ndar
dan
d th
eir i
mpl
emen
tatio
n. T
he o
pera
tiona
lsa
tisfa
ctor
ily.
qual
ity is
effi
cient
ly id
entif
ied.
cultu
re e
ncou
rage
s in
nova
tion.
Sub
-st
anda
rd q
ualit
y is
effic
ient
ly id
entif
ied.
9.
Use
of in
form
atio
n pr
oduc
ed b
yIn
form
atio
n re
latin
g to
QA
is no
use
dUs
e of
QA
info
rmat
ion
is sp
orad
icQA
info
rmat
ion
is us
ed a
s a
tool
inTh
e us
e of
info
rmat
ion
is sy
stem
atic
the
QA s
yste
m a
s a
tool
for q
ualit
yas
a to
ol fo
r qua
lity
man
agem
ent
and/
or it
s co
llect
ion
is an
end
in it
self.
qual
ity m
anag
emen
t and
enh
ance
men
tan
d th
ere
is cle
ar a
nd c
onst
ant e
viden
cem
anag
emen
t and
enh
ance
men
t in
and
enha
ncem
ent i
n ed
ucat
ion
orre
latin
g to
edu
catio
n an
d ot
her
of it
s ef
fect
ive u
se in
the
deve
lopm
ent o
fed
ucat
ion
and
othe
r act
ivitie
sot
her a
ctivi
ties.
activ
ities
. Mos
t fee
dbac
k is
put t
o us
e.ed
ucat
ion
and
othe
r act
ivitie
s.
10. M
onito
ring,
eva
luat
ion
and
The
HEI h
as n
o ov
eral
l ide
a of
the
QATh
e HE
I has
onl
y a
fain
t ide
a of
the
The
HEI m
onito
rs th
e pe
rform
ance
The
HEI m
onito
rs th
e pe
rform
ance
o ft
heco
ntin
uous
dev
elop
men
t of t
he Q
Aac
tiviti
es; t
hey
are
not m
onito
red
orov
eral
l per
form
ance
of t
he Q
A sy
stem
.of
the
QA s
yste
m a
nd is
con
scio
us o
fQA
sys
tem
and
is la
rgel
y aw
are
of it
ssy
stem
deve
lope
d.Its
act
ivitie
s ar
e ba
rely
mon
itore
d,its
maj
or e
ffect
s an
d ou
tcom
es.
effe
cts
and
outc
omes
. The
dev
elop
men
tan
d th
e de
velo
pmen
t of t
he Q
ADe
velo
pmen
t of t
he Q
A sy
stem
isof
the
QA
syst
em is
pla
nned
and
syst
em is
unp
lann
ed.
plan
ned
and
docu
men
ted.
docu
men
ted
and
the
HEI c
an cl
early
dem
onst
rate
its
dire
ctio
n an
d co
ncre
teef
fect
s.
�
APPENDIX :Audit concepts
This section gives the Finnish Higher Education Evaluation Council’s (FINHEEC’s)interpretation of the concepts used in this manual� The definitions are based onestablished usage and interpretation by international evaluation organisations which FINHEEC has adapted for Finnish higher education and evaluation culture�This means that FINHEEC is not committed to the concepts and terminology of anyindividual quality assessment method (e�g� ISO EFQM or BSC)�
Accreditation
The word ’accreditation’ (Latin ad � credere) means to prove something creditableand publicly acknowledge its worth in relation to external criteria� Accreditationusually refers either to an official approval of HEIs or their programmes or to theawarding of different quality labels to HEIs or their programmes�
Auditing
Auditing is independent external evaluation to ascertain whether a QA systemconforms to its stated objectives is effective and fits its purpose� Auditing does notaddress the objectives or the results of operations as such but evaluates the processesthat the HEI uses to manage and improve the quality of its education and otheractivities�
Certification
Certification is the verification and validation of an achieved standard or status� Itoften includes a certificate of the standard or status achieved� The certificate can beawarded by a first party (the management of an organisation) a second party (thecustomer) or a third party (an accredited external certifier)�
Criteria
See evaluation criteria�
Enhancement�led evaluation
Enhancement�led evaluation refers to evaluation geared to support HEIs inimproving their education and other activities� FINHEEC sees enhancement�ledevaluation as a user�led process in which the evaluation method is tailored accordingto the objectives of the evaluation its theme and the needs of the participants�
�
Evaluation
Evaluation is systematic appraisal and highlighting of value or comparison againstobjectives and targets and “measurement” of performance (assessment as in qualityassessment) against set criteria� FINHEEC sees evaluation as a process geared tohighlight development needs and putting forward proposals�
Evaluation model (method)
In the evaluation of higher education institutions the evaluation model orevaluation method refers to an established European approach comprising fourcomponents: (�) a national or other external evaluation organisation; (�) a self�evaluation; () a peer evaluation including audit visits; and () a public evaluationreport�
Evaluation types
Evaluation types can be summarised according to use as: �� evaluation ��accreditation � auditing and � benchmarking� These different approaches are usedto evaluate three different targets (organisations degree programmes and subjects)and they can be used for different purposes from the development of operations toindicating accountability�
Quality assurance
Quality assurance refers to the procedures processes and systems that safeguard andimprove the quality of a HEI its education and other activities� The Finnish termlaatutyö (quality work) often means the same as quality assurance but is sometimesalso used to refer to the development of QA systems�
Quality assurance system
FINHEEC’s concept ‘quality assurance system’ is based on a concept that has becomeestablished in European quality evaluation� The concept includes both qualitymanagement and quality enhancement� It can be used in two ways: it may refer tothe QA system of an individual HEI or to the national system for assuring highereducation quality� The institutional QA system refers to the entity composed of thequality assurance organisation respective responsibilities procedures processes andresources� The national QA system refers to the procedures and processes of theHEIs FINHEEC and the Ministry of Education as a whole and to legislation enactedto assure higher education quality�
Quality
Quality can be defined in many ways for instance quality as exception asperfection as fitness for purpose as value for money and as transformative� � In theauditing of QA systems quality refers to the appropriateness (fitness for purpose) ofquality assurance methods processes and systems in relation to stated objectives oraims� Understood in this way quality is verified achievement of objectives�
Quality culture
Quality culture includes both measures geared to improve quality and individual andcollective commitment to maintaining and improving quality�
Self�evaluation
Self�evaluation means that a unit or an organisation appraises its own activities theirprerequisites and outcomes� Self�evaluation is a way of collecting information on theevaluation target and a tool for HEIs to improve their activities� Self�evaluation canbe undertaken on the organisation’s own initiative or at the behest of an externalbody�
Stakeholder /interest group
Stakeholders are groups or organisations with vested interests in the matter� Thestakeholders of a HEI are its staff students the students' parents and othertaxpayers employers the Government society trade unions and higher educationgraduates�
SWOT analysis
SWOT analysis is a method used in the evaluation and development of activities� ASWOT analysis lists the organisation’s Strengths Weaknesses Opportunities andThreats�
� Harvey L� and Green D� ��� ‘Defining quality’ Assessment and Evaluation in Higher Edu�cation �� no� � pp� �–�
➙➙
➙➙
➙➙
HEIs register for an audit■ Audits are undertaken on the basis of registration� FINHEEC devises an audit
timetable together with the HEIs that have signed up�
The audit agreement is drawn up■ When initiating the audit process FINHEEC and the HEI conclude an
agreement which indicates the auditing method the international ordomestic composition of the audit group (the language) the duration of theaudit visit the timetable for the auditing process the costs and the auditee'scommitment to implementing recommended development measures and toa possible re�audit�
The audit group is appointed■ FINHEEC appoints a five�member group to audit each HEI�
The HEI collects audit material■ The HEI under review collects data and other material about its own QA
system which provides sufficient information and evidence for the auditorsto assess the performance of the QA system� The audit material is primarilycollected from existing sources�
Meeting to prepare the audit visit■ Before the actual audit visit the chair of the audit group and the FINHEEC
project manager co�ordinating the audit visit the HEI� The purpose of thevisit is to inform the HEI staff and students about the audit and to discussits implementation�
Auditors visit the HEI■ During the audit visit which lasts � to days the audit group interviews
various stakeholders visits different units for evaluation purposes andstudies materials relating to the QA system�
The audit report is written■ The audit group writes an audit report based on materials accumulated
during the audit process� In it the auditors record the findings of the audit point out strengths and best practices and give their recommendations forfurther development�
APPENDIX :The phases of the audit processand their chronological order
�
Conclusions from the audit findings■ In their report the auditors give their conclusions to FINHEEC which then
decides whether the HEI’s QA system passes the audit or whether a re�auditis needed�
�� The HEI passes the audit■ The HEI is given a certificate
indicating that its QA systemhas undergone to a nationalaudit�
■ The HEI is entered in the auditregister on the FINHEECwebsite�
The audit report is published and discussed■ The audit report is published at an open seminar where representatives of
the HEI can discuss the audit results and conclusions with members of theaudit group�
The next audit is carried out
➙
➘
➘
➙➙
➙➙
(2 yrs)
(6 yrs)
�� A re�audit is needed■ Measured against the audit
criteria the HEI’s QA system orquality assurance relating to itsbasic tasks were found to havemajor shortcomings� The HEIneeds to take measures toimprove its QA system andundergo a re�audit�
■ The audit report point out thenecessary improvement measures�
Meeting between the HEI FINHEEC and the auditinggroup■ The parties discuss the
substance of the recommendeddevelopment measures and theirorder of priority�
A re�audit is carried out with focus on therecommended improvement
�
APPENDIX �:An example of an audit visit programme
First day
The first day includes an opening session and interviews with the representatives ofthe HEI’s management staff groups students and stakeholders� The visit starts witha brief presentation of the HEI’s QA system and relevant documents�
TimeTimeTimeTimeTime ProgrammeProgrammeProgrammeProgrammeProgramme
9.00–9.30 The audit group’s own meeting9.30–9.50 Presentation of the QA system10.00–11.00 Interview with the HEI leadership and the quality/development manager11.10–12.10 Interview with the deans/heads of fields of study12.10–13.00 Lunch13.00–14.00 Interview with teaching staff14.10–15.10 Interview with students15.20–16.20 Interview with representatives of support and other services16.30–17.30 Interview with representatives of external stakeholders17.30–18.00 Meeting of the audit group
Second day
On the second day the audit group visits faculties departments or degreeprogrammes and studies quality assurance materials� The auditors themselves decideon the programme of the second and a possible third day� They choose the unitsthey will visit on the basis of the audit materials� Some of the visits may be decidedon site� The audit group also determines the people to be interviewed and theinterview methods�
The auditors may break up into smaller groups to visit the units� It is also possibleto arrange discussion sessions for different staff groups around themes arising fromthe audit materials provided by the HEI�
TimeTimeTimeTimeTime ProgrammeProgrammeProgrammeProgrammeProgramme
8.30–9.30 Perusal of the quality assurance materials9.30–12.00 Visits to specific units/Discussions on chosen themes12.15–13.00 Lunch13.00–15.30 Visits to specific units/Discussions on chosen themes16.00–16.30 Meeting of the audit group
�
Third day
If the HEI under review is large or if the audit is otherwise particularly laborious theaudit visit may be extended to ��� or three days� The third day continues with visitsto specific units� The audit group ends its visit by including session where the auditgroup gives a brief summary of the most important observations they have madeduring the visit�
TimeTimeTimeTimeTime ProgrammeProgrammeProgrammeProgrammeProgramme
9.00–10.45 Visits to specific units11.00–12.00 Interview with the management12.15–13.00 Lunch and concluding session
PUBLICATIONS OF THE FINNISH HIGHER EDUCATION EVALUATION COUNCILP. B. 133, 00171 HELSINKI, Finland • Tel. +358-9-1607 6913 • Fax +358-9-1607 6911 • www.finheec.fi
1:2000 Lehtinen, E., Kess, P., Ståhle, P. & Urponen, K.: Tampereen yliopiston opetuksen arviointi. Helsinki: Edita.2:2000 Cohen, B., Jung, K. & Valjakka, T.: From Academy of Fine Arts to University. Same name, wider ambitions.
Helsinki: Edita.3:2000 Goddard, J., Moses, I., Teichler, U., Virtanen, I. & West, P.: External Engagement and Institutional Adjustment:
An Evaluation of the University of Turku. Helsinki: Edita.4:2000 Almefelt, P., Kekäle, T., Malm, K., Miikkulainen, L. & Pehu-Voima, S.: Audit of Quality Work. Swedish
Polytechnic, Finland. Helsinki: Edita.5:2000 Harlio, R., Harvey, L., Mansikkamäki. J., Miikkulainen, L. & Pehu-Voima, S.: Audit of Quality Work. Central
Ostrobothnia Polytechnic. Helsinki: Edita.6:2000 Moitus, S. (toim.): Yliopistokoulutuksen laatuyksiköt 2001–2003. Helsinki: Edita.7:2000 Liuhanen, A.-M. (toim.): Neljä aikuiskoulutuksen laatuyliopistoa 2001–2003. Helsinki: Edita.8:2000 Hara, V. , Hyvönen, R. , Myers, D. & Kangasniemi, J. (Eds.): Evaluation of Education for the Information
Industry. Helsinki: Edita.9:2000 Jussila, J. & Saari, S. (Eds.): Teacher Education as a Future-moulding Factor. International Evaluation of Teacher
Education in Finnish Universities. Helsinki: FINHEEC.10:2000 Lämsä, A. & Saari, S. (toim.): Portfoliosta koulutuksen kehittämiseen. Ammatillisen opettajankoulutuksen
arviointi. Helsinki: Edita.11:2000 Korkeakoulujen arviointineuvoston toimintasuunnitelma 2000–2003. Helsinki: Edita.12:2000 Finnish Higher Education Evaluation Council Action Plan for 2000–2003. Helsinki: Edita.13:2000 Huttula, T. (toim.): Ammattikorkeakoulujen koulutuksen laatuyksiköt 2000. Helsinki: Edita.14:2000 Gordon, C., Knodt, G., Lundin, R., Oger, O. & Shenton, G.: Hanken in European Comparison. EQUIS
Evaluation Report. Helsinki: Edita.15:2000 Almefelt, P., Kekäle, T., Malm, K., Miikkulainen, L. & Kangasniemi, J.: Audit of Quality Work. Satakunta
Polytechnic. Helsinki: Edita.16:2000 Kells, H.R., Lindqvist, O. V. & Premfors, R.: Follow-up Evaluation of the University of Vaasa. Challenges of a
small regional university. Helsinki: Edita.17:2000 Mansikkamäki, J., Kekäle, T., Miikkulainen, L. , Stone, J., Tolppi, V.-M. & Kangasniemi, J.: Audit of Quality Work.
Tampere Polytechnic. Helsinki: Edita.18:2000 Baran, H., Gladrow, W. , Klaudy, K. , Locher, J. P. , Toivakka, P. & Moitus, S.: Evaluation of Education and
Research in Slavonic and Baltic Studies. Helsinki: Edita.19:2000 Harlio, R. , Kekäle, T. , Miikkulainen, L. & Kangasniemi, J.: Laatutyön auditointi. Kymenlaakson ammatti-
korkeakoulu. Helsinki: Edita.20:2000 Mansikkamäki, J., Kekäle, T., Kähkönen, J., Miikkulainen, L., Mäki, M. & Kangasniemi, J.: Laatutyön auditointi.
Pohjois-Savon ammattikorkeakoulu. Helsinki: Edita.21:2000 Almefelt, P., Kantola, J., Kekäle, T., Papp, I., Manninen, J. & Karppanen, T.: Audit of Quality Work. South Carelia
Polytechnic. Helsinki: Edita.1:2001 Valtonen, H.: Oppimisen arviointi Sibelius-Akatemiassa. Helsinki: Edita.2:2001 Laine, 1., Kilpinen, A., Lajunen, L., Pennanen, J., Stenius, M., Uronen, P. & Kekäle, T.: Maan-
puolustuskorkeakoulun arviointi. Helsinki: Edita.3:2001 Vähäpassi, A. (toim.): Erikoistumisopintojen akkreditointi. Helsinki: Edita.4:2001 Baran, H., Gladrow, W. , Klaudy, K. , Locher, J. P. , Toivakka, P. & Moitus, S.: |kspertiza obrazowaniq i nau^no-
issledowatelxskoj raboty w oblasti slawistiki i baltistiki (Ekspertiza obrazovanija i nauc‘´no-issledovatelskoj raboty v oblasti slavistiki i baltistiki). Helsinki: Edita.
5:2001 Kinnunen, J.: Korkeakoulujen alueellisen vaikuttavuuden arviointi. Kriteerejä vuorovaikutteisuuden arvottamisel-le. Helsinki: Edita.
6:2001 Löfström, E.: Benchmarking korkeakoulujen kieltenopetuksen kehittämisessä. Helsinki: Edita.7:2001 Kaartinen-Koutaniemi, M.: Korkeakouluopiskelijoiden harjoittelun kehittäminen. Helsingin yliopiston, Diakonia-
ammattikorkeakoulun ja Lahden ammattikorkeakoulun benchmarking-projekti. Helsinki: Edita.8:2001 Huttula, T. (toim.): Ammattikorkeakoulujen aluekehitysvaikutuksen huippuyksiköt 2001. Helsinki: Edita.9:2001 Welander, C. (red.): Den synliga yrkeshögskolan. Ålands yrkeshögskola. Helsingfors: Edita.10:2001 Valtonen, H.: Learning Assessment at the Sibelius Academy. Helsinki: Edita.11:2001 Ponkala, O. (toim.): Terveysalan korkeakoulutuksen arvioinnin seuranta. Helsinki: Edita.12:2001 Miettinen, A. & Pajarre, E.: Tuotantotalouden koulutuksen arvioinnin seuranta. Helsinki: Edita.13:2001 Moitus, S., Huttu, K., Isohanni, I., Lerkkanen, J., Mielityinen, I., Talvi, U., Uusi-Rauva, E. & Vuorinen, R.:
Opintojen ohjauksen arviointi korkeakouluissa. Helsinki: Edita.14:2001 Fonselius, J., Hakala, M.K. & Holm, K. : Evaluation of Mechanical Engineering Education at Universities and
Polytechnics. Helsinki: Edita.15:2001 Kekäle, T. (ed.): A Human Vision with Higher Education Perspective.Institutional Evaluation of the Humanistic
Polytechnic. Helsinki: Edita.1:2002 Kantola, I. (toim.): Ammattikorkeakoulun jatkotutkinnon kokeilulupahakemusten arviointi. Helsinki: Edita.2:2002 Kallio, E.: Yksilöllisiä heijastuksia. Toimiiko yliopisto-opetuksen paikallinen itsearviointi? Helsinki: Edita.3:2002 Raivola, R., Himberg, T., Lappalainen, A., Mustonen, K. & Varmola, T.: Monta tietä maisteriksi. Yliopistojen
maisteriohjelmien arviointi. Helsinki: Edita.4:2002 Nurmela-Antikainen, M., Ropo, E., Sava, I. & Skinnari, S.: Kokonaisvaltainen opettajuus. Steinerpedagogisen
opettajankoulutuksen arviointi.Helsinki: Edita.5:2002 Toikka, M. & Hakkarainen, S.: Opintojen ohjauksen benchmarking tekniikan alan koulutusohjelmissa.
Kymenlaakson, Mikkelin ja Pohjois-Savon ammattikorkeakoulut. Helsinki: Edita.6:2002 Kess, P., Hulkko, K., Jussila, M., Kallio, U., Larsen, S. , Pohjolainen,T. & Seppälä, K.: Suomen avoin yliopisto.
Avoimen yliopisto-opetuksen arviointiraportti. Helsinki: Edita.7:2002 Rantanen, T., Ellä, H., Engblom, L.-Å., Heinonen, J., Laaksovirta, T., Pohjanpalo, L., Rajamäki, T.& Woodman,
J.: Evaluation of Media and Communication Studies in Higher Education in Finland. Helsinki: Edita.8:2002 Katajamäki, H., Artima, E., Hannelin, M., Kinnunen, J., Lyytinen, H. K., Oikari, A. & Tenhunen, M.-L.:
Mahdollinen korkeakouluyhteisö. Lahden korkeakouluyksiköiden alueellisen vaikuttavuuden arviointi. Helsinki:Edita.
11:2002 Katajamäki, H. (toim.): Ammattikorkeakoulut alueidensa kehittäjinä.Näkökulmia ammattikorkeakoulujenaluekehitystehtävän toteutukseen. Helsinki: Edita.
12:2002 Huttula, T. (toim.): Ammattikorkeakoulujen koulutuksen laatuyksiköt 2002–2003. Helsinki: Edita.13:2002 Hämäläinen, K. & Kaartinen-Koutaniemi, M. (toim.): Benchmarking korkeakoulujen kehittämisvälineenä.
Helsinki: Edita.14:2002 Ylipulli-Kairala, K. & Lohiniva, V. (eds.): Development of Supervised Practice in Nurse Education. Oulu and
Rovaniemi Polytechnics. Helsinki: Edita.15:2002 Löfström, E., Kantelinen, R., Johnson, E., Huhta, M., Luoma, M., Nikko, T., Korhonen, A., Penttilä, J.,
Jakobsson, M. & Miikkulainen, L.: Ammattikorkeakoulun kieltenopetus tienhaarassa. Kieltenopetuksen arviointiHelsingin ja Keski-Pohjanmaan ammattikorkeakouluissa. Helsinki: Edita.
16:2002 Davies, L., Hietala, H., Kolehmainen, S., Parjanen, M. & Welander, C.: Audit of Quality Work. VaasaPolytechnic. Helsinki: Edita.
17:2002 Sajavaara, K., Hakkarainen, K. , Henttonen, A., Niinistö, K., Pakkanen, T. , Piilonen, A.-R. & Moitus, S.:Yliopistojen opiskelijavalintojen arviointi. Helsinki: Edita.
18:2002 Tuomi, O. & Pakkanen, P.: Towards Excellence in Teaching. Evaluation of the Quality of Education and theDegree Programmes in the University of Helsinki. Helsinki: Edita.
1:2003 Sarja, A., Atkin, B. & Holm, K.: Evaluation of Civil Engineering Education at Universitiesand Polytechnics. Helsinki: Edita.
2:2003 Ursin, J. (toim.): Viisi aikuiskoulutuksen laatuyliopistoa 2004–2006. Helsinki: Edita.3:2003 Hietala, H., Hintsanen, V., Kekäle, T., Lehto, E., Manninen, H. & Meklin, P.: Arktiset haasteet ja mahdollisuudet.
Rovaniemen ammattikorkeakoulun kokonaisarviointi. Helsinki: Edita.4:2003 Varis, T. & Saari, S. (Eds.): Knowledge Society in Progress – Evaluation of the Finnish Electronic Library –
FinELib. Helsinki: Edita.5:2003 Parpala, A. & Seppälä, H. (toim.): Yliopistokoulutuksen laatuyksiköt 2004–2006. Helsinki: Edita.6:2003 Kettunen, P., Carlsson, C., Hukka, M., Hyppänen, T., Lyytinen, K., Mehtälä, M., Rissanen, R., Suviranta, L. &
Mustonen, K.: Suomalaista kilpailukykyä liiketoimintaosaamisella. Kauppatieteiden ja liiketalouden korkea-koulutuksen arviointi. Helsinki: Edita.
7:2003 Kauppi, A. & Huttula, T. (toim.): Laatua ammattikorkeakouluihin. Helsinki: Edita.8:2003 Parjanen, M. : Amerikkalaisen opiskelija-arvioinnin soveltaminen suomalaiseen yliopistoon. Helsinki: Edita.9:2003 Sarala, U. & Seppälä, H.: (toim.): Hämeen ammattikorkeakoulun kokonaisarviointi. Helsinki: Edita.10:2003 Kelly‚ J., Bazsa, G. & Kladis, D.: Follow-up review of the Helsinki University of Technology. Helsinki: Edita.11:2003 Goddard, J., Asheim, B., Cronberg, T. & Virtanen, I.: Learning Regional Engagement. A Re-evaluation of the
Third Role of Eastern Finland universities.12:2003 Impiö, 1., Laiho, U.-M., Mäki, M., Salminen, H., Ruoho, K.,Toikka, M. & Vartiainen, P.: Ammattikorkeakoulut
aluekehittäjinä. Ammattikorkeakoulujen aluekehitysvaikutuksen huippuyksiköt 2003–2004. Helsinki: Edita.13:2003 Cavallé, C., de Leersnyder, J.-M., Verhaegen, P. & Nataf, J.-G. : Follow-up review of the Helsinki School of
Economics. An EQUIS re-accreditation. Helsinki: Edita.14:2003 Kantola, I. (toim.): Harjoittelun ja työelämäprojektien benchmarking. Helsinki: Edita.15:2003 Ala-Vähälä, T.: Hollannin peili. Ammattikorkeakoulujen master-tutkinnot ja laadunvarmistus. Helsinki: Edita.16:2003 Goddard, J., Teichler, U., Virtanen, I., West, P. & Puukka, J.: Progressing external engagement. A re-evaluation
of the third role of the University of Turku. Helsinki: Edita.17:2003 Baran, H., Toivakka, P. & Järvinen, J.: Slavistiikan ja baltologian koulutuksen ja tutkimuksen arvioinnin seuranta.
Helsinki: Edita.1:2004 Kekäle, T., Heikkilä, J., Jaatinen, P., Myllys, H., Piilonen, A.-R., Savola, J., Tynjälä, P. & Holm, K.: Ammatti-
korkeakoulujen jatkotutkintokokeilu. Käynnistysvaiheen arviointi. Tampere: Tammer-Paino Oy.2:2004 Ekholm, L., Stenius, M., Huldin, H., Julkunen, I., Parkkonen, J., Löfström, E., Metsä, K.: NOVA ARCADA –
Sammanhållning, decentralisering, gränsöverskridande. Helhetsutvärdering av Arcada – Nylands svenskayrkeshögskola 2003. Tammerfors: Tammer-Paino Oy.
3:2004 Hautala, J.: Tietoteollisuusalan koulutuksen arvioinnin seuranta.Tampere: Tammer-Paino Oy.4:2004 Rauhala, P., Karjalainen, A., Lämsä, A.-M., Valkonen, A., Vänskä, A. & Seppälä, H.: Strategiasta
koulutuksen laatuun. Turun ammattikorkeakoulun kokonaisarviointi. Tampere: Tammer-Paino Oy.5:2004 Murto, L., Rautniemi, L., Fredriksson, K., Ikonen, S., Mäntysaari, M., Niemi, L., Paldanius, K., Parkkinen, T.,
Tulva, T. , Ylönen, F. & Saari, S.: Eettisyyttä, elastisuutta ja elämää. Yliopistojen sosiaalityön ja ammatti-korkeakoulujen sosiaalialan arviointi yhteistyössä työelämän kanssa. Tampere: Tammer-Paino Oy.
6:2004 Ståhle, P., Hämäläinen, K., Laiho, K., Lietoila, A., Roiha, J., Weijo, U. & Seppälä, H.:Tehokas järjestelmä –elävä dialogi. Helian laatutyön auditointi. Tampere: Tammer-Paino Oy.
7:2004 Korkeakoulujen arviointineuvoston toimintakertomus 2000–2003.Tampere: Tammer-Paino Oy.8:2004 Luopajärvi, T., Hauta-aho, H., Karttunen, P., Markkula, M., Mutka, U. & Seppälä, H.: Perämerenkaaren
ammattikorkeakoulu? Kemi-Tornion ammattikorkeakoulun kokonaisarviointi. Tampere: Tammer-Paino Oy.9:2004 Moitus, S. & Seppälä, H.: Mitä hyötyä arvioinneista? Selvitys Korkeakoulujen arviointineuvoston 1997–2003
toteuttamien koulutusala-arviointien käytöstä. Tampere: Tammer-Paino Oy.10:2004 Moitus, S. & Saari, S.: Menetelmistä kehittämiseen. Korkeakoulujen arviointineuvoston arviointimenetelmät
vuosina 1996–2003. Tampere: Tammer-Paino Oy.11:2004 Pratt, J., Kekäle, T., Maassen, P., Papp, I., Perellon, J. & Uitti, M.: Equal, but Different – An Evaluation of the
Postgraduate Studies and Degrees in Polytechnics – Final Report.Tampere: Tammer-Paino Oy.1:2005 Niinikoski, S. (toim.): Benchmarking tutkintorakennetyön työkaluna.Tampere: Tammer-Paino Oy.2:2005 Ala-Vähälä, T.: Korkeakoulutuksen ulkoisen laadunvarmistuksen järjestelmät Ranskassa.Tampere: Tammer-
Paino Oy.3:2005 Salminen, H. & Kajaste, M. (toim.): Laatua, innovatiivisuutta ja proaktiivisuutta. Ammattikorkeakoulujen
koulutuksen laatuyksiköt 2005–2006. Tampere: Tammer-Paino Oy.4:2005 Korkeakoulujen laadunvarmistusjärjestelmien auditointi. Auditointikäsikirja vuosille 2005–2007. Tampere:
Tammer-Paino Oy.5:2005 Auditering av högskolornas kvalitetssäkringssystem. Auditeringshandbok för åren 2005–2007. Tammerfors:
Tammer-Paino Oy.1:2006 Dill, D.D., Mitra, S. K., Siggaard Jensen, H., Lehtinen, E., Mäkelä, T., Parpala, A., Pohjola, H., Ritter, M. A. & Saari, S.:
PhD Training and the Knowledge-Based Society. An Evaluation of Doctoral Education in Finland.Tampere: Tammer-Paino Oy.
2:2006 Antikainen, E.-L., Honkonen, R., Matikka, O., Nieminen, P., Yanar, A. & Moitus, S.: Mikkelin ammatti-korkeakoulun laadunvarmistusjärjestelmän auditointi. Tampere: Tammer-Paino Oy.
3:2006 Kekäle, T., Ilolakso, A., Katajavuori, N., Toikka, M. & Isoaho, K.: Kuopion yliopiston laadunvarmistus-järjestelmän auditointi. Tampere: Tammer-Paino Oy.
