Executive Summary i - QEF · Executive Summary Review of the Operations of the Quality Education Fund Management Services Agency i Background 1. Commissioned by the Education and

Executive Summary

Review of the Operations of the Quality Education Fund Management Services Agency

i

Background

1. Commissioned by the Education and Manpower Bureau, the

Management Services Agency (MSA) conducted a scoping

study in November 2001 on the operations of the Quality

Education Fund (QEF). The scoping study report was

endorsed by the QEF Steering Committee (QEFSC) on

30.1.2002. MSA proceeded with the main study on re-

engineering the QEF operations in February 2002.

Findings

2. Among the issues identified, major ones are highlighted

below –

Organisation and planning issues

3. The assessment and monitoring processes are interrelated

but they are handled by two separate subcommittees, namely

the Assessment Subcommittee (ASC) and the Promotion

and Monitoring Subcommittee (PMSC). Members of the

two subcommittees may not fully understand both the

assessment and monitoring processes of individual projects.

4. Since the QEF does not have an annual budget, it is difficult

to maintain its ability to fund worthwhile projects in the long

term, given the fluctuation in the number of applications and

capital nature of the Fund.

Process issues

5. Both ASC and QEFSC are required to assess all applications

received under a call for applications. However, under the

existing structure, it is difficult deploy resources effectively

to handle fluctuated assessment workload.

6. Because of the large number of projects approved in recent

application calls, the QEF has experienced difficulties to

cope with the monitoring workload.

7. Many existing processes are performed manually. There is

room for improvement with the assistance of information

technology.

Customer services issues

8. The introduction of major changes to application

arrangements when calling for applications may render

some projects being designed ineligible for QEF funding

and lead to abortive work of applicants. In addition, some

key QEF operation information are not publicised.

Executive Summary


ii

Recommendations

9. To address the above issues, the following major

recommendations are made –

On organisation and planning

Ø Restructure ASC and PMSC to form four subcommittees

overseeing projects throughout the project periods;

Ø Formulate an annual budget for the QEF;

On process re-engineering

Ø Delegate the authority for endorsement of small-value

applications to ASC (or the proposed subcommittees);

Ø Explore the feasibility of peer review;

Ø Categorise projects for performance monitoring;

Ø Rationalise the financial monitoring mechanism;

Ø Transfer the ownership of project assets to grantees;

Ø Automate manual operations;

On customer services

Ø Announce new funding policies and application

arrangements at the earliest possible time;

Ø Publicise key operational information.

10. A summary of the recommendations is at page 錯誤! 尚未定

義書籤。.

11. The recommendations will bring about a total saving of 8

posts and improved customer service upon full

implementation.

Way forward

12. The Project Steering Group (PSG)’s endorsement on the

recommendations should be sought. Upon the PSG’s

endorsement, the QEF is recommended to form a working

group to work out the implementation plan and details.

Table of Contents

Page

Background.......................................................................................1

Introduction...................................................................................1

Project Steering Group..................................................................1

Study objective and scope .............................................................2

Terms of reference.........................................................................2

Study methodology .......................................................................3

Overview of the QEF Operations ...................................................4

Organisation structure ...................................................................4

Business objective .........................................................................5

Business processes ........................................................................5

Workload statistics ........................................................................6

Work profile of QEF Secretariat staff ...........................................7

Management initiatives .................................................................8

Survey on External Practices........................................................10

Fund granting organisations surveyed.........................................10

Comparison of the practices of the QEF and fund granting

organisations surveyed................................................................11

Findings and Recommendations ...................................................13

Organisation and planning issues ................................................13

Process issues ..............................................................................17

Customer services issues .............................................................30

Summary of Recommendations and Benefits .............................33

Next Steps .......................................................................................35

Implementation............................................................................35

Acknowlegement ............................................................................35

Appendices

Appendix 1 List of IntervieweesAppendix 2 Core Business Processes of the QEFAppendix 3 (A) – Procedural Flowchart of the Existing System

(B) – Procedural Flowchart of the Proposed SystemAppendix 4 Caseload Statistics of the QEFAppendix 5 Work Profile of QEF SecretariatAppendix 6 Summary of External PracticesAppendix 7 Available Grants for Allocation under Different Budget ModelsAppendix 8 Merits and Constraints of Peer Review SystemAppendix 9 Financial Implication of Peer Review SystemAppendix 10 Evaluation Instruction of the Louisiana Quality Education Support FundAppendix 11 Estimated Savings from Computerisation

Background


P.1

Introduction

1. Commissioned by the Education and Manpower Bureau

(EMB), the Management Services Agency (MSA) conducted

a scoping study in November 2001 on the operations of the

Quality Education Fund (QEF). The scoping study report

recommended –

(a) a detailed study be conduced by MSA to re-engineer the

operations of the QEF; and

(b) an external advisor be engaged to develop an effective

evaluation methodology for project monitoring and to

formulate a promotion and dissemination strategy.

2. The QEF Steering Committee (QEFSC) endorsed the

recommendations on 30 January 2002.

3. This report presents the findings and recommendations of

the detailed study conducted by MSA.

4. A consultant, Melbourne University Private, has been

employed to conduct (b)1.

1 The consultancy study does not cover the promotion strategy which will be

developed separately at a later stage.

Project Steering Group

5. A Project Steering Group (PSG) is formed to –

Ø oversee and steer the study;

Ø monitor progress and provide guidance to the Project Team;

and

Ø consider and accept the study recommendations.

6. The PSG comprises –

Ø Chairman: Chairman, QEFSC

Ø Members:

- QEFSC: Chairmen of Assessment Subcommittee

(ASC) and Project Monitoring

Subcommittee (PMSC)

- EMB: DS(EM)2 and PAS(EM)8

- ED: DD of E

- MSA: ADMS(A) or PMSO(A)

Ø Secretary: AS(8)3

Background


P.2

Study objective and scope

7. The study objective is to re-engineer the work processes of

the QEF with a view to improving the efficiency, cost-

effectiveness and user-friendliness in service delivery.

8. The scope of the study is confined to the work processes of

application call, assessment and project monitoring.

Terms of reference

9. The terms of reference of this study are –

(a) to review the application call arrangement and to streamline

the application procedure (e.g. simplifying processes and

forms, making use of IT solutions, etc.);

(b) to review the assessment arrangement, budget negotiation

and documentation system for application assessment, with

a view to speeding up the assessment process and reducing

the workload of parties involved;

(c) to review the arrangement for monitoring project

implementation and expenditure status, including the

control mechanism, fixed asset management, project

closure/ termination arrangement, terms of project

agreement and grants payment arrangement;

(d) to review the QEF budgetary arrangement so that the QEF

could better coordinate its investment and funding

programmes;

(e) to conduct surveys of practices in other local and overseas

fund granting organisations with similar functions with a

view to identifying the best practices for delivery of QEF’s

services;

(f) to explore, evaluate and recommend appropriate

improvement measures; and

(g) in the light of the improvement measures identified, to

recommend changes to the organisation structure and duties

of the QEFSC, ASC, PMSC and the QEF Secretariat.

Background


P.3

Study methodology

10. The Study Team has adopted the following methodology in

the study –

Ø studying committee papers, case files, past study reports and

other relevant documents;

Ø analysing work profile of QEF Secretariat;

Ø analysing relevant statistics and management information;

Ø interviewing QEF committee members, external/expert

reviewers, grantees, and staff of QEF Secretariat, the

Treasury and Education Department (ED); and

Ø collecting information on local and overseas practices of

fund granting organisations by interviewing local staff and

Internet research.

11. A list of interviewees is at Appendix 1.

Overview of the QEF Operations


P.4

Organisation structure

12. QEF Secretariat has a strength of 49 staff. The organisation structure of the QEF is shown at Diagram 1.

Diagram 1 - Organisation Structure of the QEF as at 1.4.2002

QEF Steering Committee(12 members)


QEF Investment Committee(5 members)


Assessment Sub-committee(23 members)


Promotion and Monitoring Sub-committee(38 members)


QEF Committee

QEF Committee

QEF Secretariat

QEF Secretariat PAS(EM)8PAS(EM)8

PS IPS I

AS(8)2AS(8)2

1 PO1 APO

1 PO1 APO

SBM/ER

SPOSPO AS(8)3AS(8)3

PS IIPS IIPS IIPS II

1 PO1 APO

1 PO1 APO

EL

2 POs2 APOs

2 POs2 APOs

AR

2 POs2 APOs

2 POs2 APOs

IT Dissemination

2 POs2 PjAs

2 POs2 PjAs

Administration Section

1 ComputerCo-ordinator

2 PjAs


2 PjAs

General Registry

1 ACO1 CA1 Clerk1 WM


Finance Unit

1 ACO1 CA1 Clerk

1 ACO1 CA1 Clerk

AccountsUnit

1 Accountant1 Accountant

6 Clerks6 Clerks 2 Clerks2 Clerks

1 EO1 EO

5 Clerks5 Clerks

1 SEO1 SEO

Project Support









QEF Committee

QEF Committee

QEF Secretariat

QEF Secretariat PAS(EM)8PAS(EM)8

PS IPS I

AS(8)2AS(8)2

1 PO1 APO

1 PO1 APO

SBM/ER

SPOSPO AS(8)3AS(8)3

PS IIPS IIPS IIPS II

1 PO1 APO

1 PO1 APO

EL

2 POs2 APOs

2 POs2 APOs

AR

2 POs2 APOs

2 POs2 APOs

IT Dissemination

2 POs2 PjAs

2 POs2 PjAs

Administration Section


2 PjAs


2 PjAs

General Registry



Finance Unit

1 ACO1 CA1 Clerk

1 ACO1 CA1 Clerk

AccountsUnit

1 Accountant1 Accountant

6 Clerks6 Clerks 2 Clerks2 Clerks

1 EO1 EO

5 Clerks5 Clerks

1 SEO1 SEO

Project Support



P.5

Business objective

13. The business objective of the QEF is to fund worthwhile initiatives on a pilot basis and one-off projects that aim to raise the quality of

school education, and to promote quality school education at all levels (source: Finance Committee paper, FCR(97-98)81).

Business processes

14. Core business processes of the QEF are outlined at Diagram 2 below.

Diagram 2 - Core Business Processes

15. Details of the core business processes and the existing process maps are at Appendices 2 and 3A respectively.

Application call�

Pre-assessment�

Assessment�

Approval

Application call andassessment

Grant payment

�Performance monitoring

Financial monitoring

Project variation processing

Project monitoring

Dissemination

Promotion

Dissemination andpromotion

Process outside the scope of this study

Administration and support



P.6

Workload statistics

16. The QEF deploys PMSC members, Secretariat staff and

external/expert reviewers to monitor ongoing projects. The

overall distribution of projects monitored by these reviewers

since the inception of the QEF in 1998 is shown below.

17. Key workload statistics of the core business processes under

this review are tabulated below.

Activity No. completed in 2001Applications assessed 3 507Progress/final reports evaluated 2 291Site visit/interview conducted 1 842Financial reports checked 4 675

18. The backlog of various cases is summarised below.

Item Outstanding caseas at 5 April 2002

Progress/final reports to be evaluated 4 701Financial reports to be checked 1 0772

19. Detailed workload statistics are shown at Appendix 4.

2 Including 609 financial reports pending further clarification from grantees

Experts2% External

Reviewers40%

PMSCmembers

28%

SecretariatStaff30%



P.7

Work profile of QEF Secretariat staff

20. With the assistance of the QEF Secretariat staff, the Study

Team collected statistics on time spent by Secretariat staff in

various activities during the period November 2000 to

October 2001 by questionnaire survey.

21. The workload distributions of PO/APO, project clerk and

account clerk during the survey period are shown at

Diagrams 3 to 5. Detailed workload distributions are at

Appendix 5.

Diagram 3 - Workload distribution of PO/APO

Diagram 4 - Workload distribution of Project Clerk

Diagram 5 - Workload distribution of Account Clerk

Liaise with grantees22%

Liaise with PO / APOs11%

Conduct checking andcounter-checking of

financial reports50%

Re-group budget itemsaccording to ScheduleII of the Agreement

9%

Answer enquiries5%

Other duties3%

Applicationassessment

48%

Projectmonitoring

40%Other duties

6%

Promotion &dissemination

6%

Post/faxdocuments (e.g.progress reports

to reviewers)28%

Data input(e.g. members'

assessment,evaluationreports)

25%

General clericalduties

(e.g. filing andphotocopying)

47%



P.8

Management initiatives

Initiatives implemented

22. The QEF has implemented the following initiatives in recent

years –

On application and assessment

Ø split assessment panels into smaller groups to speed up the

assessment process (the lead time was reduced by as much

as 50%);

Ø introduced standard projects (e.g. Reading Schemes, Parent

Resource Centres, Multimedia Learning Centres) to

simplify the application procedures [Note: A more focussed

approach has been adopted in the fifth call. Only secondary,

primary, pre-primary and special schools without on-going

QEF project in hand may apply and each eligible applicant

can submit only one application];

On project monitoring

Ø reduced the submission frequency of progress and financial

reports (from four times a year to two times a year for

projects costing not more than $100,000 and three times a

year for the remaining projects) to reduce the workload of

grantees and QEF;

Ø streamlined the audit procedure (by adopting a more

systematic sample check approach) to speed up the vetting of

financial reports;

On promotion and dissemination

Ø launched various promotion activities (e.g. expositions,

roving exhibitions, seminars, Saturday talks) to promote the

QEF and disseminate best practices of funded projects;

Ø revamped the QEF Cyber Resource Centre to improve its

user-friendliness and to facilitate better sharing of

information; and

Ø formed a Working Group on QEF Dissemination Work in

October 2001 to develop a long-term strategy and work out

the implementation details.

New initiates being developed

23. We understand that the QEF Secretariat is developing the

following new initiatives to further improve the operations

of the QEF –



P.9

Operation manual

Ø The QEF Secretariat is developing an operation manual

which documents the QEF policies and procedures, e.g.

assessment procedures, assessment criteria, pricing

standards, guidelines on conflict of interest, project progress

monitoring and financial monitoring.

Ø The operation manual will serve as a clear and actionable

documentation for the QEF. It will increase the consistency

in decision-making and facilitate staff training.

Transfer the QEF Secretariat to the Education Department (ED)

Ø Given its operational nature, EMB is now considering the

transfer of the QEF Secretariat to ED. It is anticipated that

the reorganisation will facilitate greater collaboration

between the Secretariat and ED in the dissemination of best

practices of funded projects. We understand that the transfer

will be considered in the context of the overall

reorganisation of EMB and ED.

Survey on External Practices


P.10

Fund granting organisations surveyed

24. In the course of the study, the Study Team researched on practices of the following six fund granting organisations –

Organisation Grants ProgrammeLocalResearch Grants Council Competitive Earmarked Research Grant (CERG)Innovation and Technology Commission Innovation and Technology Fund (ITF)Hong Kong Arts Development Council Project Grants Programme (PGP)Hong Kong Jockey Club Hong Kong Jockey Club Charities Trust Fund (HKJCCTF)OverseasAustralian Research Council Discovery – Projects (DP)Louisiana State Board of Elementary & Secondary Education Louisiana Quality Education Support Fund (LQESF)

25. A summary of practices adopted by these organisations is at Appendix 6.



P.11

Comparison of the practices of the QEF and fund granting organisations surveyed

26. In general, the funding organisations researched operate in an open and transparent manner. They engage independent assessors to vet

funding applications. They monitor projects by examining regular reports submitted by grantees and conducting site visits on an ad hoc

basis.

27. A comparison of the practices of the QEF and the fund granting organisations surveyed is shown below.

Aspect Practice of QEF Good practice of organisations surveyedApplication assessmentAssessor Vet and approve applications by in-house staff and

committee membersAssess applications by independent peers (except ITFand HKJCCTF) and approve applications by in-houseboard of trustees or committee

Assessment criteria Use relatively general assessment criteria(e.g. quality of proposal)

Set out clear and specific assessment criteria(e.g. whether unique characters have been identified inthe project, whether student’s performance will beimproved as a result of the project?)

Transparency Limited publicity on some operational information High transparency (e.g. publicise details of assessmentprocess, disclose assessors’ assessment to applicants)

Appeal mechanism No publicity on appeal mechanism Accept appeals against procedural matters only.Appeals against professional judgement not accepted



P.12

Aspect Practice of QEF Good practice of organisations surveyedProject monitoringProject progressmonitoring

Ø Monitor progress of projects by in-house staff,committee members, external/expert reviewers

Ø Examine progress reports submitted by grantees atregular intervals

Ø Conduct at least one site visit or interview for eachproject

Similar practices adopted except that site visits areusually conducted on an ad hoc basis

Financial monitoring Ø In-house staff check financial reports and paymentreceipts

Ø Disburse grants upon receipt of satisfactoryprogress/financial reports

Ø Delegate financial monitoring responsibility torecipient organisations

Ø Require recipient organisations to appointindependent auditors to audit project expenditure

OthersElectronic submission ofapplications

Not accepted Accepted

Customer service No formal channel to collect feedback fromapplicants

Collect feedback from grantees periodically

Findings and Recommendations


P.13

28. We have identified the following issues for re-engineering –

(a) Organisation and planning

(b) Process

- Application call and assessment

- Project monitoring

- Administration and support

(c) Customer services.

29. The following paragraphs describe the issues and proposed

improvements. Unless otherwise specified, tangible

benefits of the proposed improvements are estimated based

on the workload in 2001 and findings of the workload

estimation by the QEF Secretariat.

Organisation and planning issues

Committee structure

30. QEFSC is assisted by two subcommittees, namely ASC and

PMSC. The two subcommittees are served by different

members. Each subcommittee is divided into four subject

panels3. Most of the panel convenors are members of

QEFSC.

31. In general, ASC only functions in the first half of each year

to assess funding applications whilst PMSC monitors

projects round the year. Apart from cross membership with

QEFSC, there is no formal channel for information flow

between ASC and PMSC.

32. While the current structure enables the two subcommittees to

specialise in their own tasks, it has the following constraints –

Ø ASC members may not be aware of the implementation

issues and outcomes of the supported projects. On the other

hand, PMSC members may not fully understand the

rationale behind the decisions made by ASC; and

3 The four subject panels are effective learning, all-round education, school-

based management and education research, and information technology.



P.14

Ø ASC has to process a large number of applications in a few

months’ time. The workload of PMSC is more evenly

spread since ongoing projects are implemented at different

time.

þ Restructure ASC and PMSC to form specialised

subcommittees to oversee projects throughout the

project periods

33. Two options are identified to address the above constraints –

Option

(a) - redeploy ASC andPMSC members on arotational basis

(b) - Restructure ASCand PMSC to formspecialisedsubcommittees tooversee projectsthroughout the projectperiods

Feature

Members are subject toposting to another sub-committee on a regularbasis

Members take care ofprojects from assessmentto monitoring

Pros Existing committeestructure maintained

More flexible deploymentof subcommittee membersto meet fluctuatedworkload

Cons Members’ contributionmay not be maximised

Structural change required

34. Having evaluated the pros and cons of the above two options,

it is recommended that Option (b) be adopted.

35. To tie in with the existing categorisation of projects, four

specialised subcommittees, viz Effective Learning, All-

round Education, School-based Management and Education

Research, and Information Technology, can be established.

The number of subcommittees and their size may vary

according to the prevailing workload. Under the new

structure, subcommittee members can follow each project

from application assessment to project completion. With the

through-process approach, more members can be deployed

to vet applications thus speeding up the assessment process.

Benefits

36. The new structure has the following benefits –

Ø as the assessment and monitoring processes are interrelated,

members of the proposed subcommittees will be able to

apply the knowledge and experience gained from one

process to another;

Ø with greater understanding of the approval rationale,

members will be able to discharge their monitoring duty

more effectively; and

Ø the earlier completion of the assessment work would allow

more time for QEFSC to conduct the post-assessment

review to explore improvement opportunities, if any.



P.15

Budget planning

37. The QEF usually calls for application annually but it does

not have an annual budget. It allocates grants based on

merits of individual funding applications. In 2000/01, the

Fund had a loss of $551 M from investment. However, the

QEF spent a record high amount of $1,026 M on grants and

other expenses (e.g. promotion and dissemination activities,

honorarium payment, etc.).

38. Under the existing practice, it is difficult for the QEF to

maintain its ability to fund worthwhile projects to promote

quality education in the long term, given the fluctuation in

number of applications and capital nature of the Fund.

þ Formulate an annual budget for the QEF

39. To tie in with its annual grants programme, the QEF is

recommended to formulate an annual budget taking into

account the fund balance and fund flow.

40. The following budget models can be considered –

(a) Projected cash income (perpetual fund)

- Use the projected cash yield (e.g. dividends from

equities, bond yield and interest from time deposit) to

fund the grants programme. This model is adopted by

the Sir David Trench Fund for Recreation.

- Cash yield provides a relatively steady source of

funds for allocation. As advised by the Treasury, the

current rate of return is about 3% per annum.

- The available grant per annum is about $100M.

- Since the investment tools can always generate

income to the Fund, no floor level on the fund

balance is required.

(b) Expected return on investment (perpetual fund)

- Use the expected investment return (i.e. cash yield

and capital gain) to fund the grants programme. The

Beat Drugs Fund uses this budget model.

- The availability of funds for allocation hinges on the

investment return, which is relatively volatile and

subject to the fluctuation of the economic cycle. As

advised by the Treasury, the expected rate of return

in medium term is about 4% per annum.

- The available grant per annum is about $139M.

- If it is intended to maintain the Fund’s sustainability,

a floor level needs to be set for the Fund.



P.16

(c) Fixed expenditure (non-perpetual fund)

- Limit the life span of the Fund to say, 20 years, and

deduce the amount of funds that can be made

available each year. The Fund will deplete in a pre-

determined period. By then, the funding support for

promoting quality education will discontinue.

- Based on the expected rate of investment return of

4% per annum, the available grant (including

income/loss on investment) per annum is about

$268M for 20 years.

41. Detailed fund flow projections of the above three models are

shown at Appendix 7.

42. QEFSC is recommended to explore, in consultation with

the QEF Investment Committee, the feasibility of operating

as a perpetual fund to maintain its sustainability. It is necessary

to decide on the appropriate budget model for the fund to

maximise its contributions to support quality education.

43. Upon implementation of chosen option, the QEF should

publicise the amount of funds available for allocation in

each application call. The QEF should also prioritise

worthwhile projects according to their assessment scores

and fund the most meritorious ones to gear with the

availability of funds in the financial year.

Benefits

44. The following benefits will be achieved –

Ø with a budget model in place, the QEF is able to formulate a

long term strategy for its grants programme;

Ø funds are available to provide continual support to quality

education projects (for perpetual fund model only); and

Ø it facilitates the applicants’ planning for new projects by

disclosing the funds available for allocation in each year.



P.17

Process issues

AApppplliiccaattiioonn CCaallll aanndd AAsssseessssmmeenntt PPrroocceesssseess

45. The QEF usually calls for funding applications annually. It

classifies applications into the following three categories for

processing –

Category Grant soughtSmall Not exceeding –

Ø $1M for IT projectsØ $0.6M for all-round education projectsØ $0.5M for all other projects

Medium Between the grant limits of small-valueapplications and $5M

Large More than $5M

46. The Secretariat assesses small-value applications whilst

ASC and QEFSC vet applications of medium to large value.

47. Upon endorsement by QEFSC, successful applicants are

required to sign an agreement with the Trustee, the Director

of Education Incorporated. The agreement stipulates the

grant conditions, grant disbursement arrangement and

termination conditions.

Endorsement mechanism for small-value applications

48. Currently, all small-value applications, which represent 70%

of the applications received in the fourth call or 21% of the

total grant sought, have to go through the QEF Secretariat,

ASC and QEFSC. The average amount of grant sought by

these applications is about $0.3M.

49. The Secretariat assesses small-value applications. ASC

considers the Secretariat’s assessments and makes

recommendations to QEFSC for endorsement. QEFSC

examines ASC’s recommendations and makes the final

decision. The process requires QEFSC to assume an overall

control of the assessment results.

50. In recent calls, QEFSC largely accepted the recommendationsof ASC in endorsing small-value applications –

Call No. of small-value applicationsReceived With different views

3 2 465 7 (0.28%)4 2 465 2 (0.08%)

51. The existing mechanism has the following drawbacks –

Ø multiple handling of small-value applications by ASC and

QEFSC; and



P.18

Ø longer assessment time.

þ Delegate the authority for endorsement of small-value

applications to ASC (or the proposed subcommittees)

52. QEFSC is recommended to delegate its authority for

endorsement of small-value applications to ASC (or the

subcommittees proposed in paragraph 34 above) in

accordance with Clause 4(e)(iii) of the QEF trust deed4. The

terms of reference of QEFSC and ASC should be amended

accordingly.

53. For ease of implementation, a separate budget should be

established for funding such applications. The amount or

portion of budget should be made based on the past fund

allocation pattern for small projects (the average share for

small projects in the third and fourth calls is 23%).

Benefits

54. The proposed endorsement mechanism will –

4 Clause 4(e)(iii) stipulates that QEFSC may make and subsequently amend

standing orders on matters relating to application assessment includingconsideration of applications for funding and to make recommendations to theTrustee

Ø enable QEFSC to concentrate its effort on the assessment of

medium- and high-value applications;

Ø shorten the assessment time of small-value applications by

about 21 days5 and hence reducing the lead time for project

endorsement; and

Ø in the Secretariat, save about 0.27 PO/APO and 0.06 project

clerk (or $0.16 M per annum).

Budget items of QEF projects

55. Although the QEF requests all applicants to provide an

itemised breakdown of the budget proposal, there is not a

standard list of budget items established for reference by

applicants.

56. The lack of standardised budget items creates the following

unnecessary workload –

Ø In preparing the project budget at Schedule II of the

Agreement, project teams of the Secretariat group the

detailed expenditure items of each approved project into no

more than ten block items.

5 Based on the ASC and QEFSC decision dates of some 2 400 small-value

applications processed in the 4th call.



P.19

Ø When the Accounts Unit audits the financial reports, it

breaks the block items down again into detailed expenditure

items.

þ Introduce standard budget items

57. To facilitate preparation of budget and financial checking,

the QEF is recommended to standardise the budget items

and define their ambit.

58. The application form should be re-designed so that each

budget item can be clearly presented and explained in the

application.

Benefits


Ø eliminate the workload on grouping/re-grouping of budget

items by project teams and the Accounts Unit; and

Ø save about 0.34 account clerk and 0.08 accountant (or

$0.06M per annum).

Peer review

60. Application assessment, by nature, is a complex task which

involves expertise and professional judgement in different

fields. The peer review system can provide an option for

speeding up the application assessment while maintaining

the quality.

61. According to the United States Nuclear Regulatory

Commission, a peer review is defined as –

A documented, critical review performed by peers who are

independent of the work being reviewed. The peer’s

independence from the work being reviewed means that the

peer, (a) was not involved as a participant, supervisor,

technical reviewer, or advisor in the work being reviewed,

and (b) to the extent practical, has sufficient freedom from

funding considerations to assure the work is impartially

reviewed.

62. It features the following characteristics –

Ø expert inputs;

Ø independence; and

Ø wider access to external support.

63. According to the above definition, the QEF system of using

QEFSC and ASC members, who are part of the endorsement

authority, to assess applications is not considered as a peer

review system.



P.20

64. Peer review is widely used in local and overseas academic

communities. Indeed, apart from the Innovation and

Technology Commission and Hong Kong Jockey Club, all

fund granting organisations surveyed in this review use

independent peers to assess applications. Merits and

constraints of peer review are highlighted at Appendix 8.

Feasibility of adopting peer review

65. The cost of adopting the peer review will depend very much

on the review fee, the number of applications received and

their complexity. Taking the fourth call as an example, our

analysis at Appendix 9 indicates that peer review is

financially viable.

66. There are concerns that the QEF may not be able to engage

sufficient peers to process the large amount of applications.

However, if the number of applications remains at the

current level of about 1 0006, only 150 peers7 will be needed.

The QEF may be able to engage the required peers. Hence,

the peer review may be a viable option.

6 A total of 857 applications are received in the current call (i.e. the 5th call).7 Assuming that each application is assessed by 3 peers and each peer assesses

20 applications

67. The QEF engages peers to monitor some of the approved

projects. There are currently over 700 peers of education

professionals on QEF’s “reserved” list. It can serve as base

for the QEF to solicit peers for application assessment. The

QEF should further expand the list to broaden its coverage in

various educational fields.

þ Examine the feasibility of peer review

68. The QEF is recommended to examine, in the longer term,

the feasibility of peer review taking into account the

following –

Ø expected number of applications having regard to the budget

system;

Ø willingness of peers in taking on assessment work; and

Ø availability of peers with the required expertise.

69. The implementation of peer review will bring about

profound changes to the QEF operations. ASC (or the

proposed subcommittees) will consider the peers’

assessment and make endorsement on small-value

applications. For medium- and large-value applications, the

subcommittee(s) will recommend worthwhile projects for

QEFSC’s endorsement. As such, the role of ASC (or the

proposed subcommittees) will be shifted from assessment to



P.21

endorsement of applications and members will be able to

focus on formulating strategy and policy to further promote

quality education.

70. Upon confirmation of the feasibility of peer review, the QEF

should draw up a plan to implement the system by phase to

ensure that qualified peers are available for assessment work.

The QEF may conduct a pilot exercise for the peer review on

one to two categories of projects to test and refine the

handling procedures.

71. To ensure the consistency in assessment, the QEF Secretariat

should develop, in consultation with QEFSC and ASC, clear

and specific assessment criteria for reference by the peers.

The numerical marking scheme adopted by the Louisiana

Quality Education Support Fund may serve as a good

reference ( Appendix 10).

Benefits

72. Peer review system could bring about the following benefits –

Ø provide wider access to expertise of different fields;

Ø encourage greater participation of education professionals in

QEF work;

Ø enhance the credibility of assessment;

Ø provide greater flexibility in handling fluctuated workload;

Ø reduce workload of the QEF committees and Secretariat;

and

Ø save about 3.87 PO/APO (or $80,000 per annum after

offsetting the review fee payable to peers) upon smooth

running of the peer review system.



P.22

PPrroojjeecctt MMoonniittoorriinngg PPrroocceessss

Monitoring mechanism

73. The QEF monitors approved projects to ensure that they

comply with the agreement and to identify good practices

for dissemination. The monitoring process includes –

Performance monitoring

Ø paper monitoring: evaluate progress/final reports (including

products and deliverables) submitted by grantees; and

Ø on-site monitoring: conduct at least one site visit/interview

with grantees and parties concerned, or on-site observation

at project functions.


Ø examine financial reports and end-of-project financial

statements submitted by grantees.

Grant disbursement

74. The QEF disburses grants to grantees according the agreed

payment schedule set out in the agreement. The actual

payment is subject to the timely submission of satisfactory

progress and financial reports by grantees. This

arrangement is generally in par with the practice of other

fund granting organisations surveyed.

Project termination/closure

75. During the implementation stage, the Trustee may terminate

the agreement if the grantee –

Ø fails to commence or proceed with the project after the

agreed commencement date;

Ø is in breach of the grants conditions (e.g. persistently

neglecting to carry out its obligations, misuse of fund, etc.);

Ø shall die or become bankrupt or being a corporation shall go

into liquidation; or

Ø provides incorrect or untrue project information and/or

intends to mislead the Trustee.

76. The termination arrangement effectively protects the right of

the QEF.

77. A project is considered completed when the grantee

completes the project in accordance with the agreement and

QEF satisfies with its reports and deliverables.



P.23

Performance monitoring

78. Irrespective of the project nature and complexity, the

Secretariat, PMSC and external/expert reviewers observe

the performance of approved projects by paper and on-site

monitoring.

79. As at 1 April 2002, the QEF has over 4 700 progress/final

reports to be evaluated and some 770 site visits to be

conducted. Based on the current pace of work, it will take

about 23 man-months8 to complete these assignments.

Given the large amount of workload, the QEF has to change

its existing mode of performance monitoring.

þ Classify projects for monitoring

80. To enable more efficient monitoring of projects, the QEF is

recommended to classify projects into the following two

categories –

8 Assuming that 30% of the projects are monitored by Secretariat staff (i.e. the

existing distribution) and only the latest progress reports of ongoing projectsand final reports of completed projects will be evaluated.

Category

Description

A Projects that require both paper monitoring andon-site monitoring

B Projects that can be monitored effectively bypaper monitoring alone (on-site monitoring canbe conducted on a sampling basis)

81. The followings are some useful criteria for categorising

projects –

(a) project complexity;

(b) past performance of similar projects;

(c) track record of grantee;

(d) amount of grants allocated; and

(e) project duration.

The QEF may devise further objective yardsticks, taking

into consideration members’ professional advice and past

monitoring experience. These criteria should be publicised

for applicants’ information.

82. All projects should be categorised at the assessment stage.

The category should be reviewed during progress evaluation

and site visits. The progress evaluation and site visit forms

should be redesigned accordingly.



P.24

Benefits

83. The proposed arrangement will bring about the following

benefits –

Ø reduce the workload of the QEF Secretariat (since

professional judgment is required in determining the project

category, the saving cannot be quantified at this stage); and

Ø enable the QEF to concentrate its resources on monitoring

“high-risk” projects.


84. The QEF supports a wide range of projects with grants

allocated ranging from about $2,000 to over $160 M. To

ensure the proper use of grant money, the QEF requests all

grantees to submit periodic financial reports and end-of-

project financial statements together with supporting

invoices/receipts. Grantees have to certify that the

information provided is true and correct.

85. The Accounts Unit is responsible for checking financial

reports and end-of-project financial statements according to

a set of systematic accounting procedures. The Accountant

focuses on financial reports submitted by tertiary institutions

and performs sample check on the financial reports

examined by account clerks.

86. Majority of the grantees are from public sector schools,

which have stringent internal financial control in place (as

required by the Education Department). The existing

financial control over small projects could be relaxed.

þ Rationalise the financial monitoring mechanism

87. The QEF is recommended to relax the submission

frequency of interim financial reports –

Ø from twice/thrice a year to once a year for small projects

(say, those costing $0.4M or below); and

Ø from thrice a year to twice a year for large projects (say,

those costing over $0.4M).

88. In line with the general practice, the QEF should also request

grantees to certify that the grant has been spent in

accordance with the terms and conditions of the agreement.

The contents of the financial report and end-of-project

financial statement should be modified to the effect that

grantees are explicitly made accountable for project

spending.



P.25

89. In addition, the practice of submission of invoices/receipts

to the QEF Secretariat should be ceased. However, the

requirement for grantees to keep such documents for two

years should be retained.

90. It is expected that the proposed change for monitoring small

projects will not bring about significant risk to the QEF

because transactions are reported in the financial reports and

end-of-project financial statements.

91. Nevertheless, to safeguard the interest of the QEF, it should

conduct random check on the project expenditure on site

(say, on 10% of the projects).

92. The QEF should also consider barring application from a

recipient organisation in the coming call if –

Ø there are serious irregularities on project expenditure; or

Ø the overall rating of the project (as evaluated in the final

report) is below average or poor.

93. Similar practices are also adopted by the Arts Development

Council and the Louisiana State Board of Elementary and

Secondary Education (see Appendix 6 for reference).

Benefits

94. The revised procedure will bring about the following

benefits –

Ø the administrative workload of both the grantees and the

QEF will be reduced;

Ø notional space savings arising from reduction of report

submission frequency and cessation of submission of

invoices/receipts; and

Ø 1.15 account clerk and 0.19 accountant ($0.18M per annum)

can be saved.

Project variation

95. The QEF grants funding to each project on a cash-limited

basis. It sets out the funds allocated to individual items of

the project on the project agreement. According to the

prevailing terms and conditions of the agreement, grantees

are required to seek prior approval from the QEF for

incurring expenditure exceeding 10% of the earmarked

funding for individual budget items.



P.26

96. Under this arrangement, grantees have to apply to the QEF

for virement even if the amount involved is very small.

þ Empower grantees of small projects to reallocate funds

among non-salary budget items

97. The QEF is recommended to allow grantees to reallocate

funds freely among non-salary budget items for projects

costing no more than $400,000 (which covers about 70% of

the approved projects). The fund vired should not deviate

from the ambit of the project.

98. Grantees should seek approval from the QEF for

reallocation of funds between salary and non-salary budget

items or for other purposes.

99. It is anticipated that the proposed change will not cause

significant risk to the QEF as grantees are still required to

submit end-of-project financial statements for auditing

purpose. After all, they are subject to random audit from the

QEF.

100. For projects costing over $400,000, grantees should

continue to observe the existing practice.

101. The terms and conditions of the project agreement should be

revised accordingly.

Benefits

102. The new arrangement will have the following benefits –

Ø increase the flexibility of fund reallocation;

Ø reduce the workload of both the grantees and QEF;

Ø speed up the implementation of projects which require

virement under the existing practice; and

Ø save about 0.18 PO/APO and 0.13 project clerk (or $0.11 M

per annum).

Project asset

103. Grantees are required to maintain a register of fixed assets

acquired with QEF grants. Majority of the fixed assets

includes audio-visual equipment, musical equipment, sports

equipment, computer hardware and software, and office

equipment. Many of the assets, particularly those IT and

electronic equipment, are of short life cycles. Their values

will depreciate significantly in a few years’ time.

104. Due to heavy workload on checking financial reports, the

Accounts Unit has just started checking the fixed asset

registers submitted by grantees.



P.27

105. As all the assets are owned by the QEF, it needs to spend

efforts to manage the fixed assets. Similarly, grantees are also

required to check and maintain the inventory for reporting to

the QEF.

þ Transfer the ownership of project assets to grantees

106. To reduce the inventory checking work, the QEF is

recommended to allow grantees to hold the titles of –

Ø all IT project assets (which are of short life cycle);

Ø less expensive non-IT project assets, say, those with a total

value below $100,000; and

Ø assets without reallocation potential, e.g. air-conditioning

plant room, computer network, etc.

107. For valuable assets, the QEF should –

Ø reserve the right to request grantees to make available the

assets for share use with other grantees;

Ø retain titles of valuable assets and the right to demand the

return of the assets within, say, three years after project

completion; and

Ø request grantees to check and confirm annually inventory of

the assets with allocation potential.

Benefits


Ø clear ownership and accountability of project asset;

Ø reduce the workload of the QEF on inventory checking; and

Ø cultivate a sense of belongings and enable efficient

deployment of assets.



P.28

AAddmmiinniissttrraattiioonn aanndd ssuuppppoorrtt pprroocceessss

Manual operations

109. Many daily operations of the QEF are labour intensive.

Some examples are given below –

Ø enter key information of applications/reports received into

the QEF database system, including 4 791 progress reports,

2 291 evaluation reports, 5 037 financial reports and 1 842

site visit reports (2001 statistics);

Ø input payment information into Treasury’s Payment of

Creditors System (POCS); and

Ø prepare and post or fax over 40 000 routine correspondence

annually to applicants, grantees, QEF committee members,

external reviewers, etc.

þ Automate manual operations

110. To streamline the operating procedures for greater efficiency,

the QEF is recommended to computerise its manual

operations. An illustration of the proposed computerisation

is at Diagram 6.

Diagram 6 - The proposed computer system

Funding applications,progress reports,financial reports,deliverables, etc

Applicants/grantees

Proposedcomputer system

付款給$

Secretariat staff, committee/sub-committee members,external/expert reviewers

PublicLAFIS/POCS

Internet

Project proposals,deliverables

Assessment reports,evaluation reports,site visit reports, etc

Funding applications,progress reports,financial reports,deliverables, etc

Grant payment



P.29

111. The main functions of the proposed system are as follows –

Ø accept submission of customisable electronic forms (e.g.

application form, progress report, etc) and project

deliverables or their particulars through Internet and e-mail;

Ø validate completed electronic forms against pre-defined

business rules (e.g. check whether application for virement

is required for a financial report)

Ø automate data capture of electronic forms received;

Ø provide matching function to facilitate the selection of peers

for application assessment and project monitoring;

Ø support the option of e-Agreement (i.e. electronic agreement

authenticated by e-certificate);

Ø provide alert function on pre-defined events (e.g. reports

outstanding for 2 weeks);

Ø automate issue of reminders electronically;

Ø automate grant payment through interface with POCS;

Ø compile and generate standard and ad hoc management

information reports;

Ø export data or records into common PC software formats,

e.g. Excel, Access and dBASE, for further analysis; and

Ø as part of the dissemination exercise, provide access to

approved proposals and deliverables by the public.

112. Details of the system requirements and development option

(i.e. client-own system versus lease-based IT service) should

be further refined at the Feasibility Study stage.

113. It is estimated that the capital cost of the proposed system is

about $4.5 M.

Benefits

114. The proposed computerisation will –

Ø speed up information flow between the Secretariat and other

parties, e.g. QEF committees, applicants, external/expert

reviewers;

Ø reduce workload of the Secretariat on data entry;

Ø increase data accuracy and consistency; and

Ø allow sharing of data and management information.

115. Preliminary assessment indicated that a total of $1.1M

(0.5 PO/APO, 3 project clerk and 0.4 ACO posts (or $0.7M

per annum) and production cost for CD-ROM and

maintenance cost of the Cyber Resource Centre ($0.4M per

annum)) could be saved annually. The detailed calculation

is at Appendix 11.



P.30

Customer services issues

Application arrangements

116. The QEF usually calls for applications annually. It

announces the application arrangements in September and

invites funding applications from October to December.

117. The QEF introduces new measures from time to time to

improve the application process. Some major changes

introduced in the recent two calls are listed below –

Call Major change4 Ø Accept a maximum of 3 applications per

organisation5 Ø Accept 1 application per school

Ø Not accept applications from tertiaryinstitutions and non-school organisations

118. According to the grantees interviewed, they usually plan and

design projects well before the application call. The

introduction of major changes in October may render some

projects being designed ineligible for QEF funding and lead

to abortive work.

þ Announce new application arrangements earlier

119. The QEF is recommended to formulate and announce new

funding policies and application arrangements at the earliest

possible time, preferably before the summer break.

Benefit

120. Adequate time is allowed for applicants to prepare their

applications. Thus, abortive work of applicants will be

reduced.

Conflict of interest

121. All Secretariat staff, including those employed on contract

terms, are required to observe the provisions stipulated in the

Civil Service Regulations and Civil Service Bureau

Circulars on avoidance of conflict of interest. At present, all

members will not participate in assessing applications from

organisations they belong to.

122. In the past four calls9, government departments made 47

applications seeking a total grant of $475 M. The QEF

approved 19 of the applications with grants amounting to



P.31

$296 M (or 58%). The success rate was much higher than

the overall average of 25%. It may give an impression that

government departments have an edge over other applicants.

123. Almost half of the government applications were submitted

by ED. Given that ED will likely take over the management

of the QEF Secretariat from EMB after the anticipated re-

organisation of the EMB and ED, it is appropriate to

establish a mechanism to avoid any perceived conflict of

interest of the Secretariat staff in handling applications from

government.

þ Government applications should be processed by

committee members or peer reviewers

124. The QEF Secretariat is recommended to refrain from

assessing government applications which should be

processed by committee members or peer reviewers

proposed in paragraph 68 instead.

Benefit

125. Potential conflict of interest will be reduced.

9 The QEF does not accept applications from non-school organisations,

including government departments, in the 5th call.

Operation transparency

126. The QEF has introduced a number of measures to ensure that

funding applications are processed in a fair manner.

However, some of the measures are not publicised.

127. Findings of the interview with grantees reveal that they are

generally unsatisfied with the existing level of operation

transparency of the QEF.

þ Publicise key operational information

128. To enhance its operation transparency, the QEF is

recommended to adopt the following measures –

Ø establish and issue clearer assessment criteria (which are

being reviewed by the QEF) and funding policy;

Ø publicise all information on the assessment procedures and

mechanism on avoidance of conflict of interest;

Ø publicise all information on the assessment regime, e.g.

profession and working experience of committee members;

Ø make available pricing standards (which are being compiled

and documented by the QEF in the operation manual) for

reference by applicants;



P.32

Ø provide more details on assessors’ assessment to applicants,

particularly on high-value applications. This would enable

applicants to better appreciate the professional views of

assessors and to improve the quality of their future proposals;

and

Ø conduct a post-assessment review after each application call

and publicise QEF’s general comments on the applications

received, e.g. common problems spotted such as including

irrelevant activities in project proposals;

129. In addition, the QEF should establish formal channels for

collecting feedback from applicants and grantees. For

example, the QEF could request grantees to complete a

questionnaire on QEF services in the final report.

Benefits


Ø reduce unnecessary misunderstanding and speculation from

grantees;

Ø speed up the budget negotiation process since pricing

standard and funding policy are made known to applicants;

Ø reduce abortive work by both grantees and the QEF. Being

familiar with the funding policy, grantees will be more likely

to submit proposals meeting the requirements of the QEF;

and

Ø better understand the requirements of applicants and

grantees. The QEF will be in a better position to further

enhance its service delivery and credibility.

Appeals and complaints

131. The QEF has a mechanism to handle appeals and complaints.

It considers all types of appeals and complaints filed by

applicants on its grant decisions, which include appeals and

complaints against professional judgement and procedural

matters. The QEF will review the appeal/complaint and

inform the applicant of the review result. This mechanism

has however not been publicised properly.

þ Accept appeals against procedural matters only

132. To mirror the practices of other funding organisations, the

QEF is recommended to accept appeals against procedural

matters only.

133. To improve the operation transparency, the QEF should

publicise this arrangement for handling appeals and

complaints in its application procedure pamphlet.

Summary of Recommendations and Benefits


P.33

134. Omitted.

The proposed organisation structure

135. There is no major change to the organisation structure of the QEF Secretariat. As a result of the proposed restructuring of ASC and PMSC

into four subcommittees, and the introduction of the peer review system, the overall organisation structure of the QEF is changed as

follows.

• Formulate policies and strategies• Allocate grants for medium and large projects

QEF Steering Committee

• Formulate policies and strategies• Allocate grants for medium and large projects

QEF Steering Committee

• Allocate grants for small projects• Monitor and promote QEF -funded projects• Allocate grants for small projects• Monitor and promote QEF -funded projects

•Provide secretariat support to all QEF committees/subcommittees•Perform daily QEF operations

QEF Secretariat

•Provide secretariat support to all QEF committees/subcommittees•Perform daily QEF operations

QEF Secretariat

•Assess projects•Monitoring projects assigned

(external or expert reviewers)

Peers

•Assess projects•Monitoring projects assigned

(external or expert reviewers)

Peers

•Formulate investment policies•Monitor fund investment

QEF Investment Committee

•Formulate investment policies•Monitor fund investment

QEF Investment Committee

ELSC ARSC SBM/ERSC ITSC

Summary of Recommendations and Benefits


P.34

Manpower planning

136. The workload of the QEF Secretariat fluctuates from time to time according to the number of funding applications and approved projects.

The funding and budget policy will also have an impact on the workload. To facilitate the manpower planning of the Secretariat, we have

worked out the average manpower requirements for various tasks based on 1 000 cases –

137. The QEF Secretariat should further refine the manpower requirement for performance monitoring taking into account the expected saving

from categorising projects for monitoring which cannot be quantified at this stage (paragraph 80 refers).

No. of staff required

Existing Proposed (% saving)

PO/APO ProjectClerk

AccountsClerk

Accountant PO/APO Project Clerk AccountsClerk

Accountant

Assessment 1 000 applications 2.05 0.32 0.87 (58%) 0.21 (36%)

Performance Monitoring

Progress / final report 1 000 projects

- by Secretariat 3.89 1.41 3.58 (8%) 0.13 (91%)

- by External parties 2.54 4.71 1.96 (23%) 0.74 (84%)

Site Visit 1 000 site visits

- by Secretariat 3.12 0.21 3.12 (0%) 0.08 (64%)

- by External parties 0.76 0.21 0.67 (12%) 0.07 (65%)

Financial Monitoring 1 000 projects

Checking of financial reports 1.97 0.49 1.24 (37%) 0.36 (26%)

Project Related Administration 1 000 projects 0.28 0.17 0.24 (15%) 0.14 (18%)

* Excluding the staffing requirement for promotion and dissemination

Type of WorkUnit of

Measurement

Next Steps


P.35

Implementation

138. Upon PSG’s endorsement of this study report, the QEF is

recommended to form a working group to work out the

implementation plan.

139. As some recommendations could only be implemented with

the support of other agents, the QEF should closely liaise

with –

Ø Independent Commission Against Corruption on the

proposed procedures;

Ø Department of Justice regarding the terms and conditions for

transfer of project assets;

Ø Information and Technology Services Department on the

feasibility study and system design of the proposed

computer system; and

Ø The Treasury on the technical arrangement regarding

interfacing the proposed computer system with POCS.

Acknowledgement

140. The Study Team would like to express their gratitude to QEF

committee/subcommittee members, staff of EMB and QEF

Secretariat, grantees, external/expert reviewers, staff of the

fund granting organisations surveyed for their valuable

inputs and support to the study.

Documents

Executive Summary i - QEF · Executive Summary Review of the Operations of the Quality Education Fund Management Services Agency i Background 1. Commissioned by the Education and