Nepal - Assessing the Social Impact of Rural Energy Services Alternative Energy Promotion Center

Nepal - Assessing the Social Impact of Rural Energy ServicesAlternative Energy Promotion Center

Monitoring and Evaluation Framework

November 2009

Insert relevant PwC graphic in this space

*connectedthinking

M&E Framework

Assessing the Social Impact of Rural Energy Services

November 2009

Mon

itorin

g an

d Ev

alua

tion

2

Table of Contents

Section Page1 52 10

4 153 12

5 206 35

Development of M&E framework – the contextDevelopment of M&E framework – our approach

Familiarisation and sensitisation

Review of existing M&E framework

Proposed M&E frameworkProcess flow for MIS Reporting and Evaluation

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

3

AGENDA of the programme

Day 1: Nov 5th 9:15-9:30 - Welcome and Introduction of participants 9:30:9:45 - Introduction and Objective of the Workshop, Dr. N.P. Chaulagain 9:45-9:50 - Inauguration of the Workshop, Honorable Minister, Thakur

Sharma 9:50-10:20 - Overview of M&E framework for AEPC, RakeshJha, PwC 10:10-10:50 -Remarks: By all program officials10:50-11:00 - Vote of Thanks and Closing of the session, Dr. N.P. Chaulagain 11:00-11:15 - Tea Break Technical Session-I

11:15-13:00 - Introduction to Management Information Reporting & Monitoring System (MIRMS) Mr. Rakesh Jha and Mr. Manoj Mania, PwC 13:00-14:00 - Lunch Break 14:00-15:00 - MIRMS: Manual Data Entry and upload provisions for all

renewable technologies, Mr Manoj Mania, PwC15:00-15:45 - MIRMS: MIS reporting, Mr Manoj Mania, PwC151:450-16:00 - Tea/ Coffee break16:00-17:30 - Project Monitoring – MH programmes of AEPC, Mr Manoj

Mania, PwC

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

4

AGENDA of the programme

Day 2: Nov 6 Technical Session-II 9:00-10:00 - Demonstration of MIRMS: Scorecards, KPI Monitoring &

Dashboards – Mr. Rakesh Jha and Mr. Manoj Mania, PwC

10:00-11:00 - Training and live run, Mr Manoj Mania, PwC

11:00-11:15 - Tea/ Coffee break 11:15-12:45 - Training and live run contd…Mr. Manoj Mania, PwC

12:00 – 13:00 - Concluding remarks and vote of thanks, AEPC

13:00-14:00 - Lunch 14:15 - Departure to AEPC

5

Mon

itorin

g an

d Ev

alua

tion

Section 1Development of M&E framework – the context

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

6

Context

Low electricity access in Nepal – 35%

total; Urban- 85%; Rural – 30%

Low per capita electricity consumption

– 76 unit

Cost of grid connection not feasible at

all places – Renewable can play a

major role

Grid extension in majority of villages not

viable - poor financial condition of NEA

Total MW

HH served by INPS

HH served by Micro and

Small Hydro

HH served by Alternative

Energy

2007 700 35% 8% 2%2017 2035 50% 12% 3%2027 4000 75% 20% 5%

Electrification target under the Water national Plan , 2005

Source: NEA Annual Report, FY 2008 -09

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

7

Context

• Nepal endowed with huge potential in hydropower (>40,000 MW) and other

Renewable energy sources.

• Increasing investment in hydropower – GoN, donors

• AEPC created in 1996 to promote renewable in the country.

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

8

Context

• Diverse programmes – Anecdotal evidence of positive impact but little systematic quantification of results on the ground

• WB funded study on ‘Assessing the Social Impact of Rural Energy Services’ – Oct 2008 -Sep, 2009

• Systematic M&E framework to assess the development impact of MH schemes

• Update the business plan for AEPC

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

9

Why M&E

Improving internal management

– Better planning and budgeting

– Better implementation and reporting

Generating support

– Respond to needs of stakeholders

– Explicit commitment to track contribution to global indicators e.g. MDGs

Ensuring accountability

– Demonstrating results against money spent - Demanded by govt., donors, NGOs

– Ultimately results on the ground matter - not implementation progress

Just in case: How M&E helps ME?

– Motivates by seeing the difference it makes for people

– Learn from experience: Build on success and learn from failures…

– Incentives for better performance…?

10

Mon

itorin

g an

d Ev

alua

tion

Section 2Development of M&E framework – our approach

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

11

Development of M&E and MIS system – our approach

Section 1 – development of M&E framework

12

Mon

itorin

g an

d Ev

alua

tion

Section 3Familiarisation and sensitisation

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

13

Familiarization of the context and the program by the project team

Review of policies, regulations and impact studies

Interaction with REDP, ESAP and AEPC officials

Field Visit

– Dhunne Khola (REDP)

– Tanahun and Banglung (ESAP)

Complete overview of existing M&E

Section 2 – Familiarization and Sensitization

Focus Group Discussion at Dhunne Khola Micro Hydro site

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

14

Project sensitization of the AEPC staff

Organized a sensitization workshop at the outset of the project

DAY 1: November 18, 2008

– Introduction and Context

– Overview of M&E Concepts

– Developing the Monitoring Framework for AEPC

– The Role of Evaluation

DAY 2: November 19, 2008

– M&E framework for Assessing the Social Impacts of Rural Energy Services

– Monitoring: Key Performance Indicators

– Evaluation: Survey Conceptualization and Design

Section 2 – Familiarization and Sensitization

Sensitization workshop at AEPC

15

Mon

itorin

g an

d Ev

alua

tion

Section 4Review of existing M&E framework

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

16

M&E diagnostic review - REDP

Section 3 – Review of existing M&E framework

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

17

M&E diagnostic review - ESAP

Quarterly reports are sent to AEPC with detail work plan based on the results of the survey.

Each installation is surveyed after a month of installation by the LPO and the promoter.

Technical evaluation – after 3 to 4 months by the LPO and RRESC staff.

Mid term review of the annual work plan to check whether program objectives are met.

Evaluation of LPOs by RRESC

The field technical coordinator from the RRESC office submits progress reports to AEPC by the 12th of every month – based on pre-defined formats

Section 3 – Review of existing M&E framework

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

18

MIS – Existing & Proposed for REDP

Existing

Focus more on data gathering

Targets not reflected in MIS

Manual calculation of formula fields in few formats

Project monitoring report captured only status of completion of activity

Proposed

MIS aligned for performance monitoring

Captures targets for each measurable indicator and helps evaluation

System calculates formula based on dependent fields

System would capture date of completion to highlight delay in implementation, if any

Section 5 – Development of MIRMS

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

19

MIS – Existing & Proposed for ESAP

Existing

Client server mode visible to select group of people within AEPC office(max 8 to 10 users) – local installations required

Absence of integrated information system - separate databases

Targets not reflected in MIS

KPIs not linked to performance management of business units

Requires design change in system for meeting any future requirements

Static reports with no feature to compare and perform trend analysis

Proposed

Web based system – information available to all users at any location having internet connection

Integrated AEPC database -warehousing periodic information for analysis and evaluation in the system itself

Target driven system for measuring performance every month

Dynamic system to incorporate any future requirements

Multiple ways of viewing data and conducting analysis


20

Mon

itorin

g an

d Ev

alua

tion

Section 5Proposed M&E framework

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

21

• Primary reporting interface for meeting all the reporting requirements of the

management(financial, technical, operational, projects, community

participation, capacity building)

• Eliminate the burden of disparate electronic and paper based data collection

requirements

• Provide the top management specific information about the programs and its

business units at a glance.

• Enabling the management to drive strategy by evaluating performance

against the KPIs.

• Protect the confidentiality , availability and integrity of data


MIRMS - what it offers

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

22

•Using existing computer infrastructure of AEPC server for hosting the

system on the internet

•Utilising existing manpower and computer infrastructure at the field level

•Supplements existing information base of AEPC to enhance decision

making abilities

•Integration with REDP and ESAP databases for information

consolidation

•Existing formats have been retained and new formats designed and aligned

to the KPIs in the proposed M&E framework

•Reduction of work load on M&E staff by leveraging the information for

performing evaluation and meeting top managements reporting

requirements

•

MIRMS – leveraging existing capability of AEPC


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

23

•Comprehensive MIS formats designed to facilitate regular review of project

activities for all RETs

•Structured and unified formats for ease in data gathering

•Mapping of data fields to existing transactional system

•Yearly / Monthly Targets to be set in the system to compare progress

towards achieving outputs

• Flexibility in reporting and analyzing the data

•Trend analysis

•Comparative analysis

• Dynamic reporting system with no static reports – user can extract reports

in desired format at any given point of time from any location

MIS - Design and Reporting


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

24

MIRMS will interface with existing systems either through on-line or off-line mode and extract information

User friendly screens provided for mapping and uploading of external data source to MIRMS

Information extracted either on monthly or trimesterly basis depending on availability of information

MIRMS – Mapping with existing database of AEPC


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

25

Build on the existing infrastructure - optimal utilization•Existing AEPC server

•SQL server database

Developed on a 3-tier web technology framework comprising of java technology integrated with MS SQL server

Leveraged industry best practices for designing and developing a scalable system

Creation of a centralized database repository

MIS formats designed for all programs

Development of MIRMS


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

26

Proposed MIRMS


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

27

Institutional arrangement for M&E

The proposed M&E framework is aimed to:

Align and design to meet AEPC’s vision, mission and strategic priorities.

Use M&E as a strategic planning tool.

Optimise the existing set up

Establish a drill down feature for analysis of information

Generate MIS reports at every level

Top Management, Programme Managers

Department head and District Level Monitoring

Field based operation

Section 5 – Proposed M&E framework

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

28

Functions:Operational/Technical/

Financial/CapacityBuilding

Advantages of MIRMS

Level of Management

ProjectSites

Middle Management

TopManagement

Functions:Operational/Technical/

Financial/CapacityBuilding

TransactionalActivities/Complaint

Status

Process Performance

Indicators

ProgrammePerformanceIndicators/StrategicObjective

Parameters

Review Timeline Monthly

Monthly/Trimesterly

Monthly/Trimesterly/Yearly

Need based MIRMS

Monthly monitoring of operational data is done by the field staff responsible for the respective functions.

Trimesterly monitoring of performance parameters - by the district heads or monitoring officers of respective programs

The top management reviews the program and strategic parameters on a monthly, quarterly or yearly basis to take decisions and decide the policy level interventions.


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

29

Development of KPIs

To evaluate the performance of the project against defined objective/target

Facilitate review and comparison across project/ location.

Useful for setting performance targets and efficient assessment.

Input indicator

– mostly the programme fund support, contribution in kind and technical assistance by community and awareness generation programmes

Output Indicators

– mostly the number of projects/systems installed, capacity enhancement of the target beneficiaries and the community mobilization process.

– show the benefits accruing to the community through the awareness generation and mobilization process.

Outcome Indicators

– indicators that mostly are a measure of the affordability, operational efficiency and reliability of the MH programs.


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

30

KPIs – Top Management (MH – REDP & MGSP)

No. of MH projects commissioned

Installed capacity of MH

Percentage coverage of target population through awareness generation programme

Progress against projects schedule*

Project implementation delay*

Utilization of Funds

Percentage of HHs with electricity access through MH

No. of enterprises with access to MH

Value of proposed equity by the community (equivalent to labour and cash amount) vis-a-vis actual contribution

% of community facilities with access to other RETs

% of community facilities with access to MH

Percentage of Complaints resolved (consumer level) in stipulated time period

Power availability*

Collection efficiency*


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

31

KPIs – Top Management (Biomass)

Achievement of ICS installations

Achievement of Metallic Stoves installations

Achievement of Gasifier installations

Percentage coverage of target population through awareness generation programs

Total subsidy allocation

Utilization of Funds

% Achievement of Training programmes for program staff

% Systems functioning during verification

% of community facilities with access to MH

System performance in ICS

System performance in Metallic Stoves

System performance in Gasifiers


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

32

KPIs – Top Management (Solar)

Achievement of Solar systems installed

% subsidy disbursement

Avg. delay in subsidy disbursement in all Solar systems

Percentage coverage of target population through awareness generation programs

System performance for ISPS, SHS & PVPS

% of complaints resolved in stipulated time period


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

33

Proposed Institutional arrangement for M&E


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

34

Snapshot of MIRMS


35

Mon

itorin

g an

d Ev

alua

tion

Section 6Process flow for MIS Reporting and Evaluation

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

36

Periodic Data capture through MIS formats Monthly MIS Reporting MH Project Monitoring KPI Analysis & Scorecards for Business Units Performance Review through Dashboards

Section 6 – Process flow for MIS Reporting and Evaluation

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

37

Monitoring & Evaluation Framework


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

38

Data Capture Populate monthly data in MIRMS through two options:

File uploads – mapped to existing AEPC databases

Manual Entry – data entry screens

Periodicity of data capture

One time – DPR, Funding programs, etc

Monthly

Trimesterly

Annually

Data captured at different levels of the organization depending on data availability


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

39

MIS Reporting

Dynamic reporting engine supporting multiple ways of viewing data and conducting analysis


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

40

MIS Reporting

Perform trend analysis on any selected parameter


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

41

MIS Reporting

Consolidated view of past performance data


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

42

Project Monitoring

Tracks project schedules and assists program managers to take corrective measures


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

43

KPI Analysis

Single window for top management to evaluate performance through KPIs


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

44

Program Dashboards

Single view of performance data in graphical form to different levels of user


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

45

Sample - Dashboard


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

46

Snapshot of dashboards

The dashboards would facilitate the Program Mangers and M&E staff to monitor project development

designed to provide specific information about the performance of each component of the programme and present critical measurable parameters of each program at a glance.

The dashboards would provide a graphical display of the overall performance of the programs on key performance parameters like achievement of programme objectives, community reach, financial performance, etc.

The dashboards can be generated starting from the zonal level to individual level dashboards for each project/[programme


M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

47

How to migrate to MIRMS

Form a core group of people for taking ownership of MIRMS and formulating strategies for information

Identify at least one district form each region for testing and roll out

Run this parallel to the existing M&E arrangement

Develop a team of ToT

Handholding of people engaged in M&E by ToTs

Use of operation manual

Periodic review of the system – frequency of monitoring on the higher side in the beginning

– Assess performance of the M&E team and MIRMS

– Take appropriate measures as per requirement – training, modification in MIRMS


48

Mon

itorin

g an

d Ev

alua

tion

M&E Framework


November 2009

Mon

itorin

g an

d Ev

alua

tion

49

Regulatory and policy overview

1975 – Formation of Nepal Electricity

Corporation

1984 – Nepal Electricity Act

1985- Formation of NEA

1992-Electricity Act, Water Resources Act

Classification of Hydro projects

Promotion of RE technologiesLarger role for AEPCCreation of REF

De-licesing forgeneration up to 100KWDe-regulation of powersales in isolated areas Encouragement of Private sector 1992 and 2001 - Hydro

Power Development Policy

2006 - Subsidy development Mechanism

1975 – Small Hydro Development Board

Incentives to promote MH up to 1 MW

2003 -Community Electricity Distribution by

law

Encourage Private sector in DDG based RE

Creation of community REF

2006 - Rural Energy Policy

Promotion of RETsSale of power to grid by DDGs

Emphasis on decentralized models

Three Year Interim Plan

Ex-post evaluation - replicable case studies

Documents

Nepal - Assessing the Social Impact of Rural Energy Services Alternative Energy Promotion Center