Realizing More Benefits From Logic Models · 2017-05-23 · • Logic Models are useful for understanding link between program and outcomes, but today face real challenges – accountability,

Draft

1

Realizing More BenefitsFrom Logic Models:

What Logic Modeling Can Learnfrom Benefits Realization

Presentation to Canadian Evaluation Society –

National Capital Region ChapterJune 19, 2007

June 19, 2007 1

Background

• Evaluation is intended to inform Deputies, senior governmentmanagers and taxpayers– Are outcomes/benefits being realized

– ‘Value for money’

• Logic Models are a useful Evaluation planning tool:– display program theory,

– desired outcomes or benefits that program is trying to realize, and

– identifying issues

• But Logic Models do not tell senior managers what needs to be doneto realize benefits or who should be doing it (i.e., accountability)

How can Logic Models overcome these weaknessesand be even more beneficial to senior managers?

Draft

2

June 19, 2007 2

Outline

1. What is a Logic Model?

2. Weakness of Traditional Logic Models

3. What is Benefits Realization?

4. Benefits Realization within Government of Canada (GC)

5. Comparison of Evaluation Logic Model and Benefits Realization

6. What Evaluators can Learn from Benefits Realization Results Chain

7. Conclusions

June 19, 2007 3

1. Logic Model provides ...

• Schematic of the program theory, i.e., how program issupposed to work

– to achieve intended changes and outcomes, and

– how it will make a difference for the affected stakeholders

• Visual of high level ‘road map’ linking program activities,outputs, affected stakeholders and what you want toachieve

• Way to facilitate thinking, planning, communication andshared understanding about program and its intendedoutcomes

Draft

3

June 19, 2007 4

1. Logic Models are used ...

• As the basis for:

– identifying outputs/outcomes,

– coming up with indicators and data sources for ongoing performancemeasurement, and

– developing evaluation issues and methods

• In GC context, integral part of planning:

– government program Business Case, and its performance measurement,and

– evaluation of program, policy or initiative as part of Evaluation Frameworkor Results-based Management Accountability Framework (RMAF)

• Policy Responsibility – Centre of Excellence for Evaluation, TBS

June 19, 2007 5

1. A Logic Model often looks like this ...

Objective

Activities

Outputs

ImmediateOutcomes

IntermediateOutcomes

UltimateOutcomes

Some (or all) Canadians benefit or are better off along one or more dimensions(health, wealth, literacy, security, safety, understanding who we are)

Draft

4

June 19, 2007 6

2. But Traditional Logic Models do haveweaknesses ...

Logic Models are good for identifying outputs/outcomes and evaluationissues at high level, but do not clearly identify:

• Exactly what has to be done to achieve intended outcomes

• Accountability, i.e., who is responsible for achieving outcomes

• Attribution and external factors

• Outcome box size does not reflect effort, time or resources required

• Assumptions and risks associated with achieving outcomes

Developing Logic Model may be a strategic process, but not really usedas management tool once evaluation issues are determined

June 19, 2007 7

2. The Miracles of Logic Models …

Draft

5

June 19, 2007 8

Benefits Realization can help overcomeLogic Model prescriptive and accountabilityweaknesses


2. Where does it come from?

3. Can Benefits Realization be applied toprograms and help Logic Models realizebenefits?

Over last year or two, TBS ChiefInformation Officer Branch (CIOB) pilotedBenefits Realization re-labelled as“Outcome Management”

2. Overcoming Logic Model Weaknesses …

June 19, 2007 9

Outline







7. Conclusions

Draft

6

June 19, 2007 10

3. Benefits Realization – what it is …• Traditional IT Project Management - deliver project on time, within

budget and quality targets, but business benefits often not realized

• Benefits Realization is set of activities, methods, processes, andtools for understanding, planning, tracking, managing andrealizing desired outcomes from IT initiatives or projects

• Benefits Realization methodology

– Created in Canada in 1990s

– Published in The Information Paradox

– Realization of intended outcomes or benefits includes business functionbeyond the IT project

– Logic of process to realize benefits is represented by ‘Results Chain’

– Helps manage stakeholder expectations

• Benefits Realization developed with no knowledge of ‘ProgramEvaluation’– But trying to solve similar problems

June 19, 2007 11

Assumptions / Risks

Project Initiatives

Stakeholder Initiatives

OGD Initiatives

GC Outcomes

Project Outcomes

Key Outcomes

Legend

A Key Deliverable: Results Chain links project initiatives to outputs & outcomes

3. Benefits Realization – What it can look like …

Draft

7

June 19, 2007 12

3. Benefits Realization – A closer look …Results Chain model linking new on-line tracking tools that track data on OGDcompliance to new policy and integrate with data from external stakeholders

SimplifiedExample:

Assumptions / Risks

Project Initiatives

Stakeholder Initiatives

OGD Initiatives

GC Outcomes

Project Outcomes

Key Outcomes

O-2On-line

monitoring tools

created

O-3Number of

trained employees increased

O-4Use of

tracking tools increased

O-6Tracking

costs reduced

O-7Reallocation of

resources to higher

government priorities

O-8Standardization

of data and reports

increased

O-9Compliance

to policy increased

O-10Achieve greater

compliance to policy

I-1OGDs authorize skilled resources to attend training

I-2Develop online tools to track policy outcomes

I-3Develop training

protocol and plans

I-4OGDs shift

resources from administration

to analysis

O-5Time and effort

required to produce reports

decreased

A-3Staff can

be re-focused

I-5Stakeholders modernize

their systems

A-4TBS

modernizes policies

A-1OGD senior

management commitment

A-2Data from third party sources

will be available on a timely basis

June 19, 2007 13

4. Benefits Realization withinGovernment of Canada (GC) …

• GC IT/IM projects: deliver on time, on budget to required specifications– but still challenge of achieving intended business outcomes

• CIOB exploring how Benefits Realization could increase projectsuccess and help achieve intended business outcomes

• In June 2005, ADM-level TBS Service Management Board (SMB)mandated GOL Initiative to:– … promote the Benefits Realization methodology endorsed by the TBS …

• CIOB has piloted use of Benefits Realization, e.g., Shared TravelService Initiative (STSI)

• BR viewed as supporting complex horizontal or businesstransformation initiatives by breaking down silos and crossingtraditional organizational boundaries to achieve intended outcomes

Draft

8

June 19, 2007 14

4. Benefits Realization Potential …

Benefits Realization Results Chain has potential to helpovercome weakness of Logic Model:

1. Lays out how downstream business benefits or outcomes canbe achieved

2. Supports project, program and initiative levels

3. Supports an enterprise and horizontal approach

4. Details and assigns accountabilities for realizing outcomes

5. Facilitates on-going and ex post evaluation of “hard” and “soft”benefits

June 19, 2007 15

4. But even Benefits Realization haslimitations …

Limitations of BR Results Chain• A lot of information to collect and to convey in a set of diagrams

• Too complex to present in a single diagram without extensiveexplanation

• Few senior managers are prepared to deal with this level of detail

• ‘Senior Executive’ version not always developed for each project

Draft

9

June 19, 2007 16

Outline







7. Conclusions

June 19, 2007 17

5. Comparing Evaluation Logic Modeland Benefits Realization (1 of 4) …

Planning tool - can help IT project managerand stakeholders understand logic of howdesired business outcomes can beachieved and prescribe what has to bedone to achieve outcomes

Planning tool - can help manager andstakeholders understand logic of programand theory of how outcomes can beachieved

IT Project driven – Recommended byproject manager and stakeholders

Government Program driven – Mandatedby TBS and Departmental evaluationcommittee

TBS CIOB silo – Outcome ManagementTBS CEE

Benefits Realization / Results ChainEvaluation Logic Model

Draft

10

June 19, 2007 18

5. Comparing Evaluation Logic Modeland Benefits Realization (2 of 4 cont’d) …

Chaos and accountability – clearaccountability of what has to be done toachieve intended Outcomes and who hasto do it

Chaos and hope – no clear accountabilityfor achieving Outcomes

Prescribes Business Transformation byPublic Servants and stakeholders toachieve desired outcomes

Does not address scope of change requiredby Public Servants and stakeholders toachieve outcomes

Linked to Management Improvementculture – help determine what to do toachieve intended outcomes

Linked to Evaluation culture – helpdetermine if outcomes have been achieved


June 19, 2007 19


Updated and maintained as environmentchanges and phases of project arecompleted

Once developed for Evaluation Framework,not likely to change during Evaluation cycle

Useful as planning and ongoingmanagement tool, and for monitoring andmeasuring achievement

Useful in planning Evaluation (Framework)but also maybe in Evaluation

Finite time horizon – Desired outcomesachieved

Ongoing without time limit toward UltimateOutcome


Draft

11

June 19, 2007 20


Deputies and senior managers can benefitfrom the application of the BenefitsRealization methodology

Deputies and senior managers can benefitfrom an Evaluation

Visual display is complex; not so easy tofollow

Visual display is relatively easy to follow

Explicitly addresses assumptions and risks– conveys sense of urgency and what hasto be done and who has to do it

Does not explicitly address assumptionsand risks – no sense of urgency

- Although RMAF text can address these


June 19, 2007 21

Outline







7. Conclusions

Draft

12

June 19, 2007 22

6.What Evaluators / Logic Modellers CanLearn (1 of 2) …

1. Benefits Realization Results Chain includes different stakeholders,initiatives, and assumptions and risks impacting achievement ofdesired outcomes or benefits

2. BR Results Chain lays out the complex sequence of initiatives andoutputs/outcomes and lays out the accountabilities

3. BR Results Chain is more detailed and comprehensive than a LogicModel – it tells a more complete story consistent with programtheory

4. BR Results Chain is powerful tool for dealing with:

• horizontal initiatives, projects or programs, and

• new initiatives and ‘change’ initiatives

June 19, 2007 23

6.What Evaluators / Logic Modellers CanLearn (2 of 2 cont’d) …

5. Explicitly identify what project/program, department and externalstakeholders are expected to do to help achieve outcomes, i.e.,realize benefits

6. Explicitly identify underlying assumptions about how programactivities yield outputs and how outputs are transformed intooutcomes

7. Explicitly identify external factors (i.e., risks and barriers) that mayinhibit or accelerate program achieving outcomes and how to dealwith them

8. Colour code related initiatives to convey sense of urgency

Draft

13

June 19, 2007 24

7.Conclusions …

• Current Logic Models were developed in a ‘simpler age’

• Logic Models are useful for understanding link between programand outcomes, but today face real challenges – accountability,external factors, risks, horizontality

• BR Results Chains, coming at these challenges from a differentdirection, has developed techniques and processes for addressingthese challenges

• BR Results Chains are useful for understanding how to achievedesired outcomes

• Opportunity exists to modernize Logic Models and to increase theirvalue by incorporating aspects of BR Results Chains

June 19, 2007 25

Arthur Rabinovitch, CMCMaxar Consulting Inc.

(formerly withGovernment Consulting Services)

[email protected]

For further information on the BR methodology, please contact:

David HeathPrincipal

Fujitsu Consulting

613-238-2697

[email protected]

During June and July 2007

James McGibbonPrincipal

Fujitsu Consulting

613-238-2697 ext 4401

[email protected]

Documents

Realizing More Benefits From Logic Models · 2017-05-23 · • Logic Models are useful for understanding link between program and outcomes, but today face real challenges – accountability,