Evaluating Information & Knowledge Services Using

Narrative Techniques - a case study

Evaluating Information & Knowledge Services Using

Narrative Techniques - a case study

Nerida Hart

Chair, Knowledge Management Division

Special Libraries AssociationInformation Online 2009

Nerida Hart

Chair, Knowledge Management Division

Special Libraries AssociationInformation Online 2009

Presentation OutlinePresentation Outline

What was this project all about? Potential solutions What are narrative techniques? Knowledge & Information Services approach How K&IS used narrative techniques Limitation Results Conclusions

What was this project all about? Potential solutions What are narrative techniques? Knowledge & Information Services approach How K&IS used narrative techniques Limitation Results Conclusions

BackgroundBackground

2005 managing a special library (knowledge

service) with approximately 40,000 clients 5 agencies being serviced - FaCS, Centrelink,

Child Support Agency, Social Security Appeals Tribunal, Indigenous Affairs

> 40 staff grouped into 5 service delivery sections

2005 managing a special library (knowledge

service) with approximately 40,000 clients 5 agencies being serviced - FaCS, Centrelink,

Child Support Agency, Social Security Appeals Tribunal, Indigenous Affairs

> 40 staff grouped into 5 service delivery sections

What was the problem we needed to solve?

What was the problem we needed to solve?

We were due for the dreaded review !!!! How did we prove our worth within our client

groups? Time to do this was limited and we didn’t want

to boil the ocean but concentrate on our biggest value add service - “research services”

We were due for the dreaded review !!!! How did we prove our worth within our client

groups? Time to do this was limited and we didn’t want

to boil the ocean but concentrate on our biggest value add service - “research services”

How did we go about proving our worth…How did we go about proving our worth…

Library statistics only measure input and output and we needed to show how we contributed to client outcomes

Need to cater for emergent benefits (the real benefits of the research service)

What were the real experiences of our clients Need to encourage good practices and disrupt poor

practices Looking for ways of continuously improving products

and services

Library statistics only measure input and output and we needed to show how we contributed to client outcomes

Need to cater for emergent benefits (the real benefits of the research service)

What were the real experiences of our clients Need to encourage good practices and disrupt poor

practices Looking for ways of continuously improving products

and services

Potential methodologiesPotential methodologies

Surveys - great for the number devoted managers who need ‘hard’ evidence

Focus groups - can be manipulated Interviews - time was against us Narrative - excellent - would give us

‘depth’ but which narrative technique to use

Surveys - great for the number devoted managers who need ‘hard’ evidence

Focus groups - can be manipulated Interviews - time was against us Narrative - excellent - would give us

‘depth’ but which narrative technique to use

Cynefin Complexity Framework

Cynefin Complexity Framework

Why Anecdote Circles as a narrative technique?

Why Anecdote Circles as a narrative technique?

Anecdote circles use anecdotes of personal experiences of workers to gain evidence of what is really happening in a complex organisation

They are more open-ended than focus groups and allow for the unexpected.

Focus groups are more concerned with opinions and judgements than anecdotes which are more concerned with personal experiences

Anecdotes reveal the values and behaviours of the staff – make sense of the organisation’s culture

Anecdotes link events in a meaningful way

Anecdote circles use anecdotes of personal experiences of workers to gain evidence of what is really happening in a complex organisation

They are more open-ended than focus groups and allow for the unexpected.

Focus groups are more concerned with opinions and judgements than anecdotes which are more concerned with personal experiences

Anecdotes reveal the values and behaviours of the staff – make sense of the organisation’s culture

Anecdotes link events in a meaningful way

Components of Narrative Approach using Anecdote

Circles

Components of Narrative Approach using Anecdote

Circles

Employing Narrative approach required dividing the work into 4 phases: Preparation (1/2 day) Discovery – Anecdote circles (90-120 minutes

each) Sensemaking (1 day) Intervention Design – using complexity principles

(i.e. action planning and implementation) (1/2 to 1 day)

Employing Narrative approach required dividing the work into 4 phases: Preparation (1/2 day) Discovery – Anecdote circles (90-120 minutes

each) Sensemaking (1 day) Intervention Design – using complexity principles

(i.e. action planning and implementation) (1/2 to 1 day)

Knowledge & Information Services Approach

Knowledge & Information Services Approach

Project was part of project management certification for a staff member

Parallel survey – as a control group

Narrative – use of anecdotes

Full skills and knowledge transfer from our consultants Anecdote Pty Ltd (www.anecdote.com.au)

Project was part of project management certification for a staff member

Parallel survey – as a control group

Narrative – use of anecdotes

Full skills and knowledge transfer from our consultants Anecdote Pty Ltd (www.anecdote.com.au)

Use of Narrative in K & ISUse of Narrative in K & IS

Preparation: Identification of themes:

Client information seeking behaviour Quality Accuracy and authenticity Client experiences with research staff

Preparation: Identification of themes:

Client information seeking behaviour Quality Accuracy and authenticity Client experiences with research staff

DiscoveryDiscovery

Discovery – Anecdote circles Four sessions including one regional

session 28 participants in total (6-12 per session) All sessions recorded and transcribed Anecdotes extracted from the transcripts All participants were not personally

identified in the report

Discovery – Anecdote circles Four sessions including one regional

session 28 participants in total (6-12 per session) All sessions recorded and transcribed Anecdotes extracted from the transcripts All participants were not personally

identified in the report

Survey results vs Anecdotes

Survey results vs Anecdotes

Was the information you received what you required?

Relevant/useful

Too much

Too little

Just enough

Other

Was the information you received what you required?

Relevant/useful

Too much

Too little

Just enough

Other

“We were thinking about FaCS policy priorities. The bibliography the library sent me, at very short notice, was concentrated on sources like the OECD and other government publications, because I think there was an acknowledgement that it we were going to try to introduce it to our Department we had to have sources that would be credible.”

“We were thinking about FaCS policy priorities. The bibliography the library sent me, at very short notice, was concentrated on sources like the OECD and other government publications, because I think there was an acknowledgement that it we were going to try to introduce it to our Department we had to have sources that would be credible.”

Anecdote example : Depth of Information

Anecdote example : Depth of Information

“When I got the material it was not really what (I wanted). In the end I didn’t use very much of what was suggested to me. Given that I didn’t go back and define more closely and try to work it through with the person… so it was a time factor more than anything else. I would have gone back. In the end it was easier for me to go the way I did go.”

“When I got the material it was not really what (I wanted). In the end I didn’t use very much of what was suggested to me. Given that I didn’t go back and define more closely and try to work it through with the person… so it was a time factor more than anything else. I would have gone back. In the end it was easier for me to go the way I did go.”

SensemakingSensemaking

Sensemaking One day workshop Used an independent facilitator

Objectivity Distancing ourselves from the results and emotional

involvement Enables future credibility

Identifying themes, behaviours and archetypes

Sensemaking One day workshop Used an independent facilitator

Objectivity Distancing ourselves from the results and emotional

involvement Enables future credibility

Identifying themes, behaviours and archetypes

Interventions / Action Planning

Interventions / Action Planning

Interventions Marketing for greater awareness of our services Follow up after research completed Clarification of client research requirements Revamping parts of our intranet site Enhancing our role as an information and

knowledge sharing catalyst (connectors)

Interventions Marketing for greater awareness of our services Follow up after research completed Clarification of client research requirements Revamping parts of our intranet site Enhancing our role as an information and

knowledge sharing catalyst (connectors)

LimitationsLimitations

Geographic dispersal of the clients Getting the numbers and timing right –

availability of participants (Welfare to Work priorities of FaCS and Centrelink)

Christmas looming Convincing our clients that this was important

– benefits to both us and them

Geographic dispersal of the clients Getting the numbers and timing right –

availability of participants (Welfare to Work priorities of FaCS and Centrelink)

Christmas looming Convincing our clients that this was important

– benefits to both us and them

ConclusionsConclusions

Benefits to K&IS and our clients: Tangible evidence of K&IS contribution to core

business through real examples of time saving, authenticity of information, contribution to the big picture goals of the client agencies

Skills and knowledge development of K&IS staff Stronger relationships with K&IS clients and increase

in return business Improvements in our training methods for K&IS clients Improvements in research processes which then

benefits the client groups

Benefits to K&IS and our clients: Tangible evidence of K&IS contribution to core

business through real examples of time saving, authenticity of information, contribution to the big picture goals of the client agencies

Skills and knowledge development of K&IS staff Stronger relationships with K&IS clients and increase

in return business Improvements in our training methods for K&IS clients Improvements in research processes which then

benefits the client groups

ReferencesReferences

How to run this process www.rkrk.net.au

Anecdote Pty Ltd www.anecdote.com.au

Dave Snowden’s Cynefin Framework www.cognitive-edge.com

How to run this process www.rkrk.net.au

Anecdote Pty Ltd www.anecdote.com.au

Dave Snowden’s Cynefin Framework www.cognitive-edge.com

Questions and Comments

Contact details: nerida.hart@lwa.gov.auLinked in: www.linkedin.com/in/neridahart

Skype: Neridahartau