Archiving qualitative data for policy research: meeting challenges and establishing

best practices

Aileen O’Carroll

Manager, Irish Qualitative Data Archive

Irish Qualitative Data Archive

• Funded by the Irish Government under the Programme for Research in Third Level Institutions (Cycle 4)

• Several major datasets being archived, including –• Growing up in Ireland study

• Life Histories and Social Change in Ireland in 20th Century Ireland

• Re-use & Archiving of Complex Community-based Evaluation Research (RACcER) project explores issues specific to archiving community-based evaluation data, also aims to raise awareness throughout research community

Tallaght West Childhood Development Initiative (CDI)

� 1 of 3 Prevention and Early Intervention Programmes jointly funded by government & philanthropy

� Aims to improve the education and well-being outcomes for children and families in historically disadvantaged areas of Tallaght West

� 5 constituent projects: early years, literacy, pro social behaviour, healthy schools, community safety

� Major emphasis on evaluation

RACcER� RACcER co-funded by IRCHSS and CDI: demonstrator project for qualitative archiving

� Objectives:

� Scope concerns, issues and requirements in qualitative data archiving through interviews

� Explore issues and challenges for researchers, participants, funders, potential data users

� Test feasibility of archiving evaluation data

� Develop protocols and procedures for archiving evaluation data

� Research Process

Consultation > Pilot Strategy > Consultation > Archiving process

Findings – qualitative data gathered in evaluations

� Range of qualitative material collected

� Variety of participant groups

� Most material recorded, anonymised and transcribed

Findings – archiving policy

� Evaluators’ research centres didn’t generally have archiving policies for qualitative material

� Range of experience of what happens to qualitative data after project

� No evidence of researchers sharing datasets informally

� Default ethics position is to destroy after 5 years

Findings – archiving experience

� No evidence of formal archiving

� Limited experience of accessing qualitative archives,

� One example for use in teaching

� Some had archived quantitative material

� Some had accessed quantitative archives

� Range of views on resource implications of prepare qualitative material for archiving

Findings – issues with archiving

� Difficulties of anonymising in small community

“Just to follow on really from that, archiving to do what really? Archiving to make something as a resource that can be used again. In the context of some of my own work, to anonymise it sufficiently is to make it meaningless. ”

“There’s interconnections, there’s politics, and there’s alliances, and there’s you know, sort of, tensions, between people and characters and actors and you know there’s a finite cast of characters. I think in that context, it is challenging about how anonymise that work and then allow it to be available and maybe you can’t, I don’t know, maybe that’s not possible”.

“I think there are enormous issues, if you talk about archiving do you anonymise transcripts to make... so then you do all this anonymisation activity, you invest time and money in it, and what you come out with is nothing. If you remove all the contextual information and so on. Then you’ve got a set of data which would essentially be akin to what was in a report. ”

Findings – issues with archiving

Findings – issues(Consent)

� “…. could you actually be happy that you would have informed consent from somebody? That they actually fully understand what it means to have this available and what the implications are. Because, to a degree, you don’t actually know what the implications are.”

� “If we were to put on our consent forms or our information sheets that data generated from interviews will be available, anonymised, as part of an archive, we’re screwed. That’s basically it. Sorry for using such

unparliamentary language, we would not get responses, not in a country like ours. ”

Findings – issues(3)

� Wide perception of fundamental difference between evaluation research and broader topics

“we’re generally speaking about evaluation and evaluation is different. It’s not asking people whether the sky is blue or green or how you perceive that, you’re talking about somebody else’s actions or activities and putting some, essentially, value on those in what you say. So therefore, there’s another level that you have to add into the mix.”

Findings – issues(4)

� Lack of clarity on who can access data and how it could be used generates fear and reduces inclination to archive

� Concern over data being used out of context, ‘twisted’, sensationalising populations, journalists gaining access

Findings – issues(5)

� Researchers concerned about being opened up to scrutiny

� PhD/masters students using archived material could reduce valuable learning opportunities gained from fieldwork

� Resource implication of archiving

Findings – advantages

� Potential to explore alternative research questions

� Contributing to cumulative knowledge of implementation of children’s services

� Facilitate comparative analysis over time or place

Findings – advantages (2)� A valuable teaching resource

� Increase the standing of qualitative research if it is open to external examination, improved validity and reliability

� Contribute to corporate memory of commissioning organisation

Findings - risks

� Reputational risk for organisations involved at all levels

� Negative publicity risk for researched community

� User agreement could be breached

� Anonymity could be compromised –

� But...

� Need to consider likelihood of risk occurring and how serious it would be if it happened and develop approperiate archiving strategies

Findings – mitigating risks� Strict controls on user access

� Archive to facilitate collaboration between researchers

� Hierarchy of access, tailored depositor and user agreements

Consultation

With

CDI Board

Building

Informed Consent

Data

Collection

Data

CostingData

Anonymisation

Process of archiving CDI datasets

Accessing Sensitivity level of the data

Risk of Identification Risk Of Harm Sensitivity Level

Little Low Low

Some Low Medium

Any High High

Archiving Strategy Appropriate to the Level of Sensitivity

Sensitivity

Level

Participant

Consent

Anon-

ymised

Access Remove

Sensitive

Text

Embargo

(Anonymised

Sensitive

Text

Removed)

30 Year

Embargo

(Sensitive

Text Not

Removed

Low Yes Yes Restricted N/A No No

Medium Yes Yes Restricted No No No

High Yes Yes Restricted Yes No No

Thank You

RACcER Team

�Irish Qualitative Data Archive

�Jane Gray

�Julius Komolafe

�Hazel O’Byrne

�Aileen O’Carroll

�Tallaght West CDI

�Tara Murphy