1

Finessing the qual-quant distinction in research and evaluation

Action research into methodologies for assessing complex rural transformations in Malawi and Ethiopia.

James Copestake and Fiona Remnant

27 January 2015

ART project webpage: go.bath.ac.uk/art

2

Origins of the presentation

•Paper submitted based on Working Paper:

Assessing Rural Transformations: Piloting a Qualitative Impact Protocol in Malawi and Ethiopia•Response:

“On first (very quick) reading, this paper seems somewhat embedded in a quantitative paradigm (albeit with some narrative data).”• Invitation to resubmit, reflecting more on qual/quant distinction, e.g.

What is a quantitative or qualitative paradigm?

3

Summary• Creative mixing of qualitative and quantitative research is aided by

deconstructing and reconstructing the distinction between them.

• One approach is to review framing and data codification within any

research process (from initial scoping to use).

• This departs from the norm of assuming mixed method research

partitions (or nests) self-contained qual and quant. methods.

• To explore this we use the case study of our experience in designing

and testing a qualitative research protocol for impact evaluation of

NGO livelihood improvement and climate adaptation projects in

rural Ethiopia and Malawi.

4

Key concepts (1): framing

“If calculations are to be performed and completed, the agents

and goods involved in these calculations must be disentangled

and framed. In short a clear and precise boundary must be

drawn between the relations which the agents will take into

account and which will serve in their calculations and those

which will be thrown out…”

M Callon, editor. (1998) The Laws of the Markets, Oxford:

Blackwell. Page 16.

5

Key concepts (2): codification

“… the distinction between quantitative and qualitative enquiry

hinges less on the source of information than on the point at

which information is codified, or otherwise simplified. Early

codification permits rigorous statistical analysis, but at the same

time entails introducing restrictive assumptions which limit the

range of possible findings.”

J Moris and J Copestake (1993) Qualitative enquiry for rural

development: a review. London: ITDG. Page 1.

6

Key concept (3): partitioning

• Maintain a categorical distinction between quant and qual

approaches or paradigms. Identify ways in which they can be

mixed:

– In parallel (triangulation) or

– In sequence (e.g. qual pilot -> quant survey -> qual case study etc).

• NOT the main focus here – as this is the dominant discourse for

mixed methods, and the focus here is on one integrated

method.

7

Analytical framework

Infinitely complex reality

Simplified reality (with respect to time, space, ontology) to facilitate quantification

Framing(selection)

Codification

Reframing anddecodification(synthesis)

Research activities through time

(initial scoping, data collection, analysis,

dissemination)

Case study: the ART project

8

9

(a) Initial scoping

• How to assist NGOs gather timely and credible data for

internal and external use on the impact of their projects?

• Three strands to the research:

1. Monitoring

2. Qualitative assessment

3. Meta analysis of the usefulness of the methodology.

• Focus here only on Strand 2, piloting the qualitative

assessment tool (the QUIP)

10

Projects (X) Impact Indicators (Y)

Confounding Factors (Z)

Project 1. Groundnut value chain (Central Malawi)

Project 2. Diversification and resilience (Northern Malawi)

Project 3. Malt barley value chain (Southern Ethiopia)

Project 4. Diversification and resilience (Northern Ethiopia)

Food production

Cash income

Food consumption

Cash spending

Quality of relationships

Net asset accumulation

Overall wellbeing

Other?

Weather

Climate change

Crop pests and diseases

Livestock mortality

Activities of other organisations

Market conditions

Demographic changes

Health shocks

… more?

11

(b) Data collection

• Two independent local field researchers, without any knowledge of the project (blinding). Four-six days of semi-structured interviewing, two days of focus group discussions.

• Sample selection based on lists from separate quantitative monitoring of key household level indicators (IHM).

• Data collection instruments structured around any changes since project inception, split by life/livelihood domain: open questions followed by closed questions for each domain.

12

(c) Analysis

• Responses to open and closed questions entered into pre-formatted Excel sheets.

• The analyst uses the project theory of change to classify causal statements in the raw narrative data by attribution type: positive/negative explicit, implicit, incidental and unattributed.

• Change data (causal statements) are sorted into categories and summarised using simple frequency counts.

13

(d) Dissemination

• Short report summarising frequency with which households volunteered explicit, implicit, incidental causal explanations with respect to each impact domain.

• Lists of all causal explanations cited more than once.

• Appendix providing narrative data (sorted causal statements)

14

Causal Coding KeyChange attributed to: Code Explanation

Explicit project (positive) 1 Positive change attributed to project and project-linked activities

Explicit project (negative) 2 Negative change attributed to project and project-linked activities

Implicit (positive) 3 Stories confirming a mechanism by which the project aims to be achieving impact, but with no explicit reference to the project

Implicit (negative) 4 Stories questioning a mechanism by which the project aims to be achieving impact, but with no explicit reference to the project

Other attributed (positive) 5 Positive change attributed to any other forces that are not related to activities included in the commissioning agent’s theory of change

Other attributed (negative) 6 Negative change attributed to any other forces that are not related to activities included in the commissioning agent’s theory of change

Unattributed (positive) 7 Positive change not attributed to any specific cause

Unattributed (negative) 8 Negative change not attributed to any specific cause

Other ambiguous, ambivalent or neutral statements 9 Changes with no clear positive or negative implications

15

Responses to closed questions

Code

Main respondent

Age of respondent 1. Food

Production2. Cash income

3. Purchasing power

4. Food consumption 5. Assets

LL1 Female 61 = + - - +

LL2 Female 31 + + + + +

LL3 Male 49 + + + + +

LL4 Female 22 + + + + +

LL5 Female 31 - - - = -

LL6 Female 22 + + + + -

LL7 Male 26 + + + + +

LL8 Male 43 + + + + +

16

Frequency of narrative causal statements

Positive changes reported by households and focus groups

1

Project explicit3

Project implicit5

Other7

NoneFood production LL2, LL5, LL6, LL7, LL8

FL3, FL4LL3, LL6FL4 LL4

Cash income LL2, LL5, LL6, LL7FL1, FL2, FL3, FL4 LL3, LL4, LL7, LL8 LL4, LL7

Purchasing power LL2, LL6, LL7, LL8FL1, FL2, FL3, FL4 LL3 LL4

Food consumption LL2, LL7, LL8FL4 LL3 LL4

Relationships LL2, LL5, LL7, LL8FL4 LL3 LL1 LL,4

Asset accumulation LL7, LL8FL1, FL4 FL3 LL2, LL4

Notes: LL1 to LL8 refer to individual household codes

FL1 to FL4 refer to focus groups: FL1 Younger women; FL2 Older women; FL3 Older men; FL4 Younger men.

17

Frequency of narrative causal statements Negative changes reported by households and focus groups

Attribution 2 Project explicit

4 Project implicit

6 Other

8 None

Food production FL1, FL2, FL3 LL5FL1, FL3

Cash income LL6FL3, FL4

LL1, LL5

Purchasing power FL2 LL1, LL5FL1, FL3

Food consumption FL1, FL2 LL1

Relationships FL1, FL2

Asset accumulation LL5FL2

18

Drilling into narrative causal statements

3. Activities that implicitly corroborate the project’s theory of change (positive)

4. Activities that implicitly corroborate the project’s theory of change (negative)

LL3 The respondent said that they rely on farming both irrigation as well as rainfed. In the past, they used to rely on food for work but now they are growing their own food because they were inspired by their friends who were farming and doing better than them.

LL3 According to the Respondent, they used to rely on piece work as a main source of income but now they grow cassava, Groundnuts and sell these. This has been so because their friends encouraged them to do farming so that their welfare improves also.

LL4 She also reported that she occasionally sells her maize to supplement her income.

LL6 On new activities taken to help produce more food she said: "…I rent in several fields each season which has also helped to increase food production…"

She however bemoaned the low prices that itinerant vendors offer for their crops saying this reduces their profit margin.

LL7 They however reported that they are employing more piece work workers to work on their farms because they are mostly engaged in other activities like attending to customers in their tea room business. They said that employing several temporary workers is therefore something that they are doing differently from others because most of the other community members still do most of their farm work without the help of hired workers.

Section C: Food Production & Cash Income

19

Drivers of change

Food Production Cash Income Purchasing power Food

Consumption Relationships Assets

SHA support with groundnut crop (Support to grow Groundnut from SHA in the form of free seeds, advice and/or credit)

LL2, LL5, LL6, LL7 LL1, LL2, LL5, LL6, LL7, FL1, FL2, FL4

LL2, LL6, LL7FL1, FL2, FL4

LL2, LL8FL4

FL3, FL4

SHA 'pass on' livestock programme (pigs and goats provided by FIDP and SHA; further benefits accruing from livestock reproducing)

LL7, LL8 LL8, LL5, LL6FL2, FL4

FL3 FL1

SHA/FIDP advice on irrigation farming (Advice and some equipment provided by FIDP and SHA - treadle pumps, watering cans and sprays)

LL8, LL7FL3

FL1, FL4

SHA/FIDP advice on making manure

LL7, LL8FL1, FL3

LL7, LL8 LL7

Approaching farming as a business (Encouraged by SHA and spread through success of recipients)

LL3, LL7, LL8 LL3, LL6, LL8 LL3 LL5FL4

Support to plant more trees (Promoted by MALEZA and SHA)

LL1 FL1

LL1

Village savings and loans groups (CARE mentioned as running Village Bank)

LL8, LL7 LL3, LL6, LL8 FL1

Personal qualities (hard working)

LL6, LL7

Drivers of positive change

20

Most widely cited drivers of changeDomain Positive NegativeProject 1: groundnut seed, Malawi (n=8,4) Food production NGO support for groundnut crop (4,0)*

NGO advice on making manure (2,2)*NGO advice on small-scale irrigation (2,1)*

Low sale price for crops (1,3)*

Cash Income

NGO support for groundnut crop (5,3)*NGO pass-on livestock programme (3,2)*NGO support for farming as a business (3,0)*

Low sale price for crops (2,3)*

Cash spending NGO support for groundnut crop (5,3)*NGO support for farming as a business (3,0)*Village savings and loan groups (3,0)

Increased prices, including food (0,3)

Food consumption NGO support for groundnut crop (2,1)*

Increased prices, including food (0,2)

Quality of relationships NGO support for farming as a business (1,1)*

Economic hardship (0,2)

Net asset accumulation NGO support for groundnut crop (2,0)*

Asterisks indicate those drivers that explicitly or implicitly support or negate project theory

Meta analysis: (re)framing and (de)codification steps

within the research process

21

22

(a) ScopingKey step in terms of broad framing of the research

• Who? – Feedback from intended beneficiaries up the ‘aid chain’.

• What? – Reality check on agencies’ theory of change (confirmatory and exploratory evaluation).

• Why? – Learning and accountability.

• How? – Sample in-depth interviews and focus groups alongside project monitoring.

• When? – single visit with recall over specified period.

23

(b) Data collection• Interviews framed by semi-structured questionnaire

arranged in line with predetermined domains of impact.

• Open (not codified) in terms of potential sources of change: field worker and respondent blind to project theory to reduce confirmation bias. But dependent on field worker’s skill in summarising open conversation.

• Closed questions using Lickert scales to finish each section - inviting respondent to participate in codification.

24

(c) Analysis

• Post-hoc codification according to prior categories of

impact: positive/negative; explicit, implicit, incidental,

unattributable.

• Triangulation of open and closed question responses.

• Synthesis through identification of patterns in drivers

of change, and relating these to wider context.

25

(d) Dissemination and use• Standard format of short reports with summary tables

(codification) that can be visually and rapidly be

absorbed (synthesis) by staff.

• These also signpost how they can decodify by drilling

deeper into the sorted narrative data in annexes.

• Triangulation with data obtained through monitoring

and other methods (not covered here)

26

Conclusion

• Need for an alternative language in order to transcend

the qual/quan dichotomy.

• Synthesis (decodification and reframing) can take place

within a research method.

• This is a form of internal triangulation informed by

comparing different ways of framing and coding data

from the same source.