THINKING ABOUT RESEARCH AND RESEARCH QUALITY IN YOUR ACADEMIC WORK.

Presentation for the annual Oxford-Cambridge Exchange Pam Sammons and Linda Bakkum

WHAT IS RESEARCH? Research is a disciplined attempt to address questions or solve problems

through the collection and analysis of primary data for the purpose of description, explanation, generalization and prediction (Anderson 1998, p 6)

The nature of the subject matter determines what kind of research is valid or relevant (Pring 2000, p 6)

Reasoning: deductive (Aristotle) formal steps of logic inductive (Bacon) empirical evidence for verification Inductive-deductive moving from observations to hypotheses then back

to implications (backwards & forwards). Subjective belief must be checked against objective reality, research is

self-correcting.

WHAT IS DISTINCTIVE ABOUT EDUCATIONAL RESEARCH?• The distancing of theory from practice is associated with public and

policy scepticism about value of educational research

• Need for clarity in defining key terms identified from your literature review and as used in your study e.g. ‘good’ ‘effective’ c’ompetent’ teacher, what it means to be an ‘educated’ person

• Need to attend to the ‘logic of the discourse’ the rules implicit in the use of particular words and those to which they are logically related

• For Dewey “education concerned the development of the distinctively human capacities of ‘knowing’ ‘understanding, ‘judging’; ‘behaving intelligently’ “(Pring 2000, p 12)

• What that makes your study distinctive in relation to the field of education?

KEY FEATURES OF EDUCATIONAL RESEARCH• The attempt to make sense of the activities, policies and institutions which, through the

organisation of learning, help to transform the capacities of people to live a fuller, more distinctively human life.

• The distinctive focus of educational research must be upon the quality of learning and thereby teaching

• Much writing sets up a false dichotomy between different research traditions

• Variety in approaches to educational research is desirable, depending on questions explored and philosophical position

• Is it the ‘real’ world that we observe – or one interpreted through my own personal & subjective scheme of things?

• What is the connection between language and the world language is used to describe? After Pring (2000)

• All links to notion of clarity in writing and argument & demonstrating critical engagement with substantive, theoretical & methodological literature

WHAT IS...? A research design is “an integrated statement

of and justification for the technical decisions involved in planning a research project” (Blaikie, “Designing Social Research”, p. 15).

A research project is a temporary organisation that is created with the purpose of carrying out systematic and rigorous enquiry to address a particular problem arising from a gap in knowledge (a theoretical puzzle, a pragmatic need etc).

FEATURES OF QUALITY IN (EDUCATION) RESEARCH Rigour of research process Trustworthiness Reliability/ validity Usefulness – implications for research

methodology, for policy &/practice in education

Originality Contribution to theory?

How can you demonstrate rigour in these areas in your study?

WHAT IS PHILOSOPHY OF EDUCATIONAL RESEARCH?A “second order activity” which explores

the beliefs about the nature of (social) reality or of a phenomenon (including self and other – “what exists, what it looks like, what units make it up and how these units interact with each other”) - ontology

the beliefs about the nature of educational research knowledge (and its relationships to other kinds of knowledge) - epistemology

the beliefs about principles and values (including the right, the good and the virtuous) in the practice of educational research - axiology

(see D. Bridges, 2003, p. 15; N.Blaikie, 2000, p. 8)

WHAT IS/ARE YOUR...

Quantitative?Qualitative?Mixed Methods?

•Positivist ? •Post-positivist?•Critical Theory

•Constructivism•Pragmatic?

HOW DOES THIS AFFECT YOUR CHOICE OF:

• RESEARCH AIMS & QUESTIONS?• RESEARCH DESIGN & METHODOLOGY?

...Ontological position?

...beliefs about epistemology?

QUANTITATIVE VERSUS QUALITATIVE Some researchers have argued that it may be

appropriate to think of Qualitative & Quantitative as being on a continuum Gray and Densten (1998), Tashakkori & Teddlie 2003

‘Qualitative and quantitative choices viewed as polar opposites may be viewed as a ‘false dualism’ (Frazer 1995)

Can you clarify & justify your own view and approach in your study?

How has your view evolved over the course of your PhD research?

Pragmatism supports the use of both QUAL & QUAN methods in the same study & rejects the either/or incompatibility thesis

It considers the research questions to be more important than either the method or paradigm that underlies the method – the dictatorship of the RQ

Pragmatism avoids the use of metaphysical concepts eg ‘truth’ ‘reality’

Pragmatism presents a very practical & applied philosophy

After Tashakkori & Teddlie 2003 p 20-21

Pragmatism as the Foundation for MM Research

RESEARCH QUESTIONS The ‘big’ research question: one over-arching

question The sub-questions which help to guide your

enquiryCharacteristics of Good Research Questions Clarity Empirical focus Accessible evidence Manageable Awareness of

assumptions Awareness of

implicit values

Awareness of political implications

Related to previous research

Significant Ethical Practical use (relevant) ‘fun’ (interesting to you)

source: Ingrid Lunt.

CLARITY The question(s) in your study should be

answerable i.e can be illuminated or addressed by your methodology (you are looking to find the answer to a genuine question)

The question should be intelligible to the reader who may not be an ‘expert’ in your topic (understandable)

The questions should offer the prospect of making an ‘original contribution to knowledge’ in some way (methodologically theoretically empirically etc)

Are the terms clearly defined? Are the questions precise?

source: Ingrid Lunt.

EMPIRICAL FOCUS Require that you generate data to answer question Lead you to determine methods of enquiry and data

collection NB it is usually most appropriate for methods to

follow questions; different types of questions will lead to different approaches to research and methods of data collection, but this is not always the case

Reflect on your own Research aims/RQs: How have they evolved over the course of your study How far have they driven your choice of design &

methodology and the specific methods you are using?

source: Ingrid Lunt.

SIGNIFICANT Is there a clear rationale for the question? So what? Does this question matter? Why is it of interest and to whom?

source: Ingrid Lunt.

POSSIBLE AIMS & OBJECTIVES Description: what does it look like (what, when,

where, who)? Explanation: why did it happen? Prediction: what is to be expected? Understanding: how is it grasped in human

experience? Interpretation: what does it mean? Prescription: how ought it be? Change and emancipation: how can it be

transformed for the better? Critique and disruption: what are the limitations

and hidden assumptions? How can these assumptions be challenged/ interrupted?

Etc. (e.g., exploration, demonstration, classification)

AIMS AND CLAIMS KINDS OF RESEARCH QUESTION

EXAMPLES OF RESEARCH

Explanatory What is the relationship between?

Survey, experiment

ExplanatoryDescriptivePrescriptive

What happens if . . . ? Experiment, participatory research, action research

DescriptiveExplanatory

‘What’ and ‘why’? Mixed methods research

ExplanatoryDescriptive

What happened in the past/ how to make sense of the past?

Historical research

UnderstandingInterpretative

How can we understand a situation?

Ethnographic and interpretive/Case study

CritiqueEmancipatory

How to disrupt convention and empower participants?

Critical approaches

As by Alis Oancea.

SOME INFLUENCES ON SOCIAL RESEARCH

• These can affect choice of research topic, formulation of research questions, choice of methods, choice of research design and instruments, ethics, sample & process of data collection, interpretation of data and findings, conclusions, reporting and dissemination

• Need be self-reflective, and to exhibit reflexivity about the part played by the researcher’s own values and their potential influence on research process and outcomes

• Existing knowledge base on topic, is this a new topic of interest? (generation or testing of theory more appropriate?), resources available, availability/interest of participants

• All social research is a coming together of the ideal and the feasible

Values Practical Considerations

As by Alis Oancea.

ROLE OF VALUES & OF RESEARCHER• The value determined nature of enquiry in anti- positivist

research such as Critical theory and Constructivism, Advocacy and activism encouraged, researcher transformative intellectual or passionate participant

• What can be known is mediated by interaction between investigator and subject of investigation

• For constructivists there are multiple realities, that depend on the individuals or groups holding constructions, constructions may change/be altered and thus so can ‘realities’

• Researcher and subject are interactively linked and findings are created through hermeneutical and dialectical techniques and are relative

• Aims to critique & transform (critical theory) or to understand & reconstruct, subject to continuous revisions.

HOW VALUES MAY INFLUENCE SOCIAL RESEARCH

• Choice of research area• Formulation of research questions• Choice of method• Formulation of research design & data collection

techniques• Implementation of data collection• Interpretation of data• Conclusions drawn

BODIES OF KNOWLEDGE• Theories, propositions and explanations

accumulated through enquiry, criticism, argument and counter argument. What has survived testing and criticism…public property. Their credentials depend upon their being open to public challenge and refutation.

• Any body of knowledge can only be provisional and is open to further challenge through criticism.. The link between knowledge & certainty is broken.

• Disciplined, critical and reflective thinking is the mark of educational research, at odds with unquestioning ‘common sense’ beliefs.

POINTS TO ESTABLISH IN EXAMINING DIFFERENT RESEARCH APPROACHES & IN CRITICAL READING OF RESEARCH

• Research assumptions - are they explicit?• Aims – explanation or understanding• The subjective-objective dimension• Role and definitions of theory • Doing research/reading research• Theoretical and empirical domains• Values and interpretation• Use of findings/ audience• Stages in the development of enquiry

ISSUES IN QUALITATIVE RESEARCH

Dependability, consistency, comprehensiveness, ‘checkability’, empathy, uniqueness, explanatory and descriptive potential, confirmability, “neutrality”, applicability, transferability

Generalisability Validity Reliability

Often concerns: honesty, credibility, richness, authenticity, depth, scope, subjectivity, strength of feeling, capturing uniqueness, idiographic statements, fidelity to participants’ accounts

Enriching understanding and generating theory

Fuzzy Generalisations

Falsification Using extreme

(most/least likely to fit theory), atypical, and critical cases As by Alis Oancea.

STRATEGIES FOR...

Careful, sometimes strategic selection of cases

Intense participation and effort to develop valid and rich descriptions

Challenging theories, conventional wisdom, and prior assumptions

Letting the case “talk back” – sensitivity to diversity, uniqueness, history and context

Good preparation for fieldwork Piloting and peer and

participant debriefing Justification of decisions (e.g.

transcription; recording; types of questions; extent of ‘mapping’ and ‘summarising’ in case presentation etc.)

Awareness of transcriber selectivity and other limitations

Independent audits and audit trails

Multiple coders

Generalisability Reliability

As by Alis Oancea.

STRATEGIES FOR VALIDITY Prolonged engagement

in the field Persistent observation Rich and thick

description Leaving an audit trail Reflexive diaries Respondent validation Peer debriefing Checking for

researcher effects

Making contrast/comparisons

Ruling out spurious relations

Following up surprises Using extreme cases Assessing rival

explanations Triangulation Back translation

As by Alis Oancea.

SOME ISSUES IN QUANTITATIVE RESEARCH

Fundamentally concerned with the reliability of measures.

Stability Dependability Replicability Internal reliability Inter-observer

consistency

Generalisability Validity Reliability

Measurement valididy, face validity, concurrent validity predictive validity, construct validity, convergent validity

Role of confidence intervals

Can findings be generalised outside the sample?

Importance of sample

Concept of statistical probability

STRATEGIES FOR...

Test – Retest Chronbach Alpha Multiple coders Consider the

consistency of your observations.

Controllable, predictable, consistent, replicable.(Cohen et al. 2007)

Generalisability Validity Reliability

Appropriate instrumentation,

Appropriate treatment of statistical data

Careful sampling At best strive to

minimize invalidity and maximize validity

Careful sample selection.

Random selection can be useful because of known properties.

Be cautious with making inferences.

MIXED METHODS APPROACHES

All the same problems as with Quant and Qual! But also:

Design choice Data synthesis Can your data inform one

another? Two separate studies? Quant and Qual findings dont

match? Skill and confidence in both

research approaches? Should be more than the

sum of its parts.

Careful design of each (Qual and Quant) component.

Think about how your data might be used to inform one another.

Explore what the combined set of findings indicate.

If not confident with a particular method, hit the books, ask for help!

Issues Strategies

A Dynamic Conceptual Model for MM research• QUAN MIXED METHOD QUAL

• Sphere of Concepts (abstract operations) Purposes Questions• Deductive Qs ............................................ Inductive Qs• Objective purpose ............................................ Subjective purpose• Value neutral ............................................ Value informed • Politically neutral ............................................. Transformative • Experiential sphere (concrete observations & operations) Data

Observation• Numerical data .............................................. Narrative data• Structured process .............................................. Emergent process• Statistical analysis ............................................... Content analysis• Sphere of Influence (abstract explanations & understandings)

Theories Explanations Inferences• Deductive logic ............................................... Inductive logic• Objective inference ............................................... Subjective inference• Value neutral ............................................... Value involved• Politically neutral ............................................... Transformative• after Tashakkori & Teddlie 2003

MM Designs characterised by Multiple positions along each attribute traditionally assumed to

distinguish QUAN & QUAL eg they have both confirmatory and exploratory research questions

They are near the end of one continuum on one attribute ( eg inductive questions but near the other end of the continuum on another attribute eg statistical analysis)

Multiple Method Designs (more than 1 method or more than 1 world view

A. Multi method designs ( more than 1 method but restricted to within 1 world view (eg Quan/Quan or Qual/Qual)

B. Mixed methods designs (use of QUAL & QUAN) Mixed method research (occurs only in methods stage of a study) Mixed model research (can occur in all stages of a study )

after Tashakkori & Teddlie 2003

MM Designs characterised by

• Multiple positions along each attribute traditionally assumed to distinguish QUAN & QUAL eg they have both confirmatory & exploratory research questions

• They are near the end of one continuum on one attribute ( eg inductive questions but near the other end of the continuum on another attribute eg statistical analysis )

1. Multiple Method Designs (more than 1 method or more than 1 world view)A. Multi method designs ( more than 1 method but restricted to within 1 world view

eg Quan/Quan or Qual/Qual)B. Mixed methods Designs (use of Quan & Qual methods/data collection/analysis

strategies) 1. Mixed Method research (occurs in the methods stage of study only)2. Mixed Model research (can occur in all stages of a study)– Concurrent Mixed Method design one kind of question simultaneously

addressed by collecting & analysing QUAN & QUAL data then one type inference made from both sources

– Concurrent mixed Model 2 strands of research with both types of question, both types of data & both types of analysis then both types of inferences are pulled together to create meta-inferences at the end

after Tashakkori & Teddlie 2003

Purpose / Question

Data Collection

Data Analysis

Meta - Inference

Concurrent Mixed Model Design (Fig 26.6 p688)

Inference Inference

Data Analysis

Data Collection

Purpose / Question

Purpose / Question

Data Collection

Data Analysis

Meta - Inference

Sequential Mixed Model Design (Fig 26.8 p688)

Inference Inference

Data Analysis

Data Collection

Purpose / Question

Purpose / Question

Data Collection

Data Analysis

Meta - Inference

Fully Integrated Mixed Model Design (Fig 26.11)

Inference Inference

Data Analysis

Data Collection

Purpose / Question

Your research will be informed by your readings.

Critical reading of the literature is a major part of good research!

HOW TO READ RESEARCH ARTICLES CRITICALLY (AN APPRECIATION OF STRENGTHS & WEAKNESSES/LIMITATIONS) Identify research aims/questions? Identify nature/type of study (scholarly review,

empirical work, new or secondary analysis) Identify ontological position, epistemological &

methodological assumptions Is researcher’s value position explicit? Identify location, date, sample, methods used Examine use of theory, deductive? Inductive? Are analysis methods clearly explained ? Are conclusions appropriately supported by

evidence ? What are the implications for policy/practice?.

FINAL COMMENTS No study can be ‘perfect’ Research rigour is about clarity of research process throughout Justifying your choices, design, interpretations, conclusions Persuasion of arguments re original contribution Awareness of strengths & limitations How your research fits into existing body of knowledge Implications for policy practice, future directions for research

Your viva involves an oral ‘defence’ a justification of the rigour of your research to probe your understanding and ‘ownership’ of your study

It is helpful to practice thinking, talking about and presenting your study with special attention to demonstrating rigour