Quantitative Research: Important issues pertaining to …osprey.unisa.ac.za/TechnicalReports/e4.pdf · Quantitative Research: Important issues pertaining to research methodology &

Quantitative Research: Important issues pertaining to research methodology &

analysis strategy

Helene Muller

1

Quantitative Research: Important issues pertaining to research methodology & analysis strategy

Outline

Quantitative research

– Where does Quantitative Research slot in in the bigger Research picture?

– The survey research design in quantitative research

– Questionnaire design

– Reliability and validity

Analysis strategy: options explained by means of examples

– Example 1 (1 variable, categorical)

– Example 2 (2 variables, categorical)

– Example 3 (2 variables, continuous, different measures)

– Example 4 (2 variables continuous, same measure)

– Example 5 ( 2variables, continuous and categorical)

2


Concepts in Quantitative Research

3

Concept 1: Where does Quantitative Research slot in within the Research Process?

1. When conducting research, the research approach can be either: – A qualitative (collected data: words (interviews), pictures (video)or objects (artefacts) – or a quantitative ( data collected: numerical data) – or a mixed approach. The research approach followed determines the Research Design

2. Research Designs A quantitative approach is appropriate when the research purpose is to test predictive and/ or

cause-effect type of hypotheses. Deductive logic is used and the design is suitable for phenomena that have been well developed with regard to theory and concept

Qualitative designs on the other is applicable when the research purpose is to accumulate knowledge of a (social) phenomenon that is relatively unknown and seeks understanding. Reasoning in this instance is more inductive

A mixed research design often strengthen research in the sense that the abovementioned approaches can be used to validate and enhance the research findings of each approach. (De Vos, et al 2005)


Concept in Quantitative Research (Cont.)

Concept 2: The Survey Research Design in Quantitative Research

Most of the quantitative research in Educational studies adopts a survey design type.

This presentation will focus on the survey research design

Why is the Survey design appropriate? These design type has certain properties

• In quantitative research a ‘tool/ or measuring instrument’ is required to evaluate the phenomenon under investigation • The ‘tool-of-measurement’ in survey designs is the questionnaire (standardized or designed) • The design type generates/ collects numeric data • The information collected is objective and reliable (if properly structured, administered and validated) • Reliable findings because respondent selection & analysis processes adhere to statistical principles • The questionnaire is purpose-designed according to structured theoretical knowledge and research questions • Collects only essential information

4


Concept in Quantitative research (cont)

Concept 3: Questionnaire design

**Three crucial principles to remember/ consider when designing a questionnaire

1. The format/type of questions determine the type of data generated – Closed ended questions (A list of possible options to choose from)

o Single choice (‘male’ / ‘female’) Categorical data, nominal o Multiple choice response (“all hobbies of interest listed”) Categorical o Rating levels (‘agree’ / ‘undecided / ’disagree’) Categorical, ordinal o Exact figures (weight, age) Continuous

– Open ended (Written text response) Categorical data

2. The type of data generated dictates the type of statistical analyses that can be performed

3. The analysis techniques applied determines the type of research questions (hypotheses) that can be addressed and answered

5


Concepts in Quantitative research (cont)

Concept 4: Reliability and validity in research

Quantitative research strives to present valid and reliable research finding.

• Reliability refers to the consistency of a measure. A questionnaire/ or test is considered reliable if the same result is obtained repeatedly when the questionnaire is re-administered/ or tested repeated (Thermometer reading)

• Validity refers to the extent to which a questionnaire / or test measures what it purports to measure (Thermometer does not measure blood pressure)

There are various types of validity and reliability involved in the various stages of the quantitative research process. Measures have been developed to ensure valid & reliable research findings. That is why quantitative research adhere to

– Protocol for questionnaire design

– Protocol while administering a questionnaire

– Measures to ensure data integrity

– Measures to ensure validity and reliability (appropriate analysis strategy)

6


Analysis strategy

• Once a sample for the research has been selected (according to a selection protocol/ sampling method) and questionnaires administered, the responses of the respondents is electronically captured as data(examples to follow)

• The next step is the research process is then the analysis strategy. The analysis strategy refers to the type of statistical analysis techniques the researcher plans to use in analyzing the data

• The analysis strategy, along with questionnaire design, determines the type of research questions and hypotheses that the researcher will be able to answer from the analysis results.

• The importance of the previous statement is best illustrated by means of example. The rest of the presentation discusses analysis techniques and their applicability to answering specific types of research questions

7

Questionnaire: Reading methods

Section A

1. Please indicate your age

2. Please indicate your gender

3. Years of experience teaching junior level primary school

Reading methods questionnaire 2

4 – 8: Other biographical attributes

Section B

9. Please tick the reading approaches/ methods that you use in your classroom

Method √

1. Phonics

2. Look-&-say

3. Eclectic

4. Language expression

5. THRASS

6. Readers

7. Other books in class

1. >30

2. 31-40

3. 41-50

4. 51-65

1. male 2. Female

1. <11 yrs 2. 11-20 yrs 3. 21+yrs

8


Example questionnaire


10. Please provide the class average for the following first term evaluations

11. Please provide class average of the second term reading evaluation before & after introduction of Look & Say reading approach


12. Please indicate your level of agreement , with the following statements, where 1: agree++ 2: agree 3: undecided 4: disagree 5: disagree++

Class average, Reading ………

Class average, Spelling ……….

Class average Before ……….

Class average After ……….

Agreement statements 1 2 3 4 5

1. Reading skills are important

2. Reading skills form the basis of the learning process

3. Sound reading habits should be fostered early

4 . Bed time stories contribute towards love of reading

5. Parents’ reading habits determine their children‘s

6. One can never read enough

9


Example questionnaire (continued)


Example: Questionnaire, electronically captured data

Electronically captured data from Reading Methods Research Questionnaire

no

Age

Gender

Exp

Q9.1

Q9.1

Q9.3

.

.

.

.

Q9.7

R e a d

Write

Before

After

q1

q2

q3

q4

q5

q6

1 <30 M 1 1 1 1 . 0 43 55 43 55 2 3 1 2 2 1

2 51-56 F 3 1 1 0 . 0 56 77 56 77 3 1 2 2 3 1

3 31-40 M 3 0 1 0 . 1 78 79 76 79 1 1 3 2 2 1

4 31-40 M 2 0 1 1 . 0 54 56 78 71 4 4 5 3 3 3

5 <30 F 3 1 1 1 . 1 65 79 56 67 1 2 2 1 1 2

6 41-50 F 1 1 0 1 .. 0 66 72 65 78 2 1 1 1 1 3

7 41-50 F 1 1 1 0 .. 0 79 84 78 82 2 2 1 3 1 1

… … … … … … … … …

...

… … … … … … … … …

10

• Type of data recorded and analysis techniques applicable:

• Type of research questions that can be answered with the data collected from the questionnaire

Categorical data on biographical properties (age, gender, exp.)

Therefore limited analyses & non-parametric tests, such as • One-way frequency tables & graphs (bar graphs, spider plots, pie charts) •Descriptive techniques

Research questions pertaining to a description of the sample (the respondents selected) in terms of their biographical attributes can be answered:

“Fifty seven percent of the respondents in the sample were younger than 35” ….. Gender ………. Experience ……….

11


Example 1: Section A, Analysis strategy

Analysis technique: One way frequency table

Table 1 Frequency distribution of student respondents

in Reading methods research project

Age category

Frequency Percentage Total

<30 27 36.0 27

30-35 16 21.3 43

36-40 15 20.0 58

41-45 11 14.7 69

46+ 6 8.0 75

0

5

10

15

20

25

30

Age distribution

Age distribution

12


Example 1: Analysis strategy, Section A (Age)



Categorical, multiple responses (‘select’ (0) or ‘not selected’ (1))

Non-parametric classification techniques (one-way and two-way ), such as • One-way and two-way frequency tables & graphs (bar graphs) • Chi-square tests to establish dependencies are appropriate techniques

Research questions pertaining to • a general overview of a specific trend in the research

“Phonics & Readers are the two most commonly used reading methods”

• dependency between two classification attributes (yes/no & reading

method) “ The frequency-of-use(yes/ no) and reading method is dependent: implying frequency of

use of Reading methods differ significantly”

13


Example 2: Analysis strategy Section B, q9

Method Yes, use No Total

Phonics 74 2 76

Look&Say 48 27 75

Eclectic 43 32 75

L _Express 31 44 75

THRASS 11 64 75

Readers 69 4 73

O_ Books 57 17 74

Probability of (Chi-square statistic = 168.40) < 0.0001 ***

74

48

43

31

11

69

57

2

27

32

44

64

4

17

Utilization of reading methods

Yes No

14


Example 2: Analysis strategy, Section B q9 .



Continuous data: two sets of attribute measures (read & spell tests)

Parametric analysis techniques, such as • Linear regression analysis & graphs (linear regression line and scatter plot) • F & t tests to establish significance of linear trend describing the relationship between the two measures

Research questions pertaining to a more detailed description of the relationship between two measures on the same respondents :

•“A statistically significant relationship between reading & spelling averages of educator respondents exist” • “ The relationship can be describe by the following regression line: Read avg = 51.1 + 1.33*(Spell avg.)”

15


Example 4: Section B q11, Analysis strategy

30

40

50

60

70

80

90

100

30 50 70 90 110

Y-a

xis:

Tes

t 2

Regression Test 1 against Test 2 X-axis test 1

Test 2

Linear (Test 2)

Respondent no

Read Test1

Spell Test2

1 43 55

2 56 77

3 78 79

4 54 56

5 65 79

6 66 72

7 79 84

… … …

16


Example 3: Analysis strategy, Section B, q10



Continuous data: one attribute two ‘paired measures’ (‘before’ & ‘after’ )

Parametric & non-parametric analysis techniques, such as • Paired difference test Students’ t-test (parametric) or Signed Rank test (non-parametric)

Research questions pertaining a different kind of relationship between two measures of the same attribute – ‘before’ and after’ measures:

• “Does a statistically significant relationship exist between the two measures of the same

attribute? (reading skills)” • “Did the intervention treatment implemented between the before & after evaluation improve reading skills?’

17


Example 4: Analysis strategy, Section B q10

Respond no

Before After Diff(After-Before)

1 43 55 12

2 56 77 21

3 76 79 3

4 78 71 -5

5 56 67 11

6 65 78 13

7 78 82 4

….. ….. ….. …..

Parametric and non-parametric difference test results

Students’ t-test: Probability of (t-statistic = 2.64) = 0.039 *

Signed Rank test Probability of (S-statistic=11) = 0.078

Significance legend: * : 5% level of significance ** : 1% level of significance ***: 0.1% level of significance

18


Example 4: Analysis strategy, Section B q10



Continuous & Categorical data (attitude & age)

Various parametric and non-parametric analysis techniques are applicable • Calculation attitude scales, verification of internal consistency reliability • Analysis of variance on attitude scale values to evaluate effect of biographical attributer on attitudes • Bonferroni multiple comparison of means test to describe nature of biographical effect

Research questions pertaining to the general attitude-trend exhibited by respondents and the effect of biographical attributes on their reading attitude • “ Do respondents exhibit a positive attitude towards reading?” • “ Does age significantly affect respondent’s attitude towards reading? “ • “ Do younger people exhibit a more negative attitude than older people?”

19


Example 5: Analysis strategy, section B q12

Data for q 12

Respond no

Age q1 q2 q3 q4 q5 q6

1 <30 2 3 1 2 2 1

2 51-56 3 1 2 2 3 1

3 31-40 1 1 3 2 2 1

4 31-40 4 4 5 3 3 3

5 <30 1 2 2 1 1 2

6 41-50 2 1 1 1 1 3

7 41-50 2 2 1 3 1 1

….. …..

• Scale reliability testing results

• Calculate attitude score for each respondent .

Example 1st respondent:

score₁ = (2+3+1+2+2+1)/5 = 2.2

• Analysis of variance results

Scale reliability test

Construct Items

included Cronbach

alpha Mean score

Std dev

Read attitude

q12.1-q12.6

0.79 1.23 0.23

Summary Analysis of variance & Bonferroni multiple comparisons of means test results

Construct F statistic (F prob)

Age Read att. Score

Mean

Read score 171.23

(0.0001) *** 0.011**

<30: 4.21 a 31-40: 2.98 a 41+: 1.44 b

20


Example 5: Analysis strategy, section B q12


Conclusion

• The importance of – properly planned quantitative research,

– carefully formulated research questions

– cleverly designed questionnaire according to the research questions/ hypotheses stated cannot be over-emphasized

• Enjoy your studies!

21

• http://psychology.about.com/od/researchmethods/f/reliabilitydef.htm

• Wikipedia

• De Vos, Strydom, Fouche, Delport

22

http://psychology.about.com/od/researchmethods/f/reliabilitydef.htm



Documents

Quantitative Research: Important issues pertaining to …osprey.unisa.ac.za/TechnicalReports/e4.pdf · Quantitative Research: Important issues pertaining to research methodology &