GRU6014

SRU 6024KAEDAH PENYELIDIKAN

KUANTITATIF2015

ASSIGNMENT 4

WRITING REPORT

TOPIC:

QUESTIONNAIRE DESIGN AND DATACOLLECTION METHOD

GROUP MEMBERS

BIL NAME MATRIC NUM

1. NORLAILATULAKMA BOLHASSAN M20142002147

2. NORSHIMA BINTI HAMADAN M20141000263

3. EDWIN ROY A/L MOSES M20141000841

SUBMIT TO:

DR. HAFSAH BINTI TAHA

CONTENTS

Matter Pages

1.0 Introduction 1

2.0 Questionnaire Design

2.1 Introduction To Questionnaire 1

2.2 Selecting, Designing And Developing Questionnaires 2

2.21 To Collect Information 2

2.22 An Appropriate Questionnaire 2

2.23 Use An Existing Instrument 3

2.24 Valid And Reliable Questionnaire 3

2.25 Questionnaire’s Form 4

2.3 Types Of Questions 5

2.31 Closed-Ended Questions 5

2.32 Open-Ended Questions 6

2.4 Types Of Measurement Scales 7

2.41 Nominal Scales. 7

2.42 Ordinal Scales. 7

2.43 Interval And Ratio Scales. 8

2.5 To Obtain Valid Information 9

2.6 To Evaluate Research Questionnaire 14

2.61 Example of a bad questionnaire and how to improve 15

2.7 Conclusion 19

3.0 Data Collection Methods 19

3.1 Introduction To Data Collection Methods 19

3.11 Example of data collection research design 21

3.2 Data-collection instrument 22

3.21 Researcher completed document 24

3.211 Rating Scales 24

3.212 Interview Schedules 26

3.213 Direct observation 27

3.214 Tally Sheets 27

3.215 Performance checklist 28

3.22 Subject completed instruments 28

3.221 Self-checklist 28

3.222 Attitude Scales 28

3.2221 Likert Scale 29

3.2222 Bogardus Social Distance Scale 30

3.2223 Thurstone Scale 30

3.2224 Semantic Differential Scale 31

3.23 Performance Test 31

3.24 Analysis of document 32

4.0 Triangulation 32

4.1 Control for Researcher Errors and Biases 32

4.11 Data triangulation 33

4.12 Investigator triangulation 34

4.13 Theory triangulation 34

4.14 Methodological triangulation 35

4.15 Environmental triangulation 36

5.0 Conclusion & Summary 37

References 38

1.0 INTRODUCTION

The primary goal of this report is to help educators, students, and we ourselves learn the many

ways in research that can enable us to think methodically about problems of practice and work

toward our solutions. To achieve this goal, this report focuses on helping us to develop two skills.

The first skill is the ability to design questionnaire for our research and other publications

relevant to specific problems of practice. The second skill is the ability to understand how to

collect data for research.

This report is a major revision of the questionnaire and survey data collection. In this topic,

places much more emphasis on questionnaire design and data collection methods. By this report,

we hope that we could help some educators and students face in their workplace and on how they

can use the findings of their own or others' research to solve them.

2.0 QUESTIONNAIRE DESIGN

2.1 Introduction to Questionnaire

A questionnaire is a set of questions in paper-and-pencil or computer format that typically

measures many variables (M.D Gall, 2013). Questionnaires offer an objective means of collecting

information about people's knowledge, beliefs, attitudes, and behaviour (Boynton & Greenhalgh,

2004). For examples:

“Do our patients like our opening hours?”

“What do teenagers think of a local antidrugs campaign and has it changed their attitudes?”

“Why don't doctors use computers to their maximum potential?”

Questionnaires can be used as the sole research instrument such as in a cross sectional survey or

within clinical trials or epidemiological studies. For another example, a questionnaire might ask

respondents about their interest towards chemistry subject, what the factors that they like in the

chemistry subject are, how chemistry teachers influence their enjoyment in the subject, and what

their achievements in the subject are. The response to each question constitutes a separate

variable in the research study.

1

2.2 Selecting, Designing and Developing Questionnaires

2.21 To collect information

Researchers may have different assumptions about precisely what information they would like

their study to generate. A formal scoping exercise will ensure that they clarify goals and if

necessary reach an agreed compromise. It will also flag up potential practical problems. For

example, how long the questionnaire will be and how it might be administered.

2.22 An appropriate questionnaire

People often decide to use a questionnaire for research questions that need a different method.

Sometimes, a questionnaire will be appropriate only if used within a quantitative or mixed

methodology study. Research participants must be able to give meaningful answers (with help

from a professional interviewer if necessary).

Examples where questionnaires were used inappropriately:

Broad area of research:

Professional behaviour

Example of research question:

How do general practitioners manage low back pain?

Why is a questionnaire not the most appropriate method?

What doctors’ say they do is not the same as what they actually do, especially when they

think their practice is being judged by others.

What methods should be used instead?

Direct observation or video recording of consultations; use of simulated patients;

systematic analysis of medical records

2

2.23 Use an existing instrument

Using a previously validated and published questionnaire will save researchers’ time and

resources. They will be able to compare their own findings with those from other studies.

Researchers need only give outline details of the instrument when they write up their work, and

may find it easier to get published. If there is no questionnaire available, researchers will have to

construct their own. Using one or more standard instruments alongside a short bespoke

questionnaire could save researchers the need to develop and validate a long list of new items.

2.24 Valid and Reliable questionnaire

A valid questionnaire measures what it claims to measure. In reality, many fail to do this. For

example, a self-completion questionnaire that seeks to measure people's food intake may be

invalid because it measures what they say they have eaten, not what they have actually eaten.

Similarly, responses on questionnaires that ask general practitioners how they manage particular

clinical conditions differ significantly from actual clinical practice (Boynton & Greenhalgh,

2004)

An instrument developed in a different time, country, or cultural context may not be a valid

measure in the group that researchers are studying.

Example 1:

The item “I often attend gay parties” may have been a valid measure of a person's sociability

level in the 1950s, but the wording has a very different connotation today.

Reliable questionnaires yield consistent results from repeated samples and different

researchers over time. Differences in results come from differences between participants, not

from inconsistencies in how the items are understood or how different observers interpret the

responses. A standardized questionnaire is one that is written and administered so all participants

are asked the precisely the same questions in an identical format and responses recorded in a

uniform manner. Standardizing a measure increases its reliability.

Just because a questionnaire has been piloted on a few of researchers’ colleagues, used in

previous studies, or published in a peer reviewed journal does not mean it is either valid or

reliable.

3

2.25 Questionnaire’s form

The response format should be standardize and maximize by providing clear information and

instructions:

1. State who you are, outline purpose of survey

2. Keep confidentiality and voluntary participation

3. Provide clear instructions as to how each questions should be answer

4. How to return back the questionnaire and by what date

Example:

UNIVERSITI PENDIDIKAN SULTAN IDRIS

FAKULTI SAINS DAN MATEMATIK

BORANG SOAL SELIDIK PERSEPSI MURID TINGKATAN

EMPAT TERHADAP GAYA PENGAJARAN GURU KIMIA

Jutaan terima kasih diucapkan kerana kesudian menjadi responden bagi kajian ini. Borang soal

selidik ini dikemukakan untuk menilai persepsi murid tingkatan empat terhadap gaya pengajaran

guru kimia .

Maklumat yang diperoleh daripada kajian ini adalah sulit dan hasil kajian ini hanya

digunakan untuk laporan kajian saya sahaja. Diharap kerjasama yang diberikan oleh pihak ANDA

dapat membantu saya mendapatkan maklumat yang tepat mengenai persepsi anda terhadap gaya

pengajaran guru kimia.

Terima kasih.

Yang benar,

……………………………………...

Norlailatulakma Bolhassan

Fakulti Sains dan Matematik,

4

2

1

1

Bagi setiap item, sila bulatkan untuk menyatakan pendapat anda mengikut skala yang diberikan.

Bil. Perkara

San

gat

tid

ak

Set

uju

Tid

ak S

etu

ju

Set

uju

San

gat

1. Guru sentiasa berpakaian kemas ke sekolah. 1 2 3 4

2.3 Types of Questions

The nature of the questions and the way they are asked are extremely important in survey

research. Poorly worded questions can doom a survey to failure. Hence, they must be clearly

written in a manner that is easily understandable by the respondents. There are two of questions’

types, which are close-ended questions and open-ended questions. (Frankell & Wallen, 2009)

2.31 Closed-ended questions

Most surveys rely on multiple-choice or other forms of what are called closed-ended questions.

Multiple-choice questions allow a respondent to select his or her answer from a number of

options. They may be used to measure opinions, attitudes, or knowledge.

Example 1:

Rate each of the following parts of your master's degree program by circling the number under

the phrase that describes how you feel.

Very Dissastified Dissastified Satisfied Very Satisfied

Coursework 1 2 3 4

Professors 1 2 3 4

Advising 1 2 3 4

Requirements 1 2 3 4

Cost 1 2 3 4

Other 1 2 3 4

5

3

Closed-ended questions are easy to use, score, and code for analysis on a computer.

Standardized data are provided due to all subjects respond to the same options. However they are

more difficult to write than open-ended questions. They also pose the possibility that an

individual's true response is not present among the options given. For this reason, the researcher

usually should provide an "other" choice for each item, where the subject can write in a response

that the researcher may not have anticipated. Some examples of closed-ended questions are the

following:

Example 2:

Which subject do you like least?

a. Chemistry

b. Biology

c. Physic

d. Mathematics

e. Other (specify)

2.32 Open-Ended Questions

Open-ended questions allow for more individualized responses, but they are sometimes difficult

to interpret. They are also often hard to score, since so many different kinds of responses are

received. Furthermore, respondents sometimes do not like them. Some examples of open-ended

questions are as follows:

Examples:

1. What characteristics of a person would lead you to rate him or her as a good

administrator?

2. What do you consider to be the most important problem facing classroom teachers in high

schools today?

3. What were the three things about this class you found most useful during the past

semester?

6

2.4 Types of Measurement Scales

A scale is a set of numbers that represents the range of values of a variable. For example,

currency in the United States is measured by a scale that represents money as a set of numbers

that increase in value by one-penny intervals. In the following sections, we describe four types of

scales used in educational research. Each type yields data that are amenable to different kinds of

statistical analysis.

2.41 Nominal Scales.

A nominal scale is a set of numbers that represent a variable whose values are categories that

have the properties of being mutually exclusive and not orderable.

Example 1:

Marital status is a good example of a nominal scale. An individual can be married or not

married, but not both. We can assign numbers to married and nonmarried individuals, such

as 1 to represent married individuals and 2 to represent nonmarried individuals. This

numbering is arbitrary but useful for entering information about a research sample into a

database for statistical analysis.

Nominal scales sometimes are called categorical scales because each number on the scale

represents a different category. In a research study involving a variable that is measured on a

nominal scale, researchers typically will report a frequency count of the number of individuals or

objects that are members of each category.

Example 2:

Researchers might be asked to determine the number of teachers in each of the 50 states

and commonwealths. Each state and commonwealth is a separate category.

2.42 Ordinal Scales.

An ordinal scale is a set of numbers that represent a variable whose different values can be placed

in order of magnitude, but the difference between any two sets of adjacent values might differ.

The values often are called ranks (or rankings). We commonly see ordinal scales in sports.

7

Example 1:

In a golf tournament, the person with the best score occupies first place, the next best scorer

occupies second place, and so forth. Each "place" represents a different rank.

An important limitation of ordinal scales is that they yield numerical data in the form of ranks,

but the ranks do not necessarily represent equal differences from one rank to the next.

Example 2:

Suppose the top-ranked student in a class has a GPA of 3.98. The second-ranked student

might have a CPA of 3.97 or 3.85 or 3.62. In other words, the second-place ranking only

indicates that the student has the next-lower GPA than the student with the first-place

ranking, but it does not indicate how much lower the GPA is.

Despite this limitation, the rankings produced by ordinal scales often are meaningful and

important to various constituencies.

Example 3:

Scholarships might be awarded to the top five students in a class, irrespective of how much

or how little they differ on the criteria used to determine the rankings. In selecting

candidates for a job, the search committee

2.43 Interval and Ratio Scales.

An interval scale is a set of numbers that represent a variable whose different values can be

placed in order of magnitude, with an equal interval between any two adjacent values. An easy

way to understand this type of scale is to think of an ordinary ruler. The distance between the 1-

inch point and the 2-inch point is equal to the distance between the 5-inch point and the 6-inch

point; the distance between the 17.4-inch point and the 17.8-inch point is equal to the distance

between the 11.4-inch point and the 11.8-inch point, and so forth.

8

2.5 To Obtain Valid Information

In this section, we indicate the characteristics on how to obtain valid information. There are some

of considerations that a researcher should be careful:

Ask purposeful questions

Ask concrete questions

Use time periods based on importance of the questions

Use conventional language

Use complete sentences

Avoid abbreviations

Review questions with experts and potential respondents

Use shorter questions

Avoid two-edged questions

Avoid negative questions

Adopt/adapt questions used successfully in other questionnaires

According Brace (2008), creating surveys that yield actionable insights is about details. And

writing effective questions is the first step. There are common mistakes that keep survey

questions from being effective all the time. Here are the 7 most common:

1. Failing to avoid leading words or questions

Subtle wording differences can produce great differences in results. “Could,” “should,” and

“might” all sound about the same, but may produce a 20% difference in agreement to a question.

In additions, strong words such as “force” and “prohibit” represent control or action and can bias

your results.

Example 1:

Mistake:

‘The government should force you to pay higher taxes.’

What is the mistake about above sentence is, no one likes to be forced, and no one likes higher

taxes. This agreement scale question makes it sound doubly bad to raise taxes.

9

Improvement:

Wording alternatives can be developed. How about simple statements such as:

‘The government should increase taxes’ or

‘The government needs to increase taxes’

Example 2:

Mistake:

‘How would you rate the career of legendary player Lee Chong Wei?’

The mistake about this question tells you Lee Chong Wei is a legendary player. This type of

wording can bias respondents.

Improvement:

To overcome it, how about replacing the word “legendary” with “badminton” as in:

‘How would you rate the career of badminton player Lee Chong Wei?’

2. Failing to give mutually exclusive choices

Multiple choice response options should be mutually exclusive so that respondents can make

clear choices. We should not create ambiguity for respondents. Our survey has to review and the

ways respondents could get stuck with either too many or no correct answers has to identify.

Example 1:

What is your age?

o 0 – 10

o 10 – 20

o 20 – 30

o 30 – 40

o 40 +

If the respondent were 10, 20, or 30 years old, they cannot choose any of the answers above.

Questions like this will frustrate a respondent and invalidate our results.

10

Example 2:

What type of vehicle do you own?

o Van

o SUV

o Hilux

This question has the same problem. The respondent cannot choose any answers if they owns a

truck, hybrid, convertible, cross-over, motorcycle, or no vehicle at all?

3. Not asking direct questions

Questions that are vague and do not communicate our intent can limit the usefulness of our

results. We have to ensure the respondents know what we are asking.

Example 1:

‘What suggestions do you have for improving Limah’s Tomato Juice?’

This question may be intended to obtain suggestions about improving taste, but respondents will

offer suggestions about texture, the type of can or bottle, about mixing juices, or even suggestions

relating to using tomato juice as a mixer or in recipes.

Example 2:

‘What do you like to do for fun?’

Finding out that respondents like to play Scrabble isn’t what the researcher is looking for, but it

may be the response received. It is unclear that the researcher is asking about movies or other

forms of paid entertainment. A respondent could take this question in many directions.

4. Forgetting to add “prefer not to answer” option

Sometimes respondents may not want or be able to provide the information requested. Questions

about income, occupation, finances, family life, personal hygiene, and personal, political, or

religious beliefs can be too intrusive and be rejected by the respondent. Privacy is an important

11

issue to most people. Incentives and assurances of confidentiality can make it easier to obtain

private information.

While current research does not support that PNA (Prefer Not to Answer) options increase

data quality or response rates, many respondents appreciate this non-disclosure option.

Furthermore, different cultural groups may respond differently. One recent study found that while

U.S. respondents skip sensitive questions, Asian respondents often discontinue the survey

entirely.

Examples:

What is your race?

What is your age?

Did you vote in the last election?

What are your religious beliefs?

What are your political beliefs?

What is your annual household income?

These questions should be asked only when absolutely necessary. In addition, they should always

include an option to not answer. (e.g. “Prefer Not to Answer”).

5. Failing to cover all possible answer choices

We must cover all the options in questionnaire. If we are unsure, a pretest has to be conducting

using “Other (have to specify)” as an option. If more than 10% of respondents in a pretest or

otherwise select “other”, we are probably missing an answer. The “Other” text our test

respondents have provided should be review and the most frequently mentioned new options has

to be add to the list.

Example:

‘You indicated that you eat at KFC’s fast food once every 3 months. Why don’t you eat at

KFC’s more often?’

o There isn’t a location near my house

o I don’t like the taste of the food

o Never heard of it

12

This question does not include other options, such as healthiness of the food, price or some

“other” reason. Over 10% of respondents would probably have a problem answering this

question.

6. Not using unbalanced scale carefully

Unbalanced scales may be appropriate for some situations and promote bias in others. For

instance, a hospital might use an Excellent – Very Good – Good – Fair scale where “Fair” is the

lowest customer satisfaction point because they believe “Fair” is absolutely unacceptable and

requires correction.

The key is to correctly interpret the scale. If “Fair” is the lowest point on a scale, then a

result slightly better than fair is probably not a good one. Additionally, scale points should

represent equi-distant points on a scale. That is, they should have the same equal conceptual

distance from one point to the next.

For example, researchers have shown the points to be nearly equi-distant on the strongly

disagree–disagree–neutral–agree–strongly agree scale. Bottom point is set as the worst possible

situation and top point as the best possible, then evenly spread the labels for scale point’s in-

between.

Example:

What is your opinion of Perodua’s auto-repair?

o Pretty good

o Great

o Fantastic

o Incredible

o The Best Ever

This question puts the center of the scale at fantastic, and the lowest possible rating as “Pretty

Good.” This question is not capable of collecting true opinions of respondents.

13

7. Not asking only one question at a time/ double barreled

There is often a temptation to ask multiple questions at once. This can cause problems for

respondents and influence their responses. These form of question known as double barreled

question. We should review each question and make sure it asks only one clear question.

Example 1:

‘What is the fastest and most economical Internet service for you?’

This is really asking two questions. The fastest is often not the most economical.

Example 2:

‘How likely are you to go out for dinner and a movie this weekend?’

Even though “dinner and a movie” is a common term, this is two questions as well. It is best to

separate activities into different questions or give respondents these options:

o Dinner and Movie

o Dinner Only

o Movie Only

o Neither

2.6 To Evaluate Research Questionnaire

According to M. D. Gall (2010), in evaluating a research questionnaire, we must consider about

the following questions:

1. Was the questionnaire pretested?

A research participant might interpret a questionnaire item differently than intended by the

researcher. Therefore, the researcher should pilot, that is, try out, the questionnaire and analyze

the responses of a small sample of individuals before starting the main study. Results of this pilot

study should be used to refine the questionnaire. If a pilot study has been done, we can have more

confidence that the findings reported in the main study are valid.

14

2. Does the questionnaire include leading questions?

A copy of the questionnaire sometimes is included in the research report. For leading questions,

which are questions framed in such a way that individuals are given hints about the kind of

response that is expected. Results obtained from leading questions are likely to be biased, so they

should be interpreted with caution.

3. Does the questionnaire include psychologically threatening questions?

The researcher should avoid questionnaire items that might be psychologically threatening to the

respondents. For example, a questionnaire sent to school principals concerning the morale of their

teachers would be threatening to some principals, because low morale suggests that they are

failing in part of their job. If they feel threatened, the principals might not complete and return the

questionnaire. If they do return it, little confidence can be placed in the accuracy of their

responses because of their ego involvement in the situation.

4. Do the individuals who received the questionnaire have the requested information?

Researchers inadvertently might send a questionnaire to a sample that does not have the desired

information. If this happens, the sample will provide inaccurate information simply not complete

the questionnaire.

2.61 Example of a bad questionnaire and how to improve

BORANG SOAL SELIDIK

Borang soal selidik ini adalah bertujuan untuk mendapatkan maklum balas mengenai

PENGGUNAAN ICT DALAM PROSES PENGAJARAN DAN PEMBELAJARAN

DIKALANGAN GURU-GURU SAINS SEKOLAH MENENGAH

Soal selidik ini mengandungi empat (4) muka surat yang terbahagi kepada dua (2) bahagian:

Bahagian A dan Bahagian B.

Bahagian A : Latar belakang guru.

Bahagian B : Set soal selidik mengenai bentuk penggunaan ICT

15

Guru dikehendaki menjawab SEMUA pernyataan dalam setiap bahagian, sebagaimana yang

diarahkan. Segala maklumat yang anda berikan akan diRAHSIAkan. Sila berikan jawapan anda

dengan ikhlas.

Kerjasama anda didahului dengan ucapan terima kasih

ARAHAN

Untuk setiap pernyataan di bawah, anda dikehendaki memilih salah satu daripada lima (5) aras persetujuan. Sila ( / ) pada pilihan anda.

PANDUAN

1 = Sangat Tidak Setuju (STS)2 = Tidak Setuju (TS)3 = Tidak Pasti (TP)4 = Setuju (S)5 = Sangat Setuju (SS)

BAHAGIAN B

BENTUK PENGGUNAAN ICT

No. Item STS1

TS2

KP3

S4

SS5

1 Saya kerap melayari internet untuk mencari sumber rujukan

2 Saya sering mengintegrasikan pelbagaipenggunaan bahan multimedia (powerpoint, LCD projektor) dalam P&P

3 Saya menggalakkan guru menggunakan perkhidmatan internet untuk mendapatkan rujukan-rujukan pembelajaran

4 Saya tahu menggunakan powerpoint secara interaktif

5 Saya suka menggunakan email

6 Portal sekolah membantu saya memuat naik semua nota dan dokumen yang diperlukan oleh guru lain.

7 Saya percaya dengan menyertai ‘newsgroup’/ ‘group’ seperti slideshare di internet adalah baik untuk berkongsi maklumat atau bahan –bahan P&P

8 Saya yakin, pembelajaran secara’networking’ dapat menjimatkan masa

9 Saya biasa menggunakan grafik atau animasi komputer semasa mengajar untuk menerangkan masalah-masalah tertentu

10 Saya menggunakan power point semasa sesi P&P

11 Saya menggunakan CD Coursework KPM dalam P&P

12 Masa dan sumber yang terhad menyukarkan saya untuk merekabentuk bahan pembelajaran berasaskan komputer

13 Sambungan internet sering mengalami gangguan dan tiada penyelenggaraan

16

Statement 1:

Based on the instruction in questionnaire above, the mistake that the researcher has been done is:

“Guru dikehendaki menjawab SEMUA pernyataan dalam setiap bahagian, sebagaimana

yang diarahkan.”

The wording indicates the respondents were asked forcedly to answer. So, to improve the

wording is:

“Guru diminta menjawab semua pernyataan dalam setiap bahagian seperti yang

ditunjukkan dalam panduan.”

Statement 2:

Based on the second mistake:

“ARAHAN”

Untuk setiap pernyataan di bawah, anda dikehendaki memilih salah satu daripada lima (5) aras persetujuan. Sila ( / ) pada pilihan anda.

The statement can be improve by:

”PANDUAN”

Untuk setiap pernyataan di bawah, anda diminta memilih salah satu daripada lima (5) aras persetujuan. Sila ( / ) pada pilihan anda.

Also additional improvement can be put there:

Bagi setiap item, sila bulatkan untuk menyatakan pendapat anda mengikut skala yang diberikan.

Bil. Perkara

San

gat

tid

ak

Set

uju

Tid

ak S

etu

ju

Set

uju

San

gat

1. Guru sentiasa berpakaian kemas ke sekolah. 1 2 3 4

17

3

Statement 3

Questions 1: “Saya kerap melayari internet untuk mencari sumber rujukan”

The mistake here is the word of “kerap”. We do not know and cannot identify the understanding

of each respondents about the frequently they surf internet to find the resources. Some of

individuals might think that 2 or 3 times in a week, can be too frequent. But another would might

think the frequently is every day to surf internet. So we cannot identify the word of “kerap”

should be how many times to surf internet.

The questions appropriate place in the part A, which is demography part. To improve the

question is:

“Saya melayari internet untuk mencari sumber rujukan”

a. 1 – 2 kali seminggu

b. 3 – 4 kali seminggu

c. 5 – 6 kali seminggu

d. Tidak pernah

Statement 4:

Question 2: Saya sering mengintegrasikan pelbagai penggunaan bahan multimedia (powerpoint,

LCD projektor) dalam P&P

The word of “sering” also inappropriate to be used in the question. It could be improve by

eliminate the word of “sering”.

“Saya mengintegrasikan pelbagai penggunaan bahan multimedia (powerpoint, LCD

projektor) dalam P&P”

Statement 5:

Question 7: “Saya percaya dengan menyertai ‘newsgroup’/ ‘group’ seperti slideshare di internet

adalah baik untuk berkongsi maklumat atau bahan –bahan P&P”

The mistakes in the question are the wording is too long, contain double barrel, and the major is

ambiguous which is the intention of the question is not clear. So, the question can be removed

out.

18

Statement 6:

There are too many questions that contain double-barrel in the questionnaire.

Question 9:

“Saya biasa menggunakan grafik atau animasi komputer semasa mengajar untuk

menerangkan masalah-masalah tertentu.”

Question 12:

“Masa dan sumber yang terhad menyukarkan saya untuk merekabentuk bahan

pembelajaran berasaskan komputer.”

Question 13:

“Sambungan internet sering mengalami gangguan dan tiada penyelenggaraan.”

So, to improve the questions above, we should eliminate the words of “atau” and “dan”

2.7 Conclusion

As we have reviewed above, designing a questionnaire study that produces usable data is not as

easy as it might seem. Awareness of the pitfalls is essential both when planning research and

appraising published studies.

3.0 DATA COLLECTION METHODS

3.1 Introduction to data collection methods

Whatever the research design may be, the empirical data will be collecting. In fact, the collection

of empirical data to answer questions or test hypotheses is the very essence of research. Based on

M. D Gall (2010), empirical data are direct observations of the phenomena that researchers are

studying. By contrast, beliefs, ideas, and theories are claims about what researchers would find if

they collected empirical data.

Generally, the whole process of preparing to collect data is called instrumentation. It

involves not only the selection or design of the instruments but also the procedures and the

conditions under which the instruments will be administered. Several questions arise:

19

1. Where will the data be collected? This question refers to the location of the data

collection. For examples, in a classroom, in a schoolyard, in a private home, or on the

street.

2. When will the data be collected? This question refers to the time of collection. When is it

to take place? In the morning? Afternoon? Evening? Over a weekend?

3. How often are the data to be collected? This question refers to the frequency of collection.

How many times are the data to be collected? Only once? Twice? More than twice?

4. Who is to collect the data? This question refers to the administration of the instruments.

Who is to do this? The researcher? Someone selected and trained by the researcher?

These questions are important because how researchers answer them may affect the data

obtained. The data provided by any instrument may be affected by any or all of the preceding

considerations. The most highly regarded of instruments will provide useless data, for instance, if

administered incorrectly by someone disliked by respondents, under noisy, inhospitable

conditions, or when subjects are exhausted.

All the above questions are important for researchers to answer, therefore, before they

begin to collect the data they need. A researcher's decisions about location, time, frequency, and

administration are always affected by the kinds of instrument to be used. And for it to be of any

value, every instrument, no matter what kind, must allow researchers to draw accurate

conclusions about the capabilities or other characteristics of the people being studied.

For each of the variables in quantitative in nature that researcher plan to study, the

measurement should be indicate whether the researcher will measure it by a test, questionnaire,

interview, observational procedure, or content analysis. It should be indicate whether the measure

is already available or whether will need to develop it. For each measure stated, which types of

validity and reliability are relevant and how researcher will check them should be indicating.

(MD Gall, 2010)

If the study is qualitative in nature, whether the data collection will focus on etic or emic

perspectives or both must be indicate. How the data will be collecting on each case feature that

researcher has chosen for study and the nature of researcher involvement in the data-collection

process.

20

3.11 Examples Of Data Collection in Research Design

Example 1: Outline of a proposal for a descriptive research study

A. Purpose of study

The purpose of this study is to learn how educators go about aligning curriculum content

with instruction and test content under conditions involving a federal or state mandate to

improve students' learning.

B. Methods of Data Collection

a) Measures

A questionnaire will use as data-collection instrument. It will be mailed to the principal of

each school. The questionnaire will include:

i. A scale on which the principal will rate his school's involvement in a curriculum-

instruction-test alignment process.

ii. A list of problems and solutions. For example of problem, school staff members who

question the need for an alignment process. For example of solution, is recruiting an

external consultant that is commonly found in the literature on school improvement.

iii. Space for the principal to indicate problems and solutions not on the list, and also to

make comments about the alignment process.

b) Validity and Reliability

Three phases that involves in this method:

i. A small sample of principals not involved in the study will be asked to evaluate the

questionnaire items for clarity and relevance to the alignment process. The questionnaire

will make changes based on their feedback.

ii. A small group of principals in the research sample will be select for a reliability check.

Each of these principals will be ask to nominate another educator in the school who is

knowledgeable about the school's activities over the past year.

iii. Each of these nominated individuals will be ask to complete the same questionnaire. If

the questionnaire is reliable, we should find a high level of agreement between the

principal's responses and the other individual's responses.

Example 2: Outline of a proposal for a case study (Qualitative study)

21

A. Purpose of study

The purpose of this study is to learn how educators go about aligning curriculum content

with instruction and test content under conditions involving a federal or state mandate to

improve students' learning.

B. Methods of Data Collection

a) Measures

The case study is exploratory, so measures that capture a wide range of data will be use.

The significant events will be observe and take notes in the process. These notes will provide

the basis for interviewing event participants about their perceptions of specific incidents that

occurred during an event.

If an upcoming event seems particularly significant, efforts will be made to videotape

it. The video with the event participants will be watching to obtain their perceptions as the

event unfolds.

Furthermore, stakeholders who were not directly involved in the event but who will be

affected by it will be interview. The significant documents prepared by stakeholders will be

collect during the alignment process.

b) Validity and Reliability

One researcher will be the primary observer and interviewer. However, another researcher

occasionally will observe the same event as a check on inter-observer reliability. Also, another

interviewer will interview research participants who have the same perspective (e.g., two

teachers at the same grade level) to determine whether both interviewers ask designated

questions and collect similar kinds of data.

c) Emic and Etic Perspective

The procedures for data collection will focus on an emic perspective, that is, the perspective of

the stakeholders as they experience the curriculum alignment process. The researchers who

collect data will not participate in the alignment process. They will maintain a supportive

perspective but will act primarily as observers. If asked their opinion or advice, they will defer

from offering it.

3.2 Data-collection instrument

22

Many of the instruments are completed by either the researchers or subjects in particular study.

For examples:

Instruments of data collection

Researcher completes Subject completes Research design/methodology

Rating scales Questionnaires Survey /descriptive

Correlation

Group comparison

Experimental

Interview schedules Self-checklists Survey /descriptive

Correlation

Group comparison

Experimental

Case study

Narrative study

Observation forms Attitude Scales Survey /descriptive

Correlation

Group comparison

Experimental

Case study

Tally sheets

A listing of :

activity categories/

behaviors

Personality/ character

inventories

Experimental

Performance checklists

(contain list of

behaviors)

Achievement/aptitude tests

Survey /descriptive

Correlation

Group comparison

Experimental

Anecdotal records Performance tests Survey /descriptive

Experimental

3.21 Researcher-Completed Instruments

23

3.211 Rating Scales

A rating is a measured judgment of some sort. When people were rate, a judgment about their

behavior or something they have produced were make. Thus, both behaviors and products of

individuals can be rated.

A. Behavior Rating Scales

Behavior rating scales appear in several forms, but those most commonly used ask the observer to

circle or mark a point on a continuum to indicate the rating. The simplest of these to construct is a

numerical rating scale, which provides a series of numbers, each representing a particular rating.

Example 1:

Behavior rating scale for teachers

Instructions: For each of the behaviors listed below, circle the appropriate number, using the

following key: 5 = Excellent, 4 = Above Average, 3 = Average, 2 = Below Average, 1 = Poor.

Behaviour characteristics PoorBelow

averageAverage

Above

averageExcellent

A. Explains course material clearly 1 2 3 4 5

B. Establishes rapport with students 1 2 3 4 5

C. Asks high-level questions 1 2 3 4 5

D. Varies class activities 1 2 3 4 5

The example of behavior scale above can be improved if additional meaning to each number is

given to describe it more fully. For example:

Improvement:

Instructions: For each of the behaviors listed below, circle the appropriate number, using the

following key: 5 = Excellent, 4 = Above Average, 3 = Average, 2 = Below Average, 1 = Poor.

Definition of the number:

24

Rating 5: Among the top 5 percent of all teachers you have had.

The graphic rating scale is an attempt to improve on the vagueness of numerical rating scales. It

describes each of the characteristics to be rated and places them on a horizontal line on which the

observer is to place a check mark.

Example 2:

Graphic rating scale.

Instructions: Indicate the quality of the student's participation in the following class activities by

placing an X anywhere along each line.

No Characteristics Always FrequentlyOccasionall

ySeldom Never

1 Listens to teacher's instructions . X

2Listens to the opinions of other

studentsX

3Offers own opinions in class

discussionsX

Here again, this scale would be improved by adding definitions, such as defining:

Always as “95 to 100% of the time

Frequently as “70 to 94% of the time

B. Product Rating Scales

Researchers may wish to rate products. Examples of products that are frequently rated in

education are book reports, maps and charts, diagrams, drawings, notebooks, essays, and creative

endeavors of all sorts. Behavior ratings must be done at a particular time, when the researcher can

observe the behaviour. A big advantage of product ratings is that they can be at any time

25

3.212 Interview Schedules

Interview schedules and questionnaires are basically the same kind of instrument which is a set of

questions to be answered by the subjects of the study. Interviews are conducted orally, and the

answers to the questions are recorded by the researcher or someone the researcher has trained

(Fraenkell & Wallen, 2009)

The advantages of this instrument are that the interviewer can clarify any questions that are

obscure and also can ask the respondent to expand on answers that are particularly important or

revealing.

A big disadvantage, on the other hand, is that it takes much longer than the questionnaire to

complete. Furthermore, the presence of the researcher may inhibit respondents from saying what

they really think.

Some interview schedules phrase questions so that the responses are likely to fall in certain

categories. This is call precoding. Precoding enables the interviewer to check appropriate items

rather than transcribe responses, thus preventing the respondent from having to wait while the

interviewer records a response.

Example of a structured interview schedule:

1. Would you rate pupil academic learning as excellent, good, fair, or poor?

a. If you were here last year, how would you compare pupil academic learning to previous

years?

b. Please give specific examples.

2. Would you rate pupil attitude toward school generally as excellent, good, fair, or poor?

a. If you were here last year, how would you compare pupil attitude toward school generally to

previous years?

b. Please give specific examples.

26

3.213 Direct observation

Example:

Directions:

Place a check mark each time the teacher:

Frequency

a. asks individual students a question / / / / / / 6

b

.

asks questions to the class as a whole/ / 2

c. disciplines students / 1

d

.

asks for quiet/ / / 3

e. asks students if they have any questions / 1

f. sends students to the chalkboard / / 2

3.214 Tally Sheets

A tally sheet is a device often used by researchers to record the frequency of student behaviors,

activities, or remarks. A tally sheet is simply a listing of various categories of activities or

behaviors on a piece of paper.

Examples:

How many high school students follow instructions during fire drills?

How many instances of aggression or helpfulness do elementary students exhibit on the

playground?

How often do students in Mr. Jordan's fifth-period U.S. history class ask questions? How

often do they ask inferential questions?

So, tally sheets can help researchers record answers to these kinds of questions efficiently.

3.215 Performance checklist

One of the most frequently used of all measuring instruments is the checklist. A performance

checklist consists of a list of behaviour that make up a certain type of performance.

27

Example of performance checklist noting student action in science laboratory:

Action Yes / No

Takes slide

Wipes slide with lens paper

Wipe slide with cloth

Moves bottle of culture along the table

3.22 Subject completed instruments

3.221 Self-checklist

A self-checklist is a list of several characteristics or activities presented to the subjects of the

study. The individuals are asked to study the list and then to place a mark opposite the

characteristics they possess or the activities in which they have engaged for a particular length of

time. Self-checklists are often used when researchers want students to diagnose or to appraise

their own performance.

Example of a self-checklist for use with elementary school students:

No Mon Tue Wed Thu Fri

1. I participated in class discussion / / /

2. I did not interrupt others while they were speaking / / / / /

3. I encourage others to offer their opinions /

3.222 Attitude Scales

The basic assumption that underlies all attitude scales is that it is possible to discover attitudes by

asking individuals to respond to a series of statements of preference. A scale is a type of

composite measure that is composed of several items that have a logical or empirical structure

28

among them. That is, scales take advantage of differences in intensity among the indicators of a

variable.

For example, when a question has the response choices of "always," "sometimes," "rarely,"

and "never," this is a scale because the answer choices are rank-ordered and have differences in

intensity.

There are several different types of scales. There are four commonly used scales in social

science research:

3.2221 Likert Scale

Likert scales are one of the most commonly used scales in social science research. It is named

after its creator, psychologist Rensis Likert. On a survey or questionnaire, the Likert scale

typically has the following format:

Strongly agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

It should be noted that the individual questions that use this format are called Likert items

while the Likert scale is a sum of several Likert items. To create the scale, each answer choice is

assigned a score (say 0-4) and the answers for several Likert items can be summed together for

each individual to get an overall Likert score.

Example 1:

A researcher is interested in measuring prejudice against women. One way to do that would be to

create a series of statements reflecting prejudice ideas, each with the Likert response categories

listed above.

The items might be

"Women shouldn’t be allowed to vote"

"Women can’t drive as well as men."

29

Each of the response categories would then assign a score of 0 to 4. For example, assign a score

of 0 to "strongly disagree," a 1 to "disagree," a 2 to "neither agree or disagree," and so on. The

scores for each of the statements would then be summed for each respondent to create an overall

score of prejudice. If there had 5 statements and a respondent answered "strongly agree" to each

item, his overall prejudice score would be 20, indicating a very high degree of prejudice against

women.

3.2222 Bogardus Social Distance Scale

The Bogardus social distance scale was created by Emory Bogardus as a technique for measuring

the willingness of people to participate in social relations with other kinds of people.

For example, a researcher interested in the extent to which U.S. Christians are willing to associate

with, say, Muslims. We might ask the following questions:

1.Are you willing to live in the same country as Muslims?

2. Are you willing to live in the same community as Muslims?

3. Are you willing to live in the same neighborhood as Muslims?

4. Are you willing to live next door to a Muslim?

The clear differences in intensity suggest a structure among the items. Presumably if a person is

willing to accept a certain association, he or she is willing to accept all those that precede it on the

list.

The Bogardus scale demonstrates that scales can be important data reduction tools. By

knowing how many relationships with Muslims a given respondent will accept, we know which

relationships were accepted. A single number can thus accurately summarize five or six data

items without a loss of information.

3.2223 Thurstone Scale

The Thurstone scale, created by Louis Thurstone, is intended to develop a format for generating

groups of indicators of a variable that have an empirical structure among them. For example,

studying about discrimination, a list of items would put together (maybe use 10 items in this

example) and then ask respondents to assign scores of 1 to 10 to each item. In essence, they are

30

ranking the items in order of which is the weakest indicator of discrimination all the way to

which is the strongest indicator.

Once the respondents have scored the items, the researcher examines the scores assigned to

each item by all the respondents to determine which items the respondents agreed upon the most.

If the scale items were adequately developed and scored, the economy and effectiveness of data

reduction present in the Bogardus social distance scale would appear.

3.2224 Semantic Differential Scale

The semantic differential scale asks respondents of a questionnaire to choose between two

opposite positions using qualifiers to bridge the gap between them.

Example:

To get respondents’ opinions about a new comedy television show.

First, what dimensions were wish to measure is decide and then two opposite terms that represent

those dimensions is finding. For example, "enjoyable" and "unenjoyable," "funny" and "not

funny," "relatable" and "not relatable." A rating sheet for each respondent would then create to

indicate how they feel about the television show in each dimension. The questionnaire would

look something like this:

Very Much Somewhat Neither Somewhat Very Much

Enjoyable X Unenjoyable

Funny X Not Funny

Relatable X Unrelatable

3.23 Performance Test

Performance test measures an individual's performance on a particular task. An example would

be a typing test, in which individual scores are determined by how accurately and how rapidly

people type.

This test involves the evaluation of individuals as they carry out a complex real-life task. A

driving test is an example of a performance measure, because the test requires someone drive a

car while being evaluated by a state examiner.

31

3.24 Analysis of document

Researchers sometimes focus their observations on documents produced or used by research

participants. The investigation of data derived from documents is called content analysis (Punch,

2009)

Example 1:

Researchers might study how males and females are portrayed in textbooks or the issues that are

mentioned in the minutes of school board meetings.

Content analysis involves the development of categories and a frequency account of the

occurrence of each category in the document.

Example 2:

Analysis of elementary school mathematics textbooks:

Number calculation problems, word problems involving real-life situations that children

might encounter, and word problems involving real-life situations that children are not

likely to encounter.

In evaluating the soundness of a content analysis, researcher should look for evidence that:

1) the categories are clearly defined and worthy of study,

2) the procedure for selecting a sample of documents is sound, and

3) different observers are able to use the categories reliably.

4.0 TRIANGULATION

4.1 Control for Researcher Errors and Biases

Researchers acknowledge the likelihood that their own errors and biases will affect their data

collection. Therefore, they design research studies to minimize the influence of such factors.

32

Some researchers sometime using a variety of instruments to collect data. When a conclusion is

supported by data collected from a number of different instruments, its validity is thereby

enhanced. This kind of checking is often referred to as triangulation.

An approach often used in case study research is to validate findings by triangulation of

data sources. Triangulation refers to researchers' attempts to corroborate data obtained by one

method (e.g., observation of individuals) by using other methods (e.g., interviews of individuals

or examination of documents).

In case study, researchers use any methods of data collection that are appropriate to their

purpose. They might begin a case study with one method of data collection and gradually shift to,

or add, other methods. The purpose for this data-collection strategy is triangulation, also known

as crystallization (Richardson & St. Pierre. 2005), which involves the use of multiple methods to

collect data about the same phenomenon in order to confirm research findings or to resolve

discrepant findings. It can also involve the use of different data sources, methods of analysis, or

theories to check case study findings.

Triangulation is fundamental in ethnographic research. Essentially, it establishes the

validity of an ethnographer's observations. It involves checking what one hears and sees by

comparing one's sources of information (Fraenkel & Wallen, 2009)

Based on Guion (2011), triangulation is a method used by qualitative researchers to check

and establish validity in their studies. There are five types of triangulation will be examined:

1) data triangulation

2) investigator triangulation

3) theory triangulation

4) methodological triangulation

5) environmental triangulation

4.11 Data Triangulation

Data triangulation involves the use of different sources of data/information. A key strategy is to

categorize each group or type of stakeholder for the program that are evaluating. Then, be certain

to include a comparable number of people from each stakeholder group in the evaluation study.

Example:

Evaluating an afterschool program

33

First: Identify the stakeholder groups such as youth in the program, their parents, school teachers,

school administrators, afterschool program staff and volunteers.

Second: To gain insight on what the stakeholders perceive as outcomes of the program,

researcher has conduct in-depth interviews. The representatives of each stakeholder group are

interview.

Triangulate by looking for outcomes that are agreed upon by all stakeholder groups. The weight

of evidence suggests that if every stakeholder, who is looking at the issue from different points of

view, sees an outcome then it is more than likely to be a true outcome.

4.12 Investigator Triangulation

Investigator triangulation involves using several different investigators/evaluators in an

evaluation project. Typically, this would manifest as an evaluation team that consists of your

colleagues within your program area/field of study. In order to triangulate, each different

evaluator would study the program using the same qualitative method (interview, observation,

case study, or focus groups).

The findings from each evaluator would be compared. If the findings from the different

evaluators arrive at the same conclusion, then validity has been established. If the conclusions

differ substantially, then further study is warranted to uncover the "true" and "certain" finding.

Example:

To assess changes in nonverbal communication and public speaking skills, pre/post observations

of youth in the Astro Ceria public speaking program were conducted.

In order to triangulate, different colleagues in field would line up to serve as evaluators. Each

person would have the same observation check sheet for pre- and post-observations. In the final

analysis, validity would be established for those same practice changes and skills that were

identified by each different observer (per child).

While this is an effective method of establishing validity, it may not always be practical to

assemble different evaluators given time constraints and individual schedules.

4.13 Theory Triangulation

34

Theory triangulation involves the use of multiple professional perspectives to interpret a single

set of data or information. Unlike investigator triangulation, this method typically entails using

professionals outside of researcher field of study.

One popular approach is bringing together people from different disciplines; however,

individuals within disciplines are used if they are in different status positions. In theory it is

believed that individuals from different disciplines or positions bring different perspectives.

Therefore if each evaluator from the different disciplines interprets the information in the same

way (draws the same conclusions), then validity is established.

Example:

To learn what diet or healthy lifestyle practice changes, participants in a nutrition program were

interviewed.

To triangulate the information, the transcripts could then share with colleagues in different

disciplines (i.e. nutrition, nursing, pharmacy, and public health education) to see what their

findings and conclusions are. Those method are compare and again, as with others methods of

triangulation, researcher would look for congruence to establish validation in findings. As with

investigator triangulation, this method may not be feasible in all situations. Also, it may be more

time consuming to try to involve individuals from other disciplines.

4.14 Methodological Triangulation

Methodological triangulation involves the use of multiple qualitative and quantitative methods to

study the program. If the conclusions from each of the methods are the same, then validity is

established.

Example:

A case study of one of Welfare-to-Work participants to document changes in their life as a result

of participating in a program over a one- year period.

Researcher would not just use one method, but they would use interviewing, observation,

document analysis, or any other feasible method to assess the changes. Survey the participant, her

family members and case workers (quantitative method) also could be done. If the findings from

35

all of the methods draw the same or similar conclusions, then validity in the finding has been

established.

This is also a popular method of triangulation that is widely used. However, in practice, this

method may require more resources in order to evaluate the program using different methods.

Likewise, it will require more time to analyze the data/information yielded by the different

methods.

4.15 Environmental Triangulation

This type of triangulation involves the use of different locations, settings and other key factors

related to the environment in which the study took place, such as time of the day, day of the week

or reason of the year. The key is identifying which environmental factor, if any, may influence

the information of researcher received during the study. The environmental factor is changed to

see if the findings are the same. If the findings remain the same under varying environmental

conditions, then validity has been established.

Example:

To evaluate the effectiveness of money management program research.

A researcher wants to determine if his program helps participants develop budgets to minimize

spending and increase savings. If he evaluate during the holiday season, he may get different

results because spending is greatly increased during that time of year.

In order to triangulate, he would need to evaluate the budgeting, spending and saving habits

of his participants throughout the year in order to gather true and certain information on their

behavior changes.

Unlike the other types of triangulation, environment triangulation cannot be used in every

case. It is only used when it is likely that the findings may be influenced by some environmental

factor.

36

Figure 1: Triangulation and Politics

5.0 CONCLUSION & SUMMARY

A well designed questionnaire is essential to a successful survey. However, researchers must

develop their own intuition with respect to what constitutes good design since there is no theory

of questionnaires to guide them.

A good questionnaire is one which helps directly achieve the research objectives, provides

complete and accurate information; is easy for both interviewers and respondents to complete, is

so designed as to make sound analysis and interpretation possible and is brief.

There are at least nine distinct steps: decide on the information required; define the target

respondents, select the methods of reaching the respondents; determine question content; word

the questions; sequence the questions; check questionnaire length; pre-test the questionnaire and

develop the final questionnaire.

Data Collection is an important aspect of any type of research study. Inaccurate data

collection can impact the results of a study and ultimately lead to invalid results. There are many

different types of data collection methods that can be used in any evaluation. Each has its

advantages and disadvantages and must be chosen in light of the particular questions, timeframe,

and resources that characterize the evaluation task. While some evaluators have strong

preferences for quantitative or qualitative techniques, today the prevailing wisdom is that no one

approach is always best, and a carefully selected mixture is likely to provide the most useful

information.

37

REFERENCES

Boynton, P. M., & Greenhalgh, T. (2004). Selecting, designing, and developing your

questionnaire. Bmj, 328(7451), 1312-1315.

Brace, I. (2008). Questionnaire design: How to plan, structure and write survey material for

effective market research. Kogan Page Publishers.

Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2009). How to design and evaluate research in

education (7th edition). New York: McGraw-Hill.

Gall, M. D, Joyce P. Gall . (2010). Applying educational research: How to read, do, and use

research to solve problems of practice.6th edition. Pearson Higher Ed.

Gall, M. D. (2013). Applying educational research: How to read, do, and use research to solve

problems of practice. Pearson Higher Ed.

Guion, L. A., Diehl, D. C., & McDonald, D. (2011). Triangulation: Establishing the validity of

qualitative studies.

Punch, K. F. (2009). Introduction to research methods in education. Sage.

Richardson. L.. & St. Pierre. E. A. (2005). Writing: A method of inquiry. In N. K. Denzin & Y.

S. Lincoln (Eds.), The Sage handbook of qualitative research (3rd ed.. pp. 959-978). Thousand

Oaks. CA: Sage.

38