Upload
thethanh-phamnguyen
View
216
Download
0
Embed Size (px)
Citation preview
7/31/2019 HBEF4106-CHP5
1/22
58 OUM
TOPIC 5RESEARCH METHODOLOGY
TOPIC 5: RESEARCH METHODOLOGY
Assoc. Prof. Dr. Saat Md. Yasin
INTRODUCTION
This topic deals with the general concept in carrying out a research. It deals with the appropriatesteps in carrying out research methodology to ensure that a new researcher would not fall
into grave errors that could waste a lot of precious time and funding. This topic guides you inthe understanding of the process to fulfil certain objectives of this course.
OBJECTIVES
By the end of this topic, you should be able to:
1. plan a research in education successfully;
2. state the differences between qualitative and quantitative research designs;
3. describe a population and a sample;
4. select sample using random sampling;
5. develop research instruments;
6. conduct a pilot study;
7. explain how to establish the reliability and validity of the research instruments;
8. analyse quantitative and qualitative data; and
9. explain the differences between descriptive and inferential statistics.
7/31/2019 HBEF4106-CHP5
2/22
OUM 59
TOPIC 5 RESEARCH METHODOLOGY
TOPIC MIND MAP
5.1 WHAT IS RESEARCH?
Research is a formal and systematic application of the scientific and disciplined inquiry to thestudy of problems while methodology is a way or means of doing something such as the way
of making a scientific inquiry into a problem. Since the research methodology in educationalresearch is scientific and disciplined in nature, it has to adhere to certain procedures that
need to be strictly followed. First of all, any research to be carried out needs to be carefullyplanned to ensure the validity of the findings. We have to consider at least three important
factors to ensure that the research is properly implemented.
5.2 RESEARCH PLANNING
After identifying the area or the topic for the research, you need to formulate the purpose,objectives, questions and hypotheses of the research. You have to draw out the methods and
the procedures of the intended research. Therefore, you require expertise in a number of
7/31/2019 HBEF4106-CHP5
3/22
60 OUM
TOPIC 5RESEARCH METHODOLOGY
different areas. The research plan describes the nature of the participants in the research,the variables, the kind of data, the instruments to be used, the condition for data collection
and the techniques to analyse the data.
When identifying the area of study, you have to consider the type of research to be utilised. Is
it a quantitative or qualitative research? Which one is more suitable? Planning a researchusing the quantitative method is different from planning a qualitative research. You have to
consider many different factors when applying either the quantitative or the qualitative methods.
5.2.1 Research Questions and Data
Research questions are the questions that we ask to achieve the objectives for our research.
These questions can be answered through collecting of data and analysing them. After theresearch questions have been established and the topic of the study has been identified, you
have to describe the following procedures in executing the research.
(a) Describe and execute procedures to collect information. The procedures includeidentifying research participants, the strategies to collect data and activities describing,
how, when, why and from whom the data were collected. The detailed description would
enable you to smoothly carry out the research without major hitches if there were any.The flaws in carrying out the research might jeopardise the validity of the research findings.
(b) Analyse the collected data. The data would be analysed differently depending on thenature of the research. If the research was quantitative, then the data statistical analysis
might be needed to achieve the desired findings. On the other hand, if the research wasqualitative in nature, then the data analysis requires a descriptive explanation of the
findings. Lengthy notes might be scrutinised and summarised. The observations made
might be transcribed in coherent and easily understood explanations.
(c) After the data have been analysed, the result or the findings of the study has to be
presented coherently. They must be clearly stated and the implications of the findings
should also be included. The findings might be useful to interested parties for various
reasons. The government might want to improve whatever shortcomings pointed out bythe findings. The school principals might want to use them for school improvement.
1. What is the expertise needed in planning a research successfully?
2. What are the factors that you need to consider when applying either thequantitative or the qualitative methods?
7/31/2019 HBEF4106-CHP5
4/22
OUM 61
TOPIC 5 RESEARCH METHODOLOGY
5.2.2 Selection of Research Design
A research design is a plan to carry out an intended research. According to Creswell (2002),
there are eight different research designs which are frequently used in educational research.
They are either quantitative or qualitative in nature or in some instances, the combination ofboth.
(a) Quantitative Research DesignsThe experimental and quasi-experimental designs will be explained separately.
(i) Experimental DesignsThey are usually referred to as group comparison studies because researchers
compare different groups in terms of one or more outcomes. Experiments are alsocalled intervention studies because you intervene or manipulate the variables.
The experimenters design and conduct experiments where they maximise theinfluence of the independent variable on the dependent variable, and at the same
time minimise the influence of the extraneous factors. There are five keycharacteristics of experimental research designs that are of importance to you:
Participants are selected and assigned at random to different groups, namely
control and experimental groups.
Intervention is applied to one or more groups
Outcomes are measured at the end of the experiment
Procedures are designed that address potential threats to validity
Statistical comparisons of different groups are conducted.
In the educational scene, experimental research could be used in various situations.One such situation is to find out the effectiveness of a new method of teaching over
a conventional one. Two groups of students were randomly selected. One group is
used as the experimental group using the new method of teaching, while the other
is the control group using the conventional method of teaching. The topic of thelesson is chosen from the syllabus and the different treatments are given. After the
Figure 5.1: Research design
RESEARCH DESIGN
Quantitative
1. Experimental2. Quasi-experimental3. Correlational4. Survey
Combination ofQuantitative andQualitative
1. Mixed Method2. Action Research
Qualitative
1.Grounded Theory2. Ethnographic3. Narrative
7/31/2019 HBEF4106-CHP5
5/22
62 OUM
TOPIC 5RESEARCH METHODOLOGY
treatments had been completed, the outcome of the experiment is measured. Theoutcome might be in the form of the academic achievement through the test results
of the students in the control group and the experimental group. This statistical
analysis would provide the answer whether the experimental group performedsignificantly better than the control group.
(ii) Quasi-Experimental DesignMost of the situations in schools need you to use intact groups such as in studying
a new English programme. This study may involve the students in the same grade
as respondents where one class is considered the control group while the other isconsidered as the experimental group. Randomly assigning students to the two
groups would disrupt classroom learning. Because teachers often use intact groupin experiments, quasi-experimental designs are usually used.
(iii) Correlational Designs
A science teacher wants to know whether the assumption that students who are
good in mathematics are also good in science is true or false. In order to find out,she compares the examination results of her students in the two subjects, themathematics and science. She correlates the mathematics test results and the
science test results of the students. If the students who scored high marks inmathematics also scored high marks in science, we can conclude that there is a
positive correlation between the mathematics and the science results, and if this
finding is consistent with other research findings in the same area, we could saythat the assumption is correct.
The above anecdote describes one of the correlational designs used in the
educational field. Correlational research in the above case uses a statisticalprocedure. Now, with sophisticated applications and explicit procedures of
correlations, such as regression, path analysis, and causal modelling, correlationalresearch is entrenched in the family of quantitative research. Correlational researchas a design, has two primary forms, namely explanation and prediction. Correlational
research helps to explain the association between two or more variables or to predict
an outcome.
(iv) Survey DesignsSuppose the Ministry of Education (MOE) wants to know what the parents and the
students feel about having one session schools where students begin school at8.00 a.m. and ends at 4.00 p.m. The research question might be: What are the
opinion of the students and their parents toward one session school and lengtheningof the school session?By using a survey design, the MOE would gather the opinions
of the students and their parents. The survey design is an economical way of doingresearch especially when the respondents are large. The MOE would randomly
select the respondents from the two groups: students and their parents, and sendthem a mailed questionnaire, analyse the results and generalised results from of
sample to the population. This is a survey that is commonly used in the school
system. Survey designs are also frequently used to:
describe trends;
determine opinions about policy issue, such as giving loans to all students in
tertiary institutions;
background of lecturers from private and public colleges and universities;
understand important attitudes and beliefs towards dating among secondary
school students;
7/31/2019 HBEF4106-CHP5
6/22
OUM 63
TOPIC 5 RESEARCH METHODOLOGY
identify practices, such as carrying weapons to schools and gangsterism; and
evaluate programmes, such as using English for mathematics and science
subjects.
Survey designs are mainly of two types: cross-sectional and longitudinal. Cross-
sectional designs are used to collect data that reflect current attitudes, opinions or
beliefs. Longitudinal designs are used to study individuals for a period of time suchas studying people who were high achievers and low achievers in schools.
1. What research design do you use when you want to know the effectivenessof your new method of teaching oral English?
2. Explain how you implement the design.
3. What are the differences between Experimental Design and Quasi-
experimental Design?
(b) Qualitative Research DesignsThe qualitative types will be explained in the following paragraphs.
(i) Grounded Theory Designs
A teacher who had been teaching for the past twenty years had the experience ofteaching students from different socio-economic backgrounds. He found out that
most of the high achievers were from the higher socio-economic background andmost of the low achievers were from the lower socio-economic background. As a
result, he developed his own theory that students from the higher economicbackground have parents who are more concerned that their children must have
good grades. Their emphasis on discipline and spending more time in doing theirhomework enables their children to excel academically. On the other hand, students
from the lower socio-economic background spend less time on studying, lack of
parental supervision, and have non-conducive home environment for learning whichdo not help them to perform well. The teacher wanted to make a study on this
phenomenon which is grounded on a theory. He wanted to test his hypothesisthat includes the following:
Students from a higher socio-economic background spend more time studying.
Parents from a higher socio-economic background send their children to tuitionclasses.
Students from a higher socio-economic background have relatives who are high
achievers as their models.
The research that is most likely to be used in this situation is of qualitative type innature. The research based on theory, as in the above case, requires observation
as well as interviews. The data collected would be lengthy and qualitative in nature.The data would be analysed and a report of the findings has to be made. To know
how this Grounded Theory Designs work, let us examine one published groundedtheory studies (Creswell , 2002):
7/31/2019 HBEF4106-CHP5
7/22
64 OUM
TOPIC 5RESEARCH METHODOLOGY
Table 5.1: Nine Types of Ethnographic Designs
Realist ethnography An objective scientifically-written ethnography.
Confessional ethnography A report of the ethnographers fieldwork experiences.
Life history A study of one individual situated within the cultural context.
Auto-ethnography A reflective self examination by an individual set within hiscultural context.
Micro-ethnography A study focused on a specific aspect of a cultural group.
Ethnographic case study A case analysis of a person, event, activity, or process.
Critical ethnography A study of women and the cultural practices that serve todisempower and oppress them.
Postmordern ethnography An ethnography written to challenge the problems in our society.
Ethnographic novels A fictional work focused on cultural aspects of a group.
Richie, Fassinger, Linn and Johnson (1997), studied the career development process of18 high-achieving African American-Black and white women by generating a theoretical
model. They explored critical influences in the careers of these 18 women. From theinterview data, they developed a theoretical model that visually portrayed how socio-
cultural conditions, personal backgrounds, such as socio-economic background,influence the womens strength and passion for work and interconnectedness with others
in their careers.
(ii) Ethnographic DesignsThe term ethnography literally means writing about groups of people (LeCompte& Schensul, 1999: 21). Using this design, you can identify a group of people to
study, visit the setting of the group, note how they behave and interact with others,
how they think and develop a general picture of the group. It can provide a day today detailed picture of their activities. As an example, let us look at, The life of
residential school students in Malaysia. This study requires you to observe thestudents and make notes on their daily activities. You could become a participant
observer in order to know and understand the details of their daily life. By being withthem for a certain length of time, you would have an ethnographic view of the
students.
Ethnographic designs are qualitative research procedures for describing behaviour,beliefs, and language that they developed over a period of time. In the case of
residential school students in the example mentioned, they have their own patternof behaviour and their shared values which are likely to be different from other
students in non-residential schools. Van Maanen (1988); Denzin (1997); LeCompte,
Preissle and Tesch (1993) had classified nine (9) types of ethnographic designs asshown in Table 5.1.
7/31/2019 HBEF4106-CHP5
8/22
OUM 65
TOPIC 5 RESEARCH METHODOLOGY
The different ethnographic designs stated above share similar characteristics whenreporting the findings. If an investigator intends to utilise one of the ethnographic
designs above, he/she has to read the subject matter further to ensure the most
appropriate approach for the research.
(iii) Narrative Research Designs
Narrative research is a design that is used by individuals to tell their personal, firsthandaccounts to researchers. In narrative research, investigators describe the lives of
individuals, collect and tell stories about peoples lives and write narratives ofindividual experiences (Connelly & Clandinin, 1990). As a distinct form of qualitativeresearch, a narrative usually focuses on studying a single person, gathering data
from the collection of stories.
Narrative research can assume multiple forms. Narratives may include a variety ofresearch practices as was compiled by Casey (1995, 1996). Examples of types of
narrative research forms are: Autobiographies.
Biographies.
Life Writing.
Personal Accounts.
Narrative Interviews.
Personal Documents.
Documents of Life.
Life Stories and Life Histories.
Oral Histories.
Ethnohistories.
Ethnobiographies
Autoethnographies.
Ethnopsychologies.
Person-centred Ethnographies
Popular Memories.
Latin-American testimonies.
Polish memoirs.
For teachers in Malaysian schools, not all of the above types of narrative research
could be applied. Only some of the most commonly used, such as autobiographiesand biographies, are usually utilised in research.
(c) A Combination of Qualitative and Quantitative Research Designs
The following paragraphs will elaborate on the combined qualitative and quantitativeresearch designs.
(i) Mixed Method Designs
A mixed method research design is a procedure of collecting both qualitative andquantitative data in a single study. Since the 1930s, educational and social science
7/31/2019 HBEF4106-CHP5
9/22
66 OUM
TOPIC 5RESEARCH METHODOLOGY
1. If a researcher wants to find out the reasons why some students achieved
excellent academic results, while others failed to do so even though the
initial academic potentials seem to be equal, what research design doyou suggest a researcher could choose? Explain.
2. What action research can a teacher do in school to improve his/her
teaching?
researchers have combined research methods of data collection in their studies(Sieber, 1973). Sieber suggested the combination of in-depth case studies with
surveys, creating a new style of research where quantitative and qualitative methods
were used within a single study. The rationale for carrying out the mixed methoddesigns in research is mainly to ensure that majority of the factors that influence
the situation could be taken care of. Majority of the variables have been looked intousing quantitative methods, but variables that could not be taken into account through
quantitative research, could be studied through qualitative research.
If a nationwide research is to be carried out where variables of various kinds, suchas the family, the school system, the curriculum, the students, and funding were to
be studied, the mixed method design would suit the purpose. Take study habitsas an example. When the details on how a student manages his/her time and the
method of how he/she learns, qualitative research such as ethnographic researchmight be better than just quantitative data, such as the number of hours spent on
studying. Other types of data that could easily be carried out quantitatively, such asfamily income, number of siblings, and academic background should be collected
quantitatively.
(ii) Action Research DesignsAction research addresses a specific, practical issue and seeks to obtain solutions
to the problem. Similar to the mixed method research, action research uses data
collection based on either quantitative or qualitative methods. Action researchprovides educators an opportunity to carry out research in the classroom to reflecttheir teaching practices. Action research offers a means for staff development; for
teachers development as professionals, and simultaneously addressing school-wide problems. As an example, an English teacher found out that her students
were weak in English from the students test results and her interaction with them.So, she decided to enquire from the students regarding their background such astheir families socio-economic background, the number of books they read in English,
whether the family members communicate in English with each other and theirattitude toward the English language. These variables could be researched
quantitatively as well as qualitatively to collect valid data and find out the variablesthat influenced the students command of the English language.
The findings of this action research would help the English teacher to know some
of the reasons why most of her students are weak in the English language. Thiswould help her to design a remedial programme to improve the students command
of the English language. If teachers are able to carry out action research to improve
their teaching, overall quality of the students learning would improve.
7/31/2019 HBEF4106-CHP5
10/22
OUM 67
TOPIC 5 RESEARCH METHODOLOGY
5.2.3 Research Population and Sample
Figure 5.2: Research population and sample
What is the definition of population? Gay and Airasian (2003), defined population as The
population is the group of interest to the researcher, the group to which the results of the
study will ideally generalized. To determine a population is important in an educational research.The population determines how far and how much data should be collected and analysed.
The interpretation and inference of the data is also limited to the population of the study. Thepopulation also determines how much time and energy should be devoted to solve a problem.
Population also influences how much fund is needed to carry out the research. As an example,a study on the needs of the rural students to improve their academic performance in a certain
school is different from a study on the needs of rural students in Malaysia. The populations ofboth samples are different in size which also determine the types of research designs to be
carried out.
Another important factor for the researcher to consider is accessible oravailable population.In most studies, accessible population is generally a realistic one compared to the target
population which is more of an idealistic one. Although we can use the whole population forour research, it is not practical because it requires a large amount of funding, manpower and
time to carry it out successfully. Therefore, we have to devise a means that would do the
same job without distorting the findings. This is done through random sampling.
Selecting a sample is a very important step especially when conducting quantitative research.A good sample is one that is representative of the population from which it is selected. The
steps in sampling can be summarised as follows: identify the population;
determine the required sample size; and
select the sample.
Take an example in selecting a sample from the Form1 students in a school. You as a teacher,first of all, have to identify the population, that is the Form 1 students in the school. Afterwards,
you need to determine the sample size. Out of the 200 students for example, you need tosample 60 students; 30 percent of the population. You need to select randomly from the
cohort of Form 1 students of the school. You can use one of the sampling techniques. These
sampling techniques are all based on randomness in the selection of the sample. There arefour basic random sampling procedures, namely:
Simple Random Stratified
RESEARCH POPULATION AND SAMPLE
Selecting Sample
Cluster Systematic
7/31/2019 HBEF4106-CHP5
11/22
68 OUM
TOPIC 5RESEARCH METHODOLOGY
simple random sampling;
stratified sampling;
cluster sampling; and systematic sampling.
(a) Simple Random SamplingRandom sampling is a process of selecting a sample in such a way that all individuals inthe selected population have an equal chance to be selected into the sample. Random
sampling is the best way to obtain a representative sample. The procedures in simplerandom sampling involve the following specific steps:
(i) Identify the population and define it.
(ii) Determine the sample size.
(iii) List the names of all members of the population.
(iv) Assign all individuals on the list a consecutive number from zero to the requi re dnumber, for example, 000 to 200 where each individual has the same number ofdigits as the others.
(v) Select an arbitrary number from the table of random numbers. As an example, byclosing your eyes while pointing to a number in the list.
(vi) Go to the next number until the desired number of individuals has been se lect ed
for the sample. If 30 percent of the 200 members in the population are to be selected,then 60 students in the sample will fulfill the desired percentage.
(b) Stratified SamplingThis is a process of selecting a sample in such a way that identified sub-groups in thepopulation is represented in the sample in the same proportion that they exist in the
population. For example, if we want to find out the academic qualification of teachersteaching various subjects, the population of the teachers that are to be studied is not
uniform because some of these teachers have degrees while the others do not havethem. Simple random sampling cannot be applied in this situation due to the fact that it
does not represent the strata that exists in the population. Besides, simple random
sampling to the above population will increase sampling error. Therefore, simple stratifiedsampling should be used to reduce sampling error.
The first step in carrying out the study using stratified sampling is determining the strata
to be studied such as by categorising the graduate and non-graduateteachers. Thenfrom every stratum, respondents are selected at random to represent the teachers
population. Random sampling for every stratum could be generated using the samestrategy as simple random sampling. The allocation of sample size between strata canbe done proportionately. If 60% of the teachers are non-graduate and only 40% are
graduates, then the sample should consists of 60% non-graduate teachers and 40% for
graduate teachers.
(c) Cluster Sampling
Cluster sampling is a process of using a sample whenever the sampling unit is not theunit that represents the population but represents a group or a cluster within the population.
For example, we dont have a complete list of teachers. We can use cluster samplingby determining the number of clusters such as the teachers based on the states they
come from. Sampling based on clusters could be done randomly by using the table of
7/31/2019 HBEF4106-CHP5
12/22
OUM 69
TOPIC 5 RESEARCH METHODOLOGY
random numbers. When the cluster that represents the population has been randomlyselected, we must use every unit in the cluster. After we have randomly selected the
teachers from one school district such as Putrajaya, all members within the cluster
must be used as the respondents for the study.For a clearer illustration of cluster sampling, note the following steps:
(i) Identify and define the population.
(ii) Determine the required sample size.
(iii) Identify and define a logical cluster (school, teachers in the district, primary
school teachers).
(iv) List all clusters that make up the population of clusters.
(v) Estimate the average number of population members by clusters.
(vi) Determine the number of clusters needed by dividing the sample size by theestimated size of a cluster.
(vii) Randomly select the required number of clusters.
(viii) Include all members in the selected cluster in your study.
(d) Systematic Sampling
It is a method of sampling where individuals are selected from a list taking every Nthname. If N here represents 5, then every 5 th name will be selected as the sample for the
study until the required numbers are met. This method of sampling though is not usedvery often.
The major difference between systematic sampling and the other types of sampling is
that all members of the population do not have an independent chance of being selectedfor the sample. Once the first name is selected, all the rest are automatically determined
for the sample. Therefore, systematic sampling is often not as good as random sampling.There is a possibility that students from a particular ethnic group having names that are
concentrated on certain letters could be excluded from the research through omission
by using systematic sampling.
Systematic sampling involves the following steps:
(i) Identify and determine the population
(ii) Determine the desired sample size.
(iii) List the names of the population.
(iv) Determine the N by dividing the size of the population by the desired sample.
(v) Start at some random place in the population list. Close your eyes and pointyour finger on a name.
(vi) Starting at that point, take every Nth name until the required sample size is
reached.
1. What do you understand by accessible population and target population?
2. When do you utilise systematic sampling?
3. Why is random sampling preferred over other sampling techniques?
7/31/2019 HBEF4106-CHP5
13/22
70 OUM
TOPIC 5RESEARCH METHODOLOGY
5.3 RESEARCH INSTRUMENTS
We are familiar with the terms test, examination, assessmentand measurement. A test or
examination is a formal, systematic procedure for gathering information about peoplescognitive, psychomotor and affective characteristics. A standardised test or examination isadministered, scored, and interpreted in the same way no matter where or when it is
administered. These standardised tests and examinations are developed by experts. For
example, the Ujian Penilian Sekolah Rendah (UPSR) or Primary School Assessment Test,Penilaian Menengah Rendah (PMR) or Lower Secondary Assessment, Sijil Pelajaran
Malaysia(SPM) or Malaysian Certificate of Education and the Sijil Tinggi Pelajaran Malaysiaor Malaysian Higher School Certificate are all standardised to ensure that all test - takers
experience the same conditions when taking them. Assessment is a broader term than testor instrument, and encompasses the process of collecting, synthesising and interpreting
information. Tests are a subset of measurement, as are observations and interviews.Measurement is the process of quantifying or scoring ones performance.
5.3.1 Developing Research Instruments
Before you carry out a research, you have to develop the research instruments. If it is to be atest, it should be planned and developed well to suit the purpose and objectives of the research.
A well-developed test would measure the variable of the research accurately. The accuratenessof the collected data is very important to ensure that the collected data could be analysed and
interpreted correctly.
(a) Determining the Objectives
The test should be constructed according to its objectives. The behavioural objectives
to be measured should be realistic and stated clearly. Objectives should also be statedbehaviourally and not in the form of learning activities. The statement of the objectives
should be linear whereby each and every statement should be related to only one process.
Figure 5.3: Developing research instruments
Determining Objectives
Determining Contents
DEVELOPING RESEARCH
INSTRUMENTS
7/31/2019 HBEF4106-CHP5
14/22
OUM 71
TOPIC 5 RESEARCH METHODOLOGY
(b) Determining the ContentsThe contents of the area to be measured should be determined first. An effective test is
a test which has a high content validity. A test should cover all contents of the research
area. To ensure the test used covers all contents, you should first determine the contentsof the research area. The opinion of the expert in the area could be utilised to ensure the
test to be developed has high reliability and validity.
Usually the content of the area of study is wide, so you could use sampling techniques
of the content as representatives of the content. You need to determine content sampling
to represent the behaviour to be measured. If content sampling is used, you need toensure that they are representatives of the behaviour to be measured. If this method is
used, the researcher should ensure that the chosen contents could represent the overallcontents that are to be measured.
5.3.2 Reliability and Validity of Research Instruments
Both reliability and validity of research instruments are vital in any research. Validity is themost important characteristic of a test or a measuring instrument. Validity is concerned with
the appropriateness of the interpretations made from test scores. When we test, we test it fora purpose. For example, you may administer a questionnaire to find out the participants
opinions on the new methodology used in teaching mathematics compared to the traditionalmethod where talk and chalk were used. A key question for these test users is, Does this
test allow me to make the interpretation I wish to make? That is, will responses to thequestionnaire allow you to make appropriate interpretations about the respondents opinions?
Validity is important to all forms of research and all types of tests and measures. There are
four types of test validity:
content validity;
criterion-related validity;
construct validity; and
consequential validity.
(a) Content ValidityContent validity is the degree to which a test measures an intended content area. It
requires both item and sampling validity. Item validity is concerned with whether the
test items really measure the content area. Sampling validity is concerned with howwell the test samples the total content area being tested. Content validity is particularly
important for achievement tests. A test score cannot accurately reflect a students
achievement if it does not measure what the student was taught and supposed to learn.The validity of the content is determined by expert judgment. Experts in the topic covered
by the test are asked to assess its content validity. The judgment is made by comparingwhat was taught and what is being tested. The validity would be very strong when the
two coincides.
(b) Criterion-Related Validity
Criterion-related validity has two forms, namely concurrent and predictive validity.
(i) Concurrent validity is the degree to which scores on one test correlate to scoreson another equivalent test. For example, a test is developed that claims to do the
same thing as some other tests, except easier or faster. One way to determine
7/31/2019 HBEF4106-CHP5
15/22
72 OUM
TOPIC 5RESEARCH METHODOLOGY
whether the claim is true or otherwise is to administer the new and the old test to agroup of students and compare the scores. If there is a high correlation between
them, the concurrent validity of the new test is established and it may be utilised to
replace the older test.(ii) Predictive validity is the degree to which a test can predict how well individuals
will do in a future situation. Predictive validity is very important for tests that used toclassify or select individuals. In Malaysia, we use the test results of UPSR for
selection into residential schools. These test results are considered as having high
predictive validity. The selected students admitted into these schools are expectedto perform well in the next public examination three years later, namely the PMR.
No test would have perfect predictive validity, and therefore predictions based on
the scores of any test will be imperfect. However, predictions based on a combinationof several test scores will be more accurate. In order to establish the predictive
validity of a test, the criterion has to be carefully defined, which is the valid measure
of the performance to be predicted. Once the criterion has been identified, theprocedure for determining predictive validity is as follows:
Administer the test which is considered as the predictor to a group.
Wait until the behaviour to be predicted occurs.
Obtain measures of the criterion for the same group.
Correlate the two sets of score.
Evaluate the results.
Once the two results are correlated, and if the coefficient is high, the test has good
predictive validity.
(c) Construct ValidityConstruct validity is the most important form of validity because it asks the question:
Wha does this test measure? Constructs are non-observable traits, such as anxiety,
intelligence and honesty. You can only observe its effect. Constructs can explain certaindifferences between individuals. As an example, it was observed that some studentslearn faster than others or learn and remember more than others. To explain these
differences, a theory on intelligence that is related to learning was developed. This theoryexplains that students who have the above qualities are more intelligent than others.
They also perform better in examinations. Tests were developed to measure how much
intelligence a person has. The construct or theory explains that students who score
more tend to do better in school and other learning environment than those who haveless of it. Those with high scores are said to be more intelligent than others.
Research involving a construct are valid only to the extent that the instrument usedactually measures the intended construct. Determining construct validity is not the easy
and it usually involves gathering a number of pieces of evidence to demonstrate thevalidity. For example, if some psychologists construct an intelligence test instrument
called Smart Ali, the validity of the instrument should be subjected to various tests.First of all, we could see whether students who scored high on the Smart Ali test learned
faster, learn more and remember more than low scorers. We could correlate the scoresof the students at the beginning of the school year and at the end of the school year. We
could also correlate performance on the Smart Ali test with other well established
intelligence tests to observe whether the correlations were high. Scholars should alsobe invited to examine whether the test items represent the typical topics in the field of
7/31/2019 HBEF4106-CHP5
16/22
OUM 73
TOPIC 5 RESEARCH METHODOLOGY
intelligence. In addition to this confirmatory evidence, we need to seek disconfirmatoryvalidity information. For example, we do not expect this intelligence test to be highly
correlated to self-confidence or anexiety tests.
(d) Definition of ReliabilityReliability is a synonym for consistency and replicability over time, over instruments and
groups of respondents (Creswell, 2002). It is about precision and accuracy. A reliable
research must show that if it were to be done on a similar group of respondents in asimilar context, than similar results would be obtained. There are three types of reliability,namely; stability, equivalence and internal consistency.
(i) Reliability as Stability
Reliability as stability is a measure of consistency over time and over similarsamples. It means that a reliable instrument will achieve similar data from similar
respondents over time. In the experimental model, for example, this would meanthat a test and then a retest were taken within an appropriate time span, similar
results would be obtained. Correlation coefficients can be calculated for the reliability
of the pre-and post-tests.
(ii) Reliability as EquivalenceReliability could be achieved through using equivalent or alternative forms of a test
or data gathering instrument. If the equivalent forms for the test is devised andproduces similar results, then the instrument could be said as having this form of
reliability.
(iii) Reliability as Internal ConsistencyInternal consistency demands that the instrument be run only once unlike the other
two mentioned earlier (stability & equivalence). Internal consistency could bedemonstrated by running the test once only through the split-half method. Let us
suppose that a test is to be administered to a group of students. Here, the testitems are divided into two halves matching in terms of difficulty and content. If the
test is to show split-half reliability, then the marks obtained in each half should becorrelated highly to the other half.
Figure 5.4: Three types of reliability
Reliability as
Stability
Reliability as
Equivalence
Reliability as
Internal
Consistency
RELIABILITY
7/31/2019 HBEF4106-CHP5
17/22
74 OUM
TOPIC 5RESEARCH METHODOLOGY
1. Why are the reliability and validity of research instruments vital?
2. What is the difference between reliability as equivalence and reliability
as internal consistency?
3. Explain the difference between predictive validity and concurrent validity.
5.3.3 Pilot Study
A pilot study is a pre-research study undertaken on similiar set of respondents: in terms of
location, nature of setting and profile of respondents (age, gender, etc). This is done to ensure
the validity of instruments and to resolve possible problems before the actual research isdone.
Pre-testing of the instruments to be used in a research is of vital importance. A pilot study hasseveral functions, mainly to increase the reliability, validity and practicability of the questionnaire
(Oppenheim, 1992; Morrison, 1993; Wilson and McLean, 1994). Some of the objectives of
piloting the questionnaire are (Cohen, Manion & Morrison, 2000):
To check the clarity of the questionnaires including instructions and layout.
To gain feedback on the validity of the questionnaire items.
To eliminate ambiguities.
To gain feedback on the type of question and its format.
To gain feedback on response categories for closed questions.
To gain feedback on the attractiveness of the questionnaire.
To gain feedback on the layout of the questionnaire.
To check the time taken to complete the questionnaire.
To identify redundant questions.
To identify commonly misunderstood items..
To try out the coding system for data analysis.
After the pilot study, you can refine the questionnaire based on the responses of the
respondents in the pilot study. This refinement would have eliminated all the shortcomings ofthe instruments. Then, you can proceed with the actual data collection for the research.
5.4 DATA ANALYSIS
Data analysis becomes very crucial after all the required data were collected. Data analysis
has to be done before we could make any inference. Numbers have to be crunched usingstatistical analysis and statistical values have to be interpreted. Quantitative and qualitativedata have to be treated differently using guidelines provided in this module. Let us look at
more information regarding data analysis.
7/31/2019 HBEF4106-CHP5
18/22
OUM 75
TOPIC 5 RESEARCH METHODOLOGY
5.4.1 Measuring Research Variables
Data are collected from the respondents through a questionnaire or from test results. The
questionnaire items were generated from research variables. In educational research, thetypical variables are as follows:
Leadership styles of school principals.
Academic achievement of students.
Interpersonal communication skills.
Attitudes toward school.
Teaching styles of teachers.
Socio-economic background of parents.
Time spent on studying and doing home-work. Reading habits of students.
These variables are the main ideas that researchers seek to address in their studies. A variableis a characteristic or attribute of an individual or an organisation that you can measure orobserve. It also varies among individuals or organisations. When we talk about a variable
as a characteristic we mean personal aspects about the individuals, such as their age andincome level. An attribute represents how an individual in an organisation, feels, thinks or
behaves. When we discuss about individuals having self-confidence or displaying leadershiptraits, we are talking about their attributes.
Researchers measure these characteristics and attributes. Measurement means recordinginformation from individuals through:
Observation; or
asking questions.
In either case, student scores vary. The variables vary means the scores will have differentvalues depending on the respondents characteristics. For example:
SPM Examination results vary by nine possible scores: 1 being the highest, which isequivalent to a distinction, and 9 being the lowest which is equivalent to a failure.
Gender varies by two possible scores: male = 1 and female = 2.
In a quantitative study, most researchers used to record the scores by using the Likert Scale
on a 5 point scale. For example, we may state a statement: The teacher always usesexamples to explain difficult concepts. The 5 point scale used would most probably be:
5 = strongly agree
4 = agree
3 = moderately agree
2 = disagree
1 = strongly disagree
7/31/2019 HBEF4106-CHP5
19/22
76 OUM
TOPIC 5RESEARCH METHODOLOGY
Figure 5.5: The Qualitative Process of Data Analysis
Code the Text forDescription to be Usedin the Research Report
Code the Text forThemes to be Usedin the Research Report
SimultaneousIterative
text segments andCode the Data(i.e., locates
assigns a code label to them)
Read Through Data
(i.e., obtains a generalsense of material)
Prepare Data for Analysis(i.e., transcribes fieldnotes)
transcriptions, or
Collect the Datai.e., a text file, such as fieldnotes,optically scanned materials)
The respondents would choose one of the values to state their agreement or disagreementto the given statement. The high scores denote that there is a strong agreement to the
statement and vice versa. When a research variable in a construct it is usually measured by
taking the sum or average of scores of all related items in the questionnaire.
On the other hand, a qualitative study requires a different method of description. Instead ofutilising numbers to represent some ideas, the researcher might have to write lengthy
descriptive narrations, which later on have to be summarised and classified under differentselected variables. The analysis of qualitative and quantitative data will be explained in moredetails later.
5.4.2 Analysing Qualitative Data
Qualitative researchers use to analyse data by using the following steps:
Preparing and organising the data for analysis.
Describing and developing themes from the data.
Reporting the findings.
Interpreting the findings.
Validating the accuracy and credibility of the findings.
In a summary form, the qualitative process of data analysis was vividly presented by Creswell
(2002:258).
7/31/2019 HBEF4106-CHP5
20/22
OUM 77
TOPIC 5 RESEARCH METHODOLOGY
Figure 5.5 summarises the general process of analysing qualitative data. Qualitative
researchers collect data first and then prepare it for data analysis by transcribing field notes.Your next step is to read through the data to have a general sense of the material. Since there
is a lot of data, you have to code the data for easy referral later on. Qualitative researchersneed to analyse their data by reading through several times and conduct an analysis each
time. In doing so the researcher would have sifted through a lot of information which could beuseful in writing the final report. A useful conceptualisation is to think about the process as
both simultaneous and iterative, which means going back and forth a few times. Therefore,there is not one single way to analyse qualitative data. It is a process in which you try to make
sense of the information.
5.4.3 Analysing Quantitative Data
Before quantitative data could be analysed, a researcher would have to prepare and organise
the data for analysis. The following steps would be useful as a guide:
(a) First of all, you have to assemble all data by transforming it into numeric score or value
to each response category for each item. Before conducting an analysis of scores, youshould consider what types of scores to be used. For example, a single item score isan individual score to each item for each participant in your study. The next one is
summed scores which is the scores of an individual added over several items that
measure the same variable. The other is the net ordifference scores which are scoresin a quantitative study that represent a difference or a change for each individual.
(b) The next step is selecting a statistical programme. With the presence of the computer
age, selecting a suitable programme for analysing quantitative data is really not a big
problem. If you are a student in a university, use the programme that is available at theuniversity.
(c) The third step is inputting data. This is a process when you transfer the data from theresponses on instruments to a computer file to be used for analysis.
(d) Exploring and Descriptively Analysing the Data is the fourth step. By doing this, you
engage in data analysis. This procedure consists of:
exploring and describing the data; and
conducting statistical tests on the data.
By exploring the data, you examine for errors and missing data so that decisions can be
made about the missing data.(e) Cleaning and Accounting for Missing Data
You must clean the data because of errors in scoring that you make when inputting thedata. This is a process of visually inspecting the data for scores outside the accepted
range. This will alert you on the errors which were made especially after a tiring process
of inputting data. Sometimes data to individual items are missing. Missing data isinformation that is not supplied by participants to specific items. Missing data will reduce
the statistical options available for your statistical analysis. Therefore, how would youhandle this problem? The obvious approach is to avoid having missing data by providingitems such that respondents are willing and able to answer. If your database contains
missing data, there are several technical procedures that can help you. One procedure
is eliminating respondents with missing scores from the data analysis, but this practicemay reduce the number of overall respondents. Some researchers recommend
7/31/2019 HBEF4106-CHP5
21/22
78 OUM
TOPIC 5RESEARCH METHODOLOGY
1. Why should you conduct a pilot study?
2. What would you do with missing data?
3. In qualitative research, how would you transcribe field notes?
4. What are the skills needed in analysing data from quantitative research?
substituting values for the missing cases. The researcher can have the computerprogramme to substitute a value for each missing score, such as the group mean (George
& Mallery, 2001).
(f) Calculating Descriptive and Inferential StatisticsDescriptive statistics present information that helps you in assessing the general trend
in the data and in answering descriptive research questions in the studies. Theseobjectives can be achieved by running the frequency distribution and its statistics. To
understand the statistics, you need to have some knowledge about them. These statistics
are the numbers derived from formulas to measure aspects of a set of data such as:
Measures of central tendency which are summary numbers that represent a single
value in a distribution of scores (Vogt, 1999). They are expressed as an average
score (the mean), the middle of a set of scores (the median) or the most frequentlyoccurring (the mode).
Measures of variability indicate the spread of the scores in a distribution (Creswell,2002). Variance, standard deviation, and range all indicate the amount of variability ofscores.
Measures of relative standing are statistics that describe one score relative to a
group of scores (Creswell, 2002). The percentile rank and z-score are two frequentlyused statistics.
The other basic types of statistics are inferential statistics which are briefly explained
here.
Inferential statistics refer to statistics that enable a researcher to draw conclusions,inferences, or generalisations from a sample to a population. Inferential statistics are
concerned with determining how likely it is that results based on a sample are similiar tothese obtained from the population (Gay & Airasian, 2003). The question that guides
inferential statistics is whether expected differences are real, significant ones or only the
result of sampling errors. Inferences concerning populations provide only probabilitystatements where you are never certain when making an inference about a population.
In inferential statistics, there are many other factors that need to be learnt. A course on
statistics would provide sufficient knowledge regarding inferential statistics. Other termsthat are important for inferential statistics are standard error, the null hypothesis, tests of
significance, the t-test, multiple regressions, chi square, and other more advancedstatistical analysis which can be found in statistics texts.
7/31/2019 HBEF4106-CHP5
22/22
TOPIC 5 RESEARCH METHODOLOGY
SUMMARY
The topic on Research Methodology gives a brief explanation on the process of how to
implement a research successfully. This topic needs to be read with understanding withother topics written in this module. Some of the terms and procedures would be better
understood through reading other related topics. Questions are also given to help youunderstand the concepts and the process of the methodology of research. In order for you to
comprehend this topic well, you need to read other topics because all topics are inter-relatedwith each other.