Second Language Research

Quantitative, Qualitative and Mixed Methods Research

Synthetic (holistic) vs. analytic (constituent) approaches -There are two ways to approach the study of a field with many component

parts:

1-Synthetic: To grasp the whole or large parts to get a clear understanding of their possible interrelationships. It emphasizes the interdependence of the

parts .

2-Analytic: To identify the small parts for close study to fit the small pieces into a coherent picture. It focuses on the role of the constituent parts.

-These two approaches should be seen as complementing each other.

Heuristic (Inductive) and Deductive Objectives

1-Heuristic: It is hypothesis-generating, data-driven, has no preconceptions, and results in description or hypotheses.

2-Deductive: It is hypothesis-testing, hypothesis-driven, makes predictions, and results in theory.

-This distinction signifies more than using figures than non-quantitative data. It entails many things including ideological orientation, data collection methods, nature of the data collected, and data analysis methods.

-Quantitative research involves data collection procedures that lead primarily to

numerical data which are then analyzed by statistical methods .

-Qualitative research involves data collection procedures that lead primarily to open-ended, non-numerical data which are then analyzed by non-statistical methods.

-Mixed methods research involves different combinations of quantitative and qualitative research either at data collection or at the analysis levels .

Quantitative, Qualitative and Mixed Methods Research

Three Positions Regarding Qual-Quan Difference

1-Purist: It argues that these methodologies are mutually exclusive.

2-Situationalist: It accepts the strength of both traditions, but still represents an either/or approach.

3-Pragmatist: It indicates some sort of an integration of the two methods can be beneficial to corroborate (convergence in findings), elaborate (richness and detail), and initiate (offer new interpretations) from the other method.

Quantitative Research vs. Qualitative Research

-There are many approaches to dealing with research. This dichotomy is simplistic and it is best thought of as a continuum of research types.

Quantitative Research vs. Qualitative Research

Quantitative Research

-Obtrusive involving controlled measurement

-Objective and removed from the data

-Verification-oriented, confirmatory

-Outcome-oriented

-Reliable involving hard and replicable data

-Generalizeable

-Assuming a stable reality

Qualitative Research

-Naturalistic and controlled observation

-Subjective

-Discovery-oriented

-Process-oriented

-Soft data

-Ungeneralizeable

-Assuming a dynamic reality

Quantitative Research

Brief historical overview:

-Inspired by progress of natural sciences and the scientific method in 19 th century.

-It involves exploration in an objective manner including: 1) observing a phenomenon; 2) generating an initial hypothesis; and 3) testing the hypothesis by collecting and analyzing empirical data.

-It was closely related to numerical values and statistics; Rutherford says any knowledge that cannot measure numerically is poor knowledge.

-It was accompanied by major developments in statistics in the first half of the 20th century .

Quantitative Research

Major characteristics:

1-Using numbers: The most important feature. But numbers do not represent anything without contextual backing.

2-A priori categorization: Numerical categories should be pre-specified to ease understanding and expedite data analysis .

3-Variable rather than cases: It is not interested in individuals but the common features of groups. Therefore it is centered around variables that capture these

features .

4-Statistics and the language of statistics: A salient feature.

5 -Standardized procedures to assess objective reality: It is deprived of human variability and bias through a standard and rigorous set of procedures.

6-Quest for generalizeability and universal laws: All the steps boil down to generalizeable and universal laws .

Quantitative Research

Strengths:

It is systematic, rigorous, focused, tightly controlled, involving precise measurement and producing reliable and replicable data. It is quick and economical.

Weaknesses:

It has little limited general exploratory capacity because it irons out individual differences by working with concepts of averages.

Qualitative Research

Brief historical overview:

-It has been around for about a century in the social sciences .

-However, after world war II, quan methods produced substantial advances and qual methods were relegated to preliminary exploratory work .

Qualitative Research

Major characteristics:

1-Emergent research design: No aspect of its design is tightly prefigured and the study is kept open and fluid to respond in a flexible way to new details that may emerge during the process of investigation.

2-The nature of qualitative data: It works a wide range of data including interviews, texts, images, etc. Data are transferred into a textual form because data analysis is mostly done with words.

3-The characteristics of the research setting: Social phenomena occur naturally; hence, the setting is natural with no attempts to manipulate it .

4-Insider meaning: It is concerned with subjective opinions, experiences and feelings of individuals to explore the participants’ views of the situation as the insider perspective.

5-Small sample size: It is very labor-intensive and encompasses smaller samples.

6-Interpretive analysis: Research outcome is the product of the researcher’s subjective interpretation of the data, allowing several interpretations for each

dataset .

Qualitative Research

Strengths:

1-Exploratory nature 2-Making sense of complexity 3-Answering why questions 4-Broadening our understanding

5-Longitudinal examination of dynamic phenomena6-Flexibility when things go wrong

Weaknesses:

1-Sample size and generalizeability 2-Researcher role3-Lack of methodological rigor 4-Too complex or too narrow theories

5-Time consuming and labor-intensive

Mixed Methods Research

Brief historical overview:

-Using multiple data types also dates back the beginning of 20th the century.

-However, it began to take on seriously after the second half of 20th the century.

-The real breakthrough occurred in the 1970s with the introduction of the concept of ‘triangulation’ into the social sciences. It was borrowed from naval navigation. It is synonymous with combining a variety of data sources to scrutinize the same phenomenon, and validating hypotheses through multiple

methods .

Mixed Methods Research

Strengths :

1-Increasing the strengths while eliminating the weaknesses.2-Multi-level analysis of complex issues.

3-Improved validity.4-Reaching multiple audiences.

Weaknesses:

-It can be counterproductive if it is not conducted logically.

Quantitative Research Methods

1-Associatinal: To determine a relationship between two variables and its strength. It is not concerned with causation, only co-occurrence.This is usually tested statistically through correlations.

2-Experimental: The researcher deliberately manipulates one or more variables to determine the effect on another variable .

Experimental Design

-Descriptive research does not make strong conclusions about the interrelationship of the variables. It just narrates what happened or what is

happening. Through this method, one cannot make cause-effect relationships .

-Experimental research is both the most demanding and the most productive method.

-Its of two types: 1) true experimental and 2) quasi-experimental .

True Experimental Design

-It is the strongest and most rigorous method in education. All the following requirements must be met.

Characteristics:1-Randomization: Subjects are selected randomly to minimize bias exercised

over one particular subject. Every member of a given population has an equal chance of being included in the experiment. It helps the researcher to select a representative sample.

2-Experimental and control groups: There should be a causal relationship. The exp group receives the cause (independent) variable called treatment. On the other hand, the con group receives the effect (dependent ) variable called placebo.

True Experimental Design

3-Pretest: Any potential differences between the groups on the variable under investigation must be avoided. A pretest screens these initial differences.

4-Posttest: It detects the effects of the treatment on the dependent variable between the groups. This effect should exist for the exp group .

True Experimental Design

Validity of research: Findings of research should be verifiable (replication) and applicable (generalizeability). If findings enjoy these two qualities, they are said to be valid.

1-Internal validity: The extent to which the outcome is due to the manipulation imposed by the researcher not other factors. The researcher should endeavor to control as many variables as possible. It is exclusive to exp design.

2-External validity: The extent to which the outcome would apply to other similar situations .

-There should be a trade-off between them. As the researcher intends to increase one, the other will automatically decrease .

True Experimental Design

Factors affecting internal validity:

1-History effect: Whatever happens to the subjects outside the exp environment. Random selection can alleviate this factor.

2-Maturation effect: Any process that involves systematic changes over time, regardless of specific events .

3-Testing effect: The awareness about the experiment due to the pretest.

4-Selection effect: The manner in which the participants are selected, e.g. self-selection. It can be alleviated by matching

True Experimental Design

5-Mortality (or attrition) effect: It is caused by the loss of subjects during the exp, particularly in longitudinal studies .

6-Hawthorne effect: Participants might be pleased at being included in a study which can impact more on the results than any thing that actually occurs in the research.

7-Halo effect: It is due to the tendency among humans to respond positively to a person they like or the other way around.

Quasi-Experimental Design

-When one or more of the requirements are not met or deliberately ignored. The researcher attempts to compensate for the violation of certain principals. The most common types are:

1-One-shot case design: No con group, one treatment for a given period of time, and a posttest at the end. Results are neither valid nor generalizeable because of many uncontrolled variables. (X T)

2-One-group pretest posttest design : Similar to the previous one; however, a pretest is administered before the instruction. (T1 X T2)

3-Intact group design : Most teachers conduct it because students cannot be assigned randomly. They are place based on some criteria like their exam scores. Two classes, 1 exp and 1 con. Exp receives treatment and con receives a placebo. Finally, a posttest is given. (G1 X T) and (G2 O T)

Quasi-Experimental Design

4-Time-series design : No con group, several pretests are administered, followed by a treatment, and then several posttests. (T1 T2 T3 X T4 T5 T6)

5-Equivalent time-series design : The treatment is introduced and reintroduced between every other pretest and posttest. (T1 X T2/T3 O T4/etc.)

6-Repeated measures (or within-group) design: All tasks or treatments are given to different individuals in different orders. The basic characteristic is that multiple measurements come for each participant.

7-Factorial design: It involves more than one independent variable and can occur with or without randomization .

Sampling

-The questions to ask at the beginning of an investigation are :1-How many people do I need to include in my study? or, how large should

my sample be?

2-What sort of people shall I select? or, who shall my sample consist of?

-Sampling issues should be considered at the commencement of the study because they can affect the necessary initial arrangements, timing, scheduling of the project, and the various costs involved. Three concepts lie at the heart of sampling:

1-Sample: It is the group of participants whom the researcher actually examines .

Sampling

2-Population: It is the group of people about whom the study is .

3-Represantativeness: A good sample is very similar to the target population in its most important general characteristics (e.g. age, gender, ethnicity, educational background, social class) and specific features of the variable under study (e.g. L2 background, amount and type of instruction received). That is, the sample is a subset of the population and should be representative of it because the strength of the conclusions to be drawn from the results

depends on it .

Sampling Procedures

-There are generally two types of sampling procedures:

1-Probability sampling: It is scientifically sound and involves complex and expensive procedures. Here, each member of the population has a known non-zero probability of being selected. Its advantage is that sampling error can be calculated. It is the degree to which a sample might differ from the population .

2-Non-probability sampling: It tries to achieve a trade-off, which is a reasonably representative sample using resources that are within the means of the

ordinary researcher. Here, the sampling error remains unknown .

Probability Sampling Procedures

-The choice of any of these methods depends on the nature of the research problem, data collection methods, availability of a good sampling frame (all members involved), the desired level of accuracy in the sample, and the

resources .

1-Simple random sampling: All members have an equal and independent chance of being included in the sample. It is based on probability and chance, not in the sense of ‘haphazard’. Purpose is to minimize the effects of any extraneous or subjective factors. Used with small and up-to-date populations.

2-Systematic sampling: It is similar but involves a systematic rule of drawing a sample by taking every nth (3rd, 5th, 7th, etc.) case from a list of a population.

The list should not have any hidden order .

Probability Sampling Procedures

3-Stratified random sampling: It is for research with a specific focus. It is a combination of randomization and categorization. All the people in the population are divided into strata (groups or categories). Within each stratum, a simple random sample of a proportionate size is selected, e.g. to include equal numbers of both males and females.

4-Cluster sampling: It is called cluster because it refers to a group of individuals who are naturally together. The target population is divided into clusters or sections and the number of units selected at random from a given cluster is proportional to the total number of units in the cluster. It is used with widely dispersed populations, e.g. selecting some cities, then some districts,

then some schools, and finally some classes .

Non-Probability Sampling Procedures

-Most actual research in applied linguistics employs non-probability samples. These methods are cheaper and more feasible and are used with widely dispersed populations.

1-Convenience (opportunity) sampling: The sample comprises subjects who are simply available in a convenient way. There is no randomness and the likelihood of bias is high.

2-Purposive (judgment) sampling: This is an extension of convenience sampling. Sample is subjectively selected based on judgment. However, it should be representative of the population .

3-Quota sampling: It is the equivalent of stratified sampling. First the strata and their proportions are identified. Then the required numbers of participants are selected conveniently or purposefully, not randomly .

Non-Probability Sampling Procedures

4-dimensional sampling: It is a variation of quota sampling. At least one representative of every combination of the various parameters in the sampling frame must be included in the sample.

5-Snowball sampling: It is used when the desired sample characteristic is rare. It involves a ‘chain reaction’ where You initially contact some potential respondents and then ask them whether they know anybody with the same characteristics.

6-Self-selection: Respondents themselves decide to take part in your study .

Sample Size

-The question of the sample size actually means ‘how small a sample I can get away with’. The truth is ‘the larger, the better’. But this is not practically feasible. To have an optimal sample size, You should consider several broad guidelines:

1-Rules of thumb: A range of between one to ten per cent of the population within a minimum of roughly 100 participants as the magic fraction. However, for correlational research (at least 30 participants, comparative and experimental procedures (at least 15 participants in each group), factor

analytic and other multivariate procedures (at least 100 participants) .

2-Statistical consideration: Subgroups which behave differently (e.g. girls and boys) should be identified. Sample size should be set so that the minimum

size applies to the smallest subgroup in the sample .

Sample Size

3-Safety margin: when setting the final sample size, it is advisable to leave a decent margin to provide for the unforeseen or unplanned circumstances.

4-Reverse approach: Because statistical significance depends on the sample size, your principle concern should be to sample enough learners for the expected results. Therefore, first you approximate the expected magnitude of the expected results and then determine the sample size that is necessary to detect this effect, e.g. at a p<0.05 significance level, an expected correlation of

0.4 requires at least 25 participants .

Qualitative Research Methods

1-Ethnographies: They aim to describe and interpret the cultural and communicative behavior of a group. They also give an emically oriented description of the cultural practices of individuals, using categories relevant to

a particular group and cultural system.

Another feature is the holistic approach to describing a particular pattern in relation to a whole system of patterns. They focus on the group and stress the larger sociocultural context of the study, with the categories and codes being derived from the data themselves rather than being imposed from the outside.

These methods usually overlap with other qualitative methods .

Advantages:The research questions can be dynamic , subject to constant revision , and

refined as the research continues to uncover new knowledge .

They are particularly valuable when not enough is known about the context to establish narrowly defined questions or develop formal hypotheses .

Caveats:They involve intensive research over a long period of time. The researcher

must be committed to long-term data collection, detailed and continuous record keeping, and repeated and careful analysis of data. They might also create potential conflicts between a researcher’s role as an observer and a

participant .

2-Case studies: They also aim to provide a holistic description of a problem within a specific population or setting. However, they tend to give detailed

decryptions of specific learners within their learning settings (cases) .

Case studies are usually associated with a longitudinal approach, in which observations of the phenomena are made at periodic intervals for an extended

period of time .

Advantages :They allow the researcher to focus on the individual in a way that is rarely

possible in group research. They have the potential for rich contextualization and provide insights into the complexities of particular cases in their particular

contexts .

Caveats :The researcher must be careful about the generalizations as participants who

are not randomly chosen are a few. To address this concern, findings combined from multiple case studies must be used to draw firmer conclusions .

3-Interviews: they are often associated with survey-based research, as well as being a technique used by many qualitative studies. They are of three types:

Structured,Unstructured, and

Semi-structured

Advantages :They can allow researchers to investigate phenomena that are not directly

observable (perceptions). Because they are interactive, researchers can elicit additional data if initial questions are vague, off-topic or not specific enough.

Caveats:They may involve selective recall, self-delusion, memory loss from the

respondent and the researcher’s subjectivity. Multiple interviews can address the problem .

Interviewing is also a skill that requires expertise.

They also run the risk of halo effect .

4-Observations: They involve the researcher immersing in a research setting, and systematically observing dimensions of that setting, interactions,

relationships, actions, events and so on .The aim is to provide careful descriptions without influencing the events .

The data are usually collected through some combination of field notes and audio and visual recordings. They are also:

Structured ,Unstructured, and

Semi-structured

Advantages:They allow the collection of large amounts of rich data within a particular

context .The researcher can gain a deeper and multilayered understanding over time.

Caveats:They do not allow access to the participants’ motivation for their behaviors

and actions. Therefore, they may be most useful with other methods .

They also run the risk of Hawthorne effect .

5-Diaries/Journals: They include verbal protocols and other introspective methods about the learners’ internal processes and thoughts .

They are also called L2 journals or learner autobiographies .

Advantages :They can yield insights into the learning process that may be inaccessible from

the researcher’s perspective alone because learners usually record their perceptions or impressions about learning .

Caveats:Keeping a diary requires commitment on the part of the participants to

carefully and regularly provide detailed accounts of their thoughts .Moreover, these insights constitute a highly specialized population, rendering

it difficult to extend them to other contexts.

Due to the lack of structure of diary entries, data analysis can become a complex affair, making it more difficult for the researcher to find and validate patterns

in the data .