The challenge of conducting a quantitative business research: Analysis of issues with survey design, sampling, validity, and reliability

Dr. Komlan Joel Adzeh

Ph.D. in Organization and Management Alumnus,

Capella University – School of Business and Technology Capella Tower, 225 South Sixth Street, Ninth Floor, Minneapolis, MN 55402,

Email: kadzeh@capellauniversitu.edu

Abstract:

The aim of this literature research article was to provide insight into the challenges that

quantitative researchers face and to analyze the specific issues with surveys. Several studies have

been synthesized and analyzed. Building upon the current knowledge of the field, this article

highlighted some specific areas of concern, including sampling, validity, and reliability. While

survey research is a method of choice among quantitative scholars and practitioners, the rigor

with which it is conducted within the business community remains a topic of interest. This article

indicates a direction to future research.

Keywords: Business Research, Quantitative Methodology, Survey Design, Sampling,

Validity, Reliability

1. Introduction

Businesses across the world collect information to gain deep insight about market trends,

competitors, consumer behaviors, potential opportunity for growth, and any kind of threats that

could inhibit their performance. Therefore, survey research has become a growing field with

20,000 job openings in the United States alone and an estimated growth rate of 24% over the

next decade, according to the Bureau of Labor Statistics, U.S. Department of Labor (2012).

As the demand for business research continues to grow, surveys have been extremely

popular among quantitative researchers and the design of choice to investigate several issues,

particularly in marketing and customer behavior. Besides, a survey design involves collecting

data from a representative sample with the intent to generalize the findings to a larger population

(Barlett, Kotrlik, & Higgins, 2001). In addition, several researchers argued that survey design is

both economical and capable of yielding a quick turnaround in the data collection process

(Couper & Miller 2008; Frippiat & Marquis, 2010). However, this process is not always

conducted with the same rigor in business as compared with the academia.

Although surveys play a key role in business quantitative research, they are conducted in

the real-world environments, which cannot be controlled or manipulated. Further, the decline in

survey response over the last decades represents the greatest difficulty researchers have ever

faced (Schmeets, 2010). This paper discusses the challenges related to the philosophical

foundation of survey research and analyzes issues involving sampling, validity, and reliability.

2. Challenges Related to the Philosophical Foundation of Survey Research

Survey research is grounded in the positivist paradigm according to which reality is

objective, unbiased, and completely independent of both the researcher and the subject

(Bielefeld, 2006; Johnson & Onwuegbuzie, 2004). According to Firestone’s (1987) seminal

work, the premise of this world view is the acceptance that social facts are objectively

measurable, and they are free from the researcher. Carr (1994) argued that this detachment

ensures the neutrality of the study and prevents biases.

Although this world view explains the reason why findings were historically attributed to

quantitative research, it was also criticized. First, Walsham (2006) contended that there is no

such thing as objective fact that researchers can learn. Instead, reality is socially constructed

through the meanings, interpretations, and interactions of individuals with events. Second, Guba

and Lincoln (1994) argued that the positivist paradigm removes meanings and interpretations

from data in order to quantify phenomena. Moreover, Clark (1998) claimed that different

philosophical assumptions, and not just only one, have shaped the quantitative research.

Therefore, he argued that it would not be appropriate to consider the quantitative research as

rooted in a single paradigm.

Not too long ago, Amaratunga, Baldry, Sarshar, and Newton (2002) questioned the

appropriateness of quantitative survey method and argued that it aims at taking a snapshot look

at situations, which can be affected by temporal changes. Probably, the fiercest criticism was that

the quantitative method neither recognizes the importance of resource constraints nor the

variability in human behaviors (Eldabi, Irani, Paul, & Love, 2002). Similarly, Johnson and

Onwuegbuzie (2004) agreed that the knowledge that quantitative analysis produces may be too

abstract and may not reflect the specificity of the contexts, situations, or understandings of local

constituencies. Because of the ontological dichotomy in conceptualizing the quantitative research

method, interpretivism and post positivism have emerged as alternative world views of

conducting social research.

Furthermore, Harrison, Hill, and Leitch (2010) contended that interpretive researchers

aim to understand true meanings of the social world by embracing its complexity. Whereas, post

positivists considered that there is only a probable reality, and no one can certainly comprehend

the nature of the truth (Ponterotto, 2005). Both world views have changed the landscape of social

research and have an implication for the business management. Recent studies suggested

blending different paradigms altogether into one study (Onwuegbuzie, Turner, & Johnson, 2007;

Terrell, 2012). While much progress has been achieved in studying social phenomena,

quantitative business scholars continue to be challenged with issues involving sampling, validity,

and reliability.

3. Challenges Related to Sampling

Survey researchers expect to obtain a minimum sample size to produce results that are

statistically reliable and generalizable as an inadequate sample can undermine the accuracy of

the findings (Barlett, Kotrlik, & Higgins, 2001). However, several challenges present a threat to

the precision of a survey (Grove, 2006). Chief among them is the non-response. Even though, a

stream of researchers contended that a change in non-response rate does not necessary change

survey estimates, it is generally recommended reducing non response rates as much as possible

(Curtin, Presser, & Singer 2000; Keeter, Miller, Kohut, Groves, & Presser, 2000).

Methods used to reduce non responses include advance letter (Hox, 2007), incentives, or

longer periods of data collection (Grove, 2006). While these techniques are generally successful

in reducing non response threats, they are costly to implement. Additionally, the limitation of

resources obliges survey researchers to turn to nonrandom sampling instead of devoting their

energy to try to obtain appropriate sample sizes or responses. As Carr (1994) explained, random

sampling is time-consuming and very often, researchers turn to opportunistic samples, which

they can obtain easily. The downside to this approach is that, it compromises the generalization

from the findings (Onwuegbuzie & Johnson, 2006).

4. Challenges Related to Validity

Quantitative survey researchers share the excitement that the conclusions they would

draw from analyzing data would be credible, objective, and reliable. At least, that is what they

expect. However, a variety of threats can occur during any stage within the research process,

which can compromise the validity to the study. Onwuegbuzie and Johnson (2006) compiled

these threats into three categories relative to the content, criterion, and construct validity of the

survey instrument.

4.1. Content Validity

Content validity refers to the degree to which survey instrument items are relevant and

representative of the construct they intended to measure (Rossiter, 2008). It includes face

validity, item validity, and sampling validity (Onwuegbuzie & Johnson, 2006). Face validity, in

particular, describes how well a survey instrument seems to measure what it is designed to

measure. Further, Collins (2003) explained that face validity referred to an informal look to the

instrument in order to ensure that it is appropriately designed to obtain the right information from

the population of interest. Therefore, survey researchers use field pretest to inquire feedback on

areas of weakness that may need improvement. Although suggestions from field pretest may

contribute to improve the quality of the questionnaire, Betts (2011) argued that researchers

should not solely rely on them because further refinement may be necessary before the

deployment of the instrument.

4.2. Criterion Validity

Criterion validity requires thoroughly testing the instrument by comparing the values

inferred from the test to the criterion values actually observed or by correlating those values with

other measures of similar behavior (Rejas, Monfort, Campillo, Ruiz, Pardo, & Soto, 2009).

Specifically, criterion validity includes concurrent validity and predictive validity (Onwuegbuzie

& Johnson, 2006). Further, survey researchers seek expert opinions on the appearance,

relevance, and the representativeness of all components of the survey instrument in order to

ascertain that the questionnaire measures the right construct (Turocy, 2002). This method

requires asking the exact questions, identifying the true criterion, having appropriate statistical

expertise, and being able to engage skilled reviewers. This process can be a challenge

sometimes. One notorious example was about Coca-Cola Company, which, in 1985, neglected to

ask consumers, whether they preferred the brand-new Coke flavor to the old one, which could

have been a perfect criteria for benchmarking the preferences before introducing the “New

Coke” (Shuttleworth, 2009). The consequence was what News and Media reported at the time as

one of the biggest marketing fiascoes in history simply because Coca-Cola Company neglected

to ask the central question of Coke users, “Do you want a new Coke?” (Ross, 2005).

4.3. Construct Validity

Construct validity is the extent to which survey instrument is free from measurement

errors (O'Leary-Kelly & Vokurka, 1998), and it addresses the representativeness of the scale, the

appropriateness of the items, and the carefulness of their articulation (Burton & Mazerolle, 2011;

Parasuraman, Zeithaml, & Berry, 1988). In addition, Cronbach and Meehl (1955) asserted in

their seminal work that, construct validation is needed whenever a researcher considers that the

proposed survey instrument represents a specific construct and to which particular meanings are

attached. In this case, it is another challenge to ensure that potential survey participants capture

the exact meaning that the researcher attaches to the construct. Therefore, researchers may need

to conduct field pretest several times to improve the construct validation of the survey.

4.4. External and Internal Validity

According to Ihantola and Kihn (2011), external validity refers to the generalization from

the study to other samples, other times, and beyond the initial setting. More importantly, they

argued that external validity determines whether researchers can draw general conclusions based

up on the model they proposed and the data they collected. Further, Ihantola and Kihn (2011)

explained that three typical issues might jeopardize the external validity of a quantitative

investigation: (a) inaccessibility to the population or having a sample that is not representative;

(b) time validity or whether conclusions can be generalized to other periods of time; (c) and

environmental validity or whether conclusions can still hold true across settings.

While Ihantola and Kihn (2011) defined internal validity as the measurement of

variations between dependent and independent variables, which are not due to other confounding

factors, Onwuegbuzie and Johnson (2006) emphasized that even skillful researchers can still be

challenged by numerous threats that occur at various stages of a research process. Building on

previous works, Onwuegbuzie and Johnson (2006) identified several threats to both internal and

external validity at the stage of research design and data collection. Issues such as Type I and

Type II Errors, violated assumptions, misspecification error, multicollinearity, distorted graphics,

confirmation bias, and causal error were identified as serious threats to both external and internal

validity of a survey.

Therefore, it becomes extremely important for survey researchers to minimize the effect of these

threats to the quality of their studies and specify the extent to which their estimates are reliable.

5. Challenges Related to Reliability

Thompson and Vacha-Haase (2011) argued that reliability helps to determine whether the

exact study can be replicated under the same condition, and the scale can produce consistent

results of measurements. In his seminal work, Churchill (1979) recommended the use of

Cronbach’s alpha coefficient as the first measure researchers should calculate in order to assess

the quality of their data. He assumed that if all the items in a particular measure are measuring

the domain of a single construct, there should be a high inter correlation between the responses.

In other words, survey respondents will tend to score the same way across a sample of multiple

items due to the internal consistency of these items. Therefore, a low Cronbach alpha coefficient

indicates that the sample of items does not perform well in capturing the construct that the

instrument intends to measure. Whereas, a high alpha suggests that the items are the true

reflection of the construct, and they are relevant to assess the internal consistency of the

construct. Generally, survey researchers consider a minimum of .70 for alpha value to be

acceptable in reference to Nunnally’ (1978) study.

Despite the acceptance of Cronbach’s alpha as the estimate of reliability and

measurement of the internal consistency of survey items (Cortina, 1993), researchers still face

challenges in reporting, interpreting, and using reliability coefficients because of the

contradictory advice commonly found in psychometric journals (Sass, Mifsud, Helms, & Henze,

2006). In a recent study, Thompson and Vacha-Haase (2011) found that 54.6% of authors did not

mention reliability in their reports. Only 15.7% did but simply reported previous scores as if they

were applied to their own data.

The issues surrounding the report, interpretation, and the utilization of coefficient alpha

raised some concerns. Sijtsma (2009) criticized the coefficient alpha as a true measure of the

reliability. Further, Green and Yang (2009, 2011) utterly discouraged its use altogether. The

former contended that alpha, by itself, is not a measure of internal consistency because it

depends on several other statistics, and it is quite difficult to test the same individual repeatedly.

In the same line of thought, Emons, Sijtsma, and Meijer (2007) suggested increasing the number

of items for the survey instrument in a way to gather as much information as possible about the

participants (Sijtsma, 2009). Although this may be practical, evidence also showed that long

questionnaires could encourage non responses in survey research (Bean & Roszkowski, 1995;

BeVier & Roth, 1998).

Conclusion

With the rising demand for market intelligence and the increasing number of survey

researchers, quantitative survey design presents serious challenges that require a strong

commitment to the positivist principles throughout the research process. Positivist paradigm

assumes the measurability and impartiality of social phenomena. It also infers that knowledge

presents concrete characteristics, including validity, reliability, and generalizability, which

cannot be ignored. Therefore, survey researchers use statistical analyses to minimize potential

weaknesses of their instruments and prevent findings from biases. However, it has been argued

throughout the literature that significant challenges can still emerge from errors, human factors,

or both. Although survey research is daunting, it continues to be popular because researchers find

it to be cost effective, efficient to collect data, and appropriate to answer specific research

questions. While the issues of quantitative survey research that have been discussed throughout

this paper can be easily understood from a scholarly perspective, it is not always the case within

the business community, where the search for a quick answer usually compromises the scientific

rigor of the methodology. It is recommended that future studies investigate the extent to which

scholar-practitioners could strengthen survey design in business research and the credibility of

this promising profession.

References

Amaratunga, D., Baldry, D., Sarshar, M., & Newton, R. (2002). Quantitative and qualitative research in the built environment: Application of "mixed" research approach. International Journal of Productivity and Performance Management, 51(1), 17-31.

Barlett, J. E., Kotrlik, J. W., & Higgins, C. C. (2001). Organizational research: Determining appropriate sample size in survey research. Information Technology, Learning, and Performance Journal, 19(1), 43-50. http://search.proquest.com/docview/219816871?accountid=27965

Bean, A. G., & Roszkowski, M. J. (1995). The long and short of it. Marketing Research, 7(1), 20.

Betts, S. C. (2011). Extending the forer test beyond face validity: An experiential approach to teaching social science methodology. Academy of Educational Leadership Journal, 15, 143-149. http://search.proquest.com/docview/887021253?accountid=27965

BeVier, C. A., & Roth, P. L. (1998). Response rates in HRM/OB survey research: Norms and correlates, 1990-1994. Journal of Management, 24(1), 97-117. doi:10.1177/014920639802400107

Bielefeld, W. (2006). Quantitative research for nonprofit management. Nonprofit Management & Leadership, 16(4), 395-409. doi:10.1002/nml.117

Burton, L. & Mazerolle, S.M. (2011). Survey instrument validity part I: principles of survey instrument development and validation in athletic training education research. Athletic Training Education Journal, 6(1), 27-35.

Carr, L. T. (1994). The strengths and weaknesses of quantitative and qualitative research: what method for nursing?. Journal Of Advanced Nursing, 20(4), 716-721. doi:10.1046/j.1365-2648.1994.20040716.x

Churchill, G. A. (1979). A paradigm for developing better measures of marketing constructs. Journal of Marketing Research, 16(1), 64-73.

Clark, A. M. (1998). The qualitative-quantitative debate: Moving from positivism and confrontation to post-positivism and reconciliation. Journal of Advanced Nursing, 27, 1242-1249.

Collins, D. (2003). Pretesting survey instruments: An overview of cognitive methods. Quality of Life Research, 12(3), 229-238. doi:10.1023/A:1023254226592

Cortina, J. M. (1993). What is coefficient alpha? an examination of theory and applications. Journal of Applied Psychology, 78(1), 98-104. doi:10.1037/0021-9010.78.1.98

Couper, M., & Miller, P. (2008). “Web survey methods. Introduction”, Public Opinion Quarterly, 72(5), 831-835.

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302. doi:10.1037/h0040957

Curtin R., Presser S., & Singer E. (2000). The effects of response rate changes on the Index of Consumer Sentiment. Public Opinion Quarterly, 64, 413–428.

Eldabi, T., Irani, Z., Paul, R. J., & Love, P. E. D. (2002). Quantitative and qualitative decision-making methods in simulation modelling. Management Decision, 40(1), 64-73. http://search.proquest.com/docview/212077597?accountid=27965

Emons, W. H. M., Sijtsma, K., & Meijer, R. R. (2007). On the consistency of individual classification using short scales. Psychological Methods, 12(1), 105-120. doi:10.1037/1082-989X.12.1.105

Firestone, W. A. (1987) “Meaning in Method: The Rhetoric of Quantitative and Qualitative Research.” Educational Researcher, 16 (7), 16–21.

Frippiat, D., & Marquis, N. (2010). Web surveys in the social sciences: An overview. Population (16342941), 65(2), 285-311. Retrieved from http://ezproxy.library.capella.edu/login?url=http://search.ebscohost.com.library.capella.edu/login.aspx?direct=true&db=sih&AN=55435229&site=ehost-live&scope=site

Green, S. B., & Yang, Y. (2009). Commentary on coefficient alpha: A cautionary tale. Psychometrika, 74(1), 121-135. doi:10.1007/s11336-008-9098-4

Green, S. B. & Yang, Y. (2011). Coefficient alpha: A reliability coefficient for the 21st century? Journal of Psychoeducational Assessment, 29(4), 377-392. doi:10.1177/0734282911406668

Guba, E. G. & Lincoln, Y. S. (1994). Competing paradigms in qualitative research. Handbook of qualitative research, 2, 163-194.

Harrison, R. T., Hill, F. M., & Leitch, C. M. (2010). The philosophy and practice of interpretivist research in entrepreneurship: Quality, validation, and trust. Organizational Research Methods, 13(1), 67-84. doi:10.1177/1094428109339839

Hox, J. (2007). The influence of advance letters on response in telephone surveys: A meta-analysis. The Public Opinion Quarterly, 71(3), 413-443.

Ihantola, E. & Kihn, L. (2011). Threats to validity and reliability in mixed methods accounting research. Qualitative Research in Accounting and Management, 8(1), 39-58. doi:10.1108/11766091111124694

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26. doi:10.3102/0013189X033007014

Keeter, S., Miller C., Kohut A., Groves R., & Presser S. (2000). Consequences of reducing nonresponse in a large national telephone survey. Public Opinion Quarterly, 64, 125–148.

Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.

O'Leary-Kelly, S. W., & J. Vokurka, R. (1998). The empirical assessment of construct validity. Journal of Operations Management, 16(4), 387-405. doi:10.1016/S0272-6963(98)00020-5

Onwuegbuzie, A. J., & Johnson, R. B. (2006). The validity issue in mixed research. Research in the Schools, 13(1), 48-63. http://search.proquest.com/docview/211030483?accountid=27965

Parasuraman, A., Zeithaml, V. & Berry, L. (1988). "SERVQUAL: a multiple-item scale for measuring consumer perceptions of service quality", Journal of Retailing, 64, 12-40.

Ponterotto, J. G. (2005). Qualitative research in counseling psychology: A primer on research paradigms and philosophy of science. Journal of Counseling Psychology, 52(2), 126-136. doi:10.1037/0022-0167.52.2.126

Ross, E. M. (2005). It seemed like a good idea at the time. NBCNEWS.com. Retrieved from http://www.msnbc.msn.com/id/7209828/ns/us_news/t/it-seemed-good-idea-time/

Rossiter, J. R. (2008). Content Validity of Measures of Abstract Constructs in Management and Organizational Research. British Journal Of Management, 19(4), 380-388. doi:10.1111/j.1467-8551.2008.00587.x

Sass, T. L., Mifsud, V. A., Helms, J. E., & Henze, K. T. (2006). Treating cronbach's alpha reliability coefficients as data in counseling research. The Counseling Psychologist, 34(5), 630-660. doi:10.1177/0011000006288308

Schmeets, H. (2010). Increasing response rates and the consequences in the dutch parliamentary election study 2006. Field Methods, 22(4), 391-412. doi:10.1177/1525822X10381031

Shuttleworth, M. (2009). Criterion Validity: Criterion validity assesses whether a test reflects a certain set of abilities. Experiment-resources.com. Retrieved on May 5, 2012 from http://www.experiment-resources.com/criterion-validity.html

Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107-120. doi:10.1007/s11336-008-9101-0

Terrell, R. S. (2012). Mixed-methods research methodologies. The Qualitative Report, 17(1), 254.

Thompson, B., & Vacha-Haase, T. (2011). Score reliability: A retrospective look back at 12 years of reliability generalization studies. Measurement and Evaluation in Counseling and Development, 44(3), 159-168. doi:10.1177/0748175611409845

Turocy, P. S. (2002). Survey research in athletic training: The scientific method of development and implementation. Journal of Athletic Training, 37(4), 174-179.

Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, 2012-13 Edition, Survey Researchers. Retrieved on April 21, 2013 from http://www.bls.gov/ooh/life-physical-and-social-science/survey-researchers.htm

Walsham, G. (2006). Doing interpretive research. European Journal of Information Systems, 15(3), 320-330. doi:10.1057/palgrave.ejis.3000589