Running head: 1

Personal Bias in Categorizing Arguments as Fact or Opinion

Aman Siddiqi

Columbia College

2

Abstract

How do we determine an argument’s factuality? When is an argument fact and when is it

opinion. I explored the influence of personal certainty, feelings of importance and view of

definitiveness on the categorization of arguments as fact or opinion. An online survey of 450

subjects revealed that arguments were more likely to be categorized as fact when the individual

(a) only acknowledged the existence of one correct answer, (b) felt certain in his or her personal

answer (c) and believed the issue was very important. Because these variables vary greatly for

arguments, subjects did not agree when categorizing them as fact or opinion. My research shows

the subjective personal viewpoint of an individual can influence the judgment of an argument as

factual. This can help explain one reason for disagreement among the populace on scientific,

political, and social issues.

Keywords: argument; factuality; certainty; judgment; definitiveness; importance

3

Personal Bias in Categorizing Arguments as Fact or Opinion

Differentiating fact from opinion is an important critical thinking skill (Paul et al., 1989).

To evaluate the credibility and reliability of information a critical thinker must examine if

statements are fact or opinion. Identifying facts becomes more complex when dealing with

argumentation. Arguments involve the analysis of evidence through a particular viewpoint or

perspective. The conclusion of any argument is therefore as much dependent upon the method of

analysis as on the evidence. When making an argument a debater cites facts to support his or her

position. However, it is unclear if the conclusion of an analysis based upon facts can also be

called a fact. Is a court decision a fact when it is based on physical evidence? If two opposing

sides of an argument are supported by facts, can both resulting conclusions be facts? Because

arguments involve multiple sides and different interpretations, under what circumstances will a

conclusion be labeled as fact? The ambiguous factuality of arguments is a crucial question.

Political, judicial and business decisions are made once information is labeled as fact.

Understanding how individuals make this judgment has a wide reaching impact.

How are arguments evaluated? Arguments combine premises through logical progression

to reach a conclusion. When judging the soundness of a conclusion a critical thinker must

evaluate both its logical validity and the truth of all premises (Posner, 1990). The subjectivity

within this evaluation results in two problems for categorizing an arguments as fact or opinion.

First, multiple sides of an argument may be equally valid, but contain premises of varying

subjectivity. When a premise is subjective, the soundness of the conclusion becomes dependent

upon the evaluator’s view of the premise. Compare the following two statements.

4

Pizza is made with cheese. This item is a pizza. Therefore this item is made with cheese.

Pizza is very good. This is item is a pizza. Therefore this item is very good.

Both arguments are logically sound. The premise in the first statement is objective.

Pizzas are made with cheese. This fact does not vary from person to person. The conclusion is

therefore a fact (Jordan, 2007). The premise in the second statement is subjective. The soundness

of the argument depends on each individual’s view of the premise. The conclusion is therefore an

opinion. However, many arguments are far more complex than this example. Arguments often

include an array of evidence, viewpoints and interpretations. An individual may not have a full

understanding of all an argument’s premises. In addition, different individuals may hold varying

views on a premise’s validity. For example, some arguments involve non-repeatable data like the

evolution of species. Data that cannot easily be replicated gives room for different evaluations

among individuals. As well, not all phenomena can be directly observed. For example,

astronomical events are inferred from distant images. These inferences may not be agreed upon

by all. The objectivity of some premises may be unknown by the evaluator or debated among

experts. Given these difficulties, an individual must collectively judge an argument’s premises to

evaluate the conclusion.

Second, evaluating the logic of an argument can be difficult. For simple arguments,

logical validity is often clear. The following argument is logically invalid.

My blanket is blue.

The sky is blue.

Therefore, the sky is my blanket.

The argument demonstrates the logical error named affirming the consequent (Gauch,

2003). However, individuals may be ill-equipped to evaluate the logic within complex arguments

5

containing numerous premises. In addition to logical validity, critical thinkers must evaluate the

completeness of an argument. Does it explain all existing evidence or only include facts that

support the conclusion? Does the argument contradict other arguments which are necessary for

the overall conclusion? Do alternative arguments exist that explain the data as well or better?

Because arguments are constructed from facts, there may be multiple logically valid and

complete arguments that all explain the same data. Take for example the following two facts.

Fact 1: A child returns home after the appointed curfew.

Fact 2: The parent grounds the child for one week.

These same facts can be assembled into the following two opposing arguments using different

inferences. They result in opposing conclusions.

1. The parent is angry at the child's disobedience. Seeking vengeance for the child's

insubordination the parent wishes to harm the child through punishment. The parent's

decision was therefore spiteful and immoral.

2. The parent is concerned about the child's future. The parent institutes a punishment to

help cement the importance of following the parent’s guidance. The parent's decision was

therefore caring and responsible.

Arguments are the subjective analysis of objective facts. Each individual must evaluate

the overall quality of an argument’s construction. There is no ultimate right or wrong answer

(Schwartz, 1997). Given this combination of subjectivity and objectivity, how will an argument’s

conclusion be categorized as fact or opinion? The same argument may be convincing to one

individual and implausible to another. This problem is exasperated when the underlying

foundations are unclear or when no single conclusion is certain? Each individual must somehow

come to an answer on the soundness of all premises and logical relationships underlying an

6

argument. No prior research on categorizing facts has specifically focused on argument

conclusions. Rabinowitz et al. (2013) conducted a series of five studies categorizing textbook

statements as fact or belief. The statements were compared across domains including

psychology, biology, and history. Marsh et al. (2003) examined the differentiation of fact and

fiction when integrated into memory. Goodwin and Darley (2012) investigated moral beliefs as

fact or opinion.

My research question is twofold. First, is argument factuality more difficult to categorize

than objective data and personal preferences? Ancillary evidence shows subjects disagree when

categorizing moral statements as fact or opinion (Goodwin & Darley, 2008; Fernandez-Duque et

al., 2014). The moral statements used in prior research often took the form of arguments in

which more than one viewpoint was possible. In these cases the categorizations of fact and

opinion where divergent. Second, what mechanisms do subjects employ toward categorization?

My hypothesis is personal viewpoint is used to determine the factuality of an argument. When an

individual feels certain in a conclusion, it will be classified as fact. However, the same

conclusion can be seen as opinion if the evaluator is unsure. I predict the personal perspective of

the individual making a judgment will be a major determinant in the categorization of an

argument as fact or opinion.

Method

Participants

I conducted an online survey to investigate the relationship between personal certainty,

feelings of importance, definitiveness and factuality of arguments. A total of 450 subjects were

surveyed online. Subjects ranged from age 18 to 70 with 220 male and 230 female. Subjects

7

were recruited through Amazon Mechanical Turk and paid $0.10 per survey. Responses were

discarded from participants who did not answer the validation question correctly.

Procedure

Previous research used questionnaire items that expressed a specific conclusion. One

survey item from Rabinowitz et al. (2013) read “The justice system in our country is fair.” A bias

could be introduced by stating a specific viewpoint. Subjects who agree with the conclusion may

be more likely to label it as fact than those who do not. To eliminate this possibility, my

questionnaire used a unique approach. Each argument was phrased as a question. No particular

conclusion was presented. For example, one item read “Is democracy the best form of

government?”. Subjects were then questioned about their conclusions on the issue; i.e. “How

certain are you of the answer?”. The participant could feel very certain democracy is the best

form of government or very certain it is not. It either case, the subject feels very certain about

their conclusion for the argument.

From a pool of 18 arguments each participant saw four randomly selected argument topics

such as, “Did human beings and apes both evolve from a common ancestor?”, “Does Global

Warming exist?’ and “Are Americans better off than the French?”. These questions require

choosing a conclusion to various arguments that can be made on either side. In addition, subjects

saw four pieces of data and three personal preferences from a pool of 14. The full list of

argument topics, data, and preferences is included in the appendix. For each item, the participant

was asked to report:

a) Personal certainty: How certain are you of the answer? (5pt scale)

b) Personal importance: How important is the answer to you? (5pt scale)

c) Definitiveness: There is one correct answer / many equally valid answers.

8

d) Factuality: The answer is a fact / opinion / neither

The dependent variable was factuality. The three independent variables were certainty,

importance and definitiveness. The certainty score was used as an indicator for strength of

conviction. Because the arguments were described as questions, subjects marked the certainty

they felt for their personal conclusion, not a pre-designated viewpoint. The importance score

measures the argument’s relative place among issues in the subject’s view. Both variables were

ordinal ranging from one to five. A subject who states the question has only one correct answer

feels his conclusion is definitive. If the argument has multiple equally valid conclusions, there is

more than one way to view the issue.

Definitiveness should not be confused with objectivity and subjectivity. Subjective

statements originate within the mind. They vary from person to person according to judgment.

Objective statements are based on criteria external to the mind and should not vary across

individuals. Objective statements are often assumed to be definitive; having only one answer.

This is not always true. The formula x2=16 objectively has two solutions; 4 and -4. There is

more than one answer, but the origin of this variation is not human judgment. Likewise some

subjective moral issues, like “rape is wrong,” are described by subjects as having only one

correct answer (Goodwin & Darley, 2008; Wainryb et al., 2004). These statements are still

subjective since they stem from judgment. Some past research has defined objectivity as the

existence of only one correct answer (Goodwin & Darley, 2008; Wainryb et al., 2004; Kuhn et

al., 2000). I believe it is important to distinguish definitiveness from objectivity. My survey

endeavors to answer if a subject’s personal certainty in an argument’s conclusion, feeling of

importance for the issue, or viewpoint on the argument’s definitiveness affects categorization of

an argument’s factuality.

9

Results

Divergent Judgments

When categorizing data and personal preferences subjects displayed a high degree of

consensus. Data was categorized as fact by 91% of subjects and preferences as opinion by 92%,

In contrast, no consensus was displayed when categorizing arguments as fact or opinion. On

average 60% of the respondents disagreed with the remaining 40%. The normalized agreement

measurement disregards the specific direction of the judgment. For example, if 100% of subjects

judged argument A as fact and 100% of subjects judged argument B as opinion, agreement was

100% for both. Averaging the raw data would erroneously return 50% agreement. This value

would not account for agreement within each argument. Disagreement on individual arguments

ranged from a 51%-49% split to an 86%-14% majority. Overall consensus was weak to

moderate. Figure 1 displays the normalized agreement across all questions.

123456789

101112131415161718

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Majority Minority

10

Figure 1. Normalized agreement levels for judgments on factuality.

Personal Certainty, Importance, and Definitiveness are Related to Judgments on Factuality

A positive relationship was see between all three independent variables and judgments on

factuality. Subjects with greater personal certainty in an argument’s conclusion were more likely

to categorize the argument as fact. The same relationship was seen for feelings of importance and

definitiveness. The mean, median and mode of ratings for both personal certainty and importance

were higher for subjects choosing fact than those choosing opinion. Subjects choosing fact were

more likely to have ratings of 4 or 5 for both personal certainty and importance. There was a

strong correlation between a subject’s view on an argument’s definitiveness and categorizing it

as a fact (Φ = 0.55). Table 1 includes a full list of results.

Table 1

Ratings of certainty, importance and definitiveness for arguments categorized as

fact and opinion

Personal Certainty in Conclusion Fact OpinionMean Certainty 4.1 3.4Median Certainty 5 3Mode Certainty 5 3Rated 4 or 5 77% 47%

Feeling of Importance for the issue Fact OpinionMean Importance 3.8 3Median Importance 4 3Mode Importance 5 3Rated 4 or 5 63% 36%

Definitiveness of Argument* Fact OpinionPercent Definitive 87%** 13%Percent Indefinite 17% 83%***

11

Note: * p < .001, ** 95% CI [85%, 89%], *** 95% CI [81%, 85%]

This positive relationship was also seen within each individual argument. For example,

when asked “Does god exist?”, average certainty in subject conclusions was 4.5 for those who

chose fact and 3.5 for those who chose opinion. Table 2 displays responses for three argument

conclusions that were evenly divided between fact and opinion with a small remainder selecting

neither. For each argument, the mean certainty and mean importance is higher for those selecting

fact.

Table 2

Ratings of certainty and importance for arguments categorized as fact and as opinion

Mean Certainty Mean Importance

Fact OpinionFac

t Opinion Fact OpinionAre human beings the most evolved species on Earth?

47% 45% 4.5 3.6 3.9 3.2

Are thin people usually more healthy than overweight people?

44% 43% 4.2 3.4 3.8 3.1

Was the land of the Native Americans stolen by the United States?

43% 47% 4.3 3.5 3.4 2.8

As expected, no relationship was seen between personal certainty, importance, definitiveness and

factuality for data or preferences.

Explaining the Results

Subjects predominantly agree when categorizing data and preferences as fact and opinion

respectively. In contrast, when categorizing arguments as fact or opinion disagreement among

subjects is high. The source of this disagreement may lie in the complex nature of arguments. A

spectrum for the objectivity of statements can be created spanning from fact to opinion, figure 2.

Arguments lie near the middle of this spectrum. Like a fact, an argument involves objective

12

evidence external to the self. Yet those facts must be processed through the personal viewpoint

of an individual, like an opinion. When dealing with arguments, everyone examines the same

evidence, but process the evidence differently. This explains why debated topics using the same

information tend to result in a variety of conclusions.

Figure 2. Spectrum of objectivity

Whereas most people will agree 2+2=4; economic experts were divided when asked if the

2014 U.S. fiscal policy was appropriate (National, 2014). Arguments generate disagreement

because each individual can come to a different conclusion even when both sides are supported

by evidence. Evaluating an argument requires an individual to assess (a) the objectivity of all

premises; (b) the logical validity of all inferences; (c) and the completeness of the argument’s

representation of the issue. Often a complete list of information pertinent to an argument is

unavailable. As well, lay persons may not be equipped to fully analyze logical validity. Finally,

these judgments must often be made quickly. Time for a full examination is often not available.

In light of these real world limitations subjects will use heuristics to reach a conclusion

(Kahneman, 2011). Individuals will evaluate an argument quickly with little analysis using

mental shortcuts. According to Kahneman (2011) when an individual is incapable of estimating a

phenomenon, a simpler estimation will be performed in its place. Individuals may be using

definitiveness to determine factuality. If only one conclusion seems possible, the conclusion is

assumed to be fact. If the individual can more easily imagine multiple possibilities, the

conclusion is labeled opinion. This is an example of the availability heuristic (Breckler et al.,

13

2005). The more easily examples come to mind, the more likely an individual will consider those

acceptable alternatives.

Equating definitiveness with factuality can have major repercussions. As noted above, the

definitiveness of an argument’s conclusion can be highly subjective. Each individual’s

evaluation of alternative competing conclusions is based on a subjective analysis of the

argument’s premises and logic. Prejudice and close-mindedness may further limit an individual’s

ability to accurately consider alternative conclusions. Factuality may then be determined by each

individual’s limited world view.

In addition, personal certainty and feelings of importance may be influencing estimations

of definitiveness. First, personal certainty has been shown to be positively related to perceived

consensus (Marks & Miller, 1985). Individuals are more likely to attribute their viewpoints to

others as their personal certainty in the viewpoint increases. When subjects were asked to pass a

verdict in a mock trial, the degree of perceived consensus increased with the participants’

certainty in their verdicts. In addition, the researchers experimentally manipulated the degree of

certainty for some subjects by providing expert information. This additional information was

intended to positively or negatively alter a subject’s feelings of certainty in an answer. The

degree of perceived consensus further increased as verdict certainty rose. If an individual feels

very certain in a conclusion and subsequently feels most people will agree with it the person is

less likely to acknowledge alternative views as credible; more likely to view the conclusion as

definitive.

Second, when evaluating conclusions alternative to one’s own, cognitive dissonance may

be created. Individuals can become uncomfortable entertaining alternative viewpoints when

feelings on the issue are strong. An increased feeling of importance for an issue has been shown

14

to limit the ability to consider alternative ideas (Nyhan et al., 2013). The more important an

argument is to an individual, the more uncomfortable the individual will become with alternative

conclusions and the more likely to view the conclusion as definitive.

If an individual feels certain about an argument regarding an issue on which he or she has

very strong feelings, the likelihood of accepting alternative viewpoints will decrease. When the

factuality of a conclusion is determined by the individual’s view of definitiveness, personal

certainty and feelings of importance become influential variables.

Discussion

How objects and ideas are categorized greatly impacts how we view them. The United

Kingdom Supreme Court ruled in 2009 that Pringles could no longer be categorized as savory

snacks (Cohen, 2009). They were now legally potato crisps. As a result, Procter & Gamble U.K.

was required to pay $160 million in unpaid value-added tax. Categorizations not only have legal

ramifications, but psychological as well. Research at Cornell University’s food psychology lab

has found that labeling food as organic influences a consumer’s perception of taste. Cookies,

yogurt, and potato chips that were labeled organic where described as more flavorful and tasted

lower in fat as compared to identical products with no label (Wan-chen et al., 2013). Our

judgment is influenced by the categorizations we apply to ideas under consideration. A cognition

perceived as fact will be given more weight than an opinion. There is a common expression,

“that’s just your opinion”. There is no expression “that is just a fact.” A fact is powerful. Facts

are objective and considered true unless the foundational evidence is refuted. Decisions and

conclusions are trusted when based upon fact. Opinions are subjective. They may involve

evidence or be entirely founded upon preference. Conflicting opinions do not contrast because

they are both just opinions.

15

If the distinction between fact and opinion is based on an individual’s personal viewpoint,

facts can become opinion and vice versa. The viewpoint of an individual is transient. It can be

swayed by a skillful orator or well-crafted advertisement. An individual’s viewpoint may never

have been based on sound evidence or reason to begin with. This may explain why topics such as

climate change are debated back and forth while the supporting evidence is unchanged. Critical

thinkers must learn how the blended nature of arguments, composed of objective evidence with

subjective interpretation makes their conclusions uniquely susceptible to frailties in human

judgment. If an argument is labeled as fact or opinion based on each individual’s personal

viewpoint, universally accepted resolutions may never be reached. Instead, constant bickering

will ensue. A critical thinker must remember circularity is created when each individual in a

debate defines his or her own side of an argument as fact and the opposing side as opinion. This

may result in individuals judging any conclusion or viewpoint they disagree with as opinion and

those they support as fact, creating a confirmation bias.

16

Acknowledgements

The author would like to thanks Diego Fernandez-Duque, Ph.D of Villanova University for proof

reading my manuscript and offering his generous advice.

17

References

Breckler, S., Olson, J., & Wiggins, E. (2005). Social Psychology Alive. Cengage Learning.

Cohen, A. (2009, June 1). The lord justice hath ruled: Pringles are potato chips. New York Times, A20.

Fernandez-Duque, D., Leman, J., Bonner, J., McCusker, K., & Jackiewicz, M. (2014). Morality as argument: Strong opinions lead to increased perceived consensus in judgments, Paper presented at the annual meeting of the Eastern Psychological Association, Philadelphia, PA.

Gauch, H. G. (2003). Scientific method in practice. Cambridge University Press.

Goodwin, G. P., & Darley, J. M. (2008). The psychology of meta-ethics: Exploring objectivism. Cognition, 106, 1339–1366.

Goodwin, G. P., & Darley, J. M. (2012). Why are some moral beliefs perceived to be more objective than others? Journal of Experimental Social Psychology, 48(1), 250-256.

Jordan, J. (2007, October 29). Becoming elvis [Web log post]. Retrieved from Critical Thinking Cafe: http://blog.actionm.com/2007/10/becoming-elvis.html

Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux.

Kuhn, D., Cheney, R., & Weinstock, M. (2000). The development of epistemological understanding. Cognitive development, 15, 309-328.

Marks, G., & Miller, N. (1985). The effect of certainty on consensus judgments. Personality and Social Psychology Bulletin, 11(2), 165-177. doi:10.1177/0146167285112005

Marsh, E. J., Meade, M. L., & Roediger III, H. L. (2003). Learning facts from fiction. Journal of Memory and Language, 49, 519-536.

National Association of Buiness Economics. (2014). NABE Policy Survey: Economists Align on Monetary Policy but not on Fiscal Issues. Retrieved from http://nabe.com/Policy_Survey_February_2014

Nyhan, B., Reifler, J., & Ubel, P. (2013). The Hazards of Correcting Myths About Health Care Reform. Medical Care, 51(2), 127-132.

Paul, R. W., Martin, D., & Adamson, K. (1989). Critical Thinking Handbook: High School. Foundation for Critical Thinking.

Posner, R. A. (1990). The problems of jurisprudence. Harvard University Press.

Rabinowitz, M., Acevedo, M., Casen, S., Rosengarten, M., Kowalczyk, M., & Blau Portnoy, L. (2013). Distinguishing facts from beliefs: Fuzzy Categories. Journal of Language and Communication, 17(3), 241- 267.

18

Schwartz, B. (1997). Decision: How the supreme court decides cases. Oxford University Press.

The Editors of Encyclopædia Britannica. (2014, 8 24). Theory of Value. Retrieved from Encyclopedia Britannica: http://www.britannica.com/EBchecked/topic/46184/axiology#ref169892

Wainryb, C., Shaw, L. A., Langley, M., Cottam, K., & Lewis, R. (2004). Children’s thinking about diversity of belief in the early school years: Judgements of relativism, tolerance, and disagreeing persons. Child Development, 75, 687–703.

Wan-chen, J., Shimizu, M., Kniffin, K., & Wansink, B. (2013). You taste what you see: Do organic labels bias taste perceptions? Food Quality and Preference, 29(1), 33-39.

19

Appendix

Full list of arguments

1. Does god exist?

2. Did human beings and apes both evolve from a common ancestor?

3. Is democracy the best form of government?

4. Is the Chinese government oppressing freedom of religion?

5. Are human beings the most evolved species on Earth?

6. Do human beings have the right to use animals for labor and food?

7. Does Global Warming exist?

8. Is Chinese medicine as effective as Western medical techniques?

9. Was the land of the Native Americans stolen by the United States?

10. Do Americans watch too much television?

11. Does North Korea pose a security threat to the U.S.?

12. Does the United Kingdom pose a security threat to the U.S.?

13. Are Americans better off than Mexicans?

14. Are Americans better off than the French?

15. Are thin people usually more healthy than overweight people?

16. Are vegetarians usually more healthy that those who eat meat?

17. Is regular exercise necessary to be healthy?

18. Is the American population becoming overweight?

Preferences

1. Do Brussels sprouts taste good?

2. Does pizza taste good?

20

3. Is basketball or football more fun?

4. Is Japan a good place to visit?

5. Is red an attractive color?

6. Which tastes better, Coke or Pepsi?

Data

1. How many states are in the USA?

2. How tall is the Eiffel Tower?

3. What is the marathon world record?

4. How many bones are in the human body?

5. Which is heavier, a pound or a kilogram?

6. On what side of the road do people drive in Bermuda?

7. On what date is Bastille day celebrated in France?

8. What is the atomic weight of boron?

Each statements in the survey contained the following four questions. The user was required to

answer all four.

How certain are you of the answer? 5 (Absolutely Certain), 4, 3, 2, 1 (Not certain at all)

How important is the answer to you? 5 (Crucial), 4, 3, 2, 1 (Not important at all)

There is... one correct answer. many equally valid answers.

The answer is a... Fact Opinion Neither

21

Author Biography: Aman Siddiqi is a quality assurance engineer specializing in critical thinking. He holds degrees in both management and engineering and is planning a Ph.D. in the psychology of critical thinking.