Detecting Hoaxes, Frauds, and Deception in Writing Style Online

Sadia Afroz∗, Michael Brennan∗ and Rachel Greenstadt∗∗Department of Computer Science

Drexel University, Philadelphia, PA 19104Emails: sadia.afroz@drexel.edu, mb553@drexel.edu and greenie@cs.drexel.edu

Abstract—In digital forensics, questions often ariseabout the authors of documents: their identity, de-mographic background, and whether they can belinked to other documents. The field of stylometryuses linguistic features and machine learning tech-niques to answer these questions. While stylometrytechniques can identify authors with high accuracy innon-adversarial scenarios, their accuracy is reducedto random guessing when faced with authors whointentionally obfuscate their writing style or attemptto imitate that of another author. While these resultsare good for privacy, they raise concerns about fraud.We argue that some linguistic features change whenpeople hide their writing style and by identifyingthose features, stylistic deception can be recognized.The major contribution of this work is a method fordetecting stylistic deception in written documents. Weshow that using a large feature set, it is possible todistinguish regular documents from deceptive doc-uments with 96.6% accuracy (F-measure). We alsopresent an analysis of linguistic features that can bemodified to hide writing style.

Keywords-stylometry; deception; machine learn-ing; privacy;

I. INTRODUCTION

When an American male blogger Thomas Mac-Master posed as a Syrian homosexual womanAmina Arraf in the blog “A Gay Girl in Damas-cus” and wrote about Syrian political and socialissues, several news media including The Guardianand CNN thought the blog was “brutally honest,”and published email interviews of Amina1. Eventhough no one had ever spoken to or met her and noSyrian activist could identify her, “Amina” quickly

1http://www.telegraph.co.uk/news/worldnews/middleeast/syria/8572884/A-Gay-Girl-in-Damascus-how-the-hoax-unfolded.html

became very popular as a blogger. When “Amina’scousin” announced that she had been abducted bythe Syrian police, thousands of people supportedher on social media and made the US state de-partment investigate her fictional abduction2. Thisscrutiny led to the discovery of the hoax.

The authenticity of a document (or blog post)depends on the authenticity of its source. Sty-lometry can help answering the question, “whois the writer of a particular document?” Manymachine learning based methods are available torecognize authorship of a written document basedon linguistic style. Stylometry is important to se-curity researchers as it is a forensics technique thathelps detect authorship of unknown documents. Ifreliable, it can be used to provide attribution forattacks, especially when attackers release mani-festos explaining their actions. It is also importantto privacy research, as it is necessary to hide theindications of authorship to achieve anonymity.Writing style as a marker of identity is not ad-dressed in current privacy and anonymity tools.Given the high accuracy of even basic stylometrysystems this is not a topic that can afford to beoverlooked.

In the year 2000, Rao and Rohatgi ques-tioned whether pseudonymity could provide pri-vacy, showing that linguistic analysis could identifyanonymous authors on sci.crypt by comparingtheir writing to attributed documents in the RFCdatabase and on the IPSec mailing list [23]. Inthe intervening years, linguistic authorship iden-

2http://www.foxnews.com/world/2011/06/07/gay-girl-in-damascus-blogger-kidnapped-by-syrian-forces/

tification techniques have improved in accuracyand scale to handle over fifty potential authorswith over 90% accuracy [1], and even 100,000authors with significant accuracy [17]. At the sametime there has been an explosion in user-generatedcontent to express opinions, to coordinate protestsagainst repressive regimes, to whistle blow, to com-mit fraud, and to disclose everyday information.

These results have not been lost on the lawenforcement and intelligence communities. The2009 Technology Assessment for the State ofthe Art Biometrics Excellence Roadmap (SABER)commissioned by the FBI stated that, “As non-handwritten communications become more preva-lent, such as blogging, text messaging and emails,there is a growing need to identify writers not bytheir written script, but by analysis of the typedcontent [29].”

Brennan and Greenstadt [2] showed that currentauthorship attribution algorithms are highly accu-rate in the non-adversarial case, but fail to attributecorrect authorship when an author deliberatelymasks his writing style. Their work defined andtested two forms of adversarial attacks: imitationand obfuscation. In the imitation attack, authorshide their writing style by imitating another author.In the obfuscation attack, authors hide their writingstyle in a way that will not be recognized. Tradi-tional authorship recognition methods perform lessthan random chance in attributing authorship inboth cases. These results were further confirmedby Juola and Vescovi [10]. These results show thateffective stylometry techniques need to recognizeand adapt to deceptive writing.

We argue that some linguistic features changewhen people hide their writing style and by iden-tifying those features, deceptive documents can berecognized. According to Undeutsch Hypothesis[26] “Statements that are the product of experiencewill contain characteristics that are generally absentfrom statements that are the product of imagina-tion.” Deception requires additional cognitive effortto hide information, which often introduces subtlechanges in human behavior [6]. These behavioralchanges affect verbal and written communication.Several linguistic cues were found to discriminate

deceptive communication from truthful communi-cation. For example, deceivers use fewer long sen-tences, fewer average syllables per word and sim-pler sentences than truth tellers [3]. Thus, deceptivelanguage appears to be less complex and easierto comprehend. Our analysis shows that thoughstylistic deception is not lying, similar linguisticfeatures change in this form of deception.

The goal of our work is to create a frameworkfor detecting the indication of masking in writ-ten documents. We address the following researchquestions:

1) Can we detect stylistic deception in docu-ments?

2) Which linguistic features indicate stylisticdeception?

3) Which features do people generally changein adversarial attacks and which featuresremain unchanged?

4) Does stylistic deception share similar char-acteristics with other deceptions?

5) Are some adversarial attacks more difficultto detect than others?

6) Can we generalize deception detection?

This work shows that using linguistic and con-textual features, it is possible to distinguish stylisticdeception from regular writing with 96.6% accu-racy (F-measure) and identify different types ofdeception (imitation vs. obfuscation) with 87%accuracy (F-measure). Our contributions includea general method for distinguishing stylistic de-ception from regular writing, an analysis of long-term versus short-term deception, and the discoverythat stylistic deception shares similar features withlying-type deception (and can be identified usingthe linguistic features used in lying detection).

We perform analysis on the Brennan-Greenstadtadversarial dataset and a similar dataset collectedusing Amazon Mechanical Turk (AMT)3. We showthat linguistic cues that can detect stylistic decep-tion in the Extended-Brennan-Greenstadt adversar-ial dataset can detect indication of masking in thedocuments collected from the Ernest Hemingway4

3https://mturk.amazon.com4http://en.wikipedia.org/wiki/International Imitation Hemingway Competition

and William Faulker imitation contests5. We alsoshow how long-term deceptions such as the blogposts from “A Gay Girl in Damascus” are differentfrom these short-term deceptions. We found thesedeceptions to be more robust to our classifierbut more vulnerable to traditional stylometry tech-niques.

The remainder of the paper is organized asfollows. In section 2, we discuss related works inadversarial stylometry. Section 3 explains how de-ception detection is different from regular author-ship recognition. Section 4 describes our analyticapproach of detecting deception. In section 5, wedescribe our data collection methods and datasets.We follow with our results of detecting deceptionon different datasets in Section 6 and discuss theirimplications in Section 7.

II. RELATED WORK

The classic example in the field of stylometryis the Federalist Papers. 85 papers were publishedanonymously in the late 18th century to persuadethe people of New York to ratify the AmericanConstitution. The authorship of 12 of these paperswas heavily contested [18]. To discover who wrotethe unknown papers, researchers have analyzed thewriting style of the known authors and comparedit to that of the papers with unknown authorship.The features used to determine writing styles havebeen quite varied. Original attempts used the lengthof words, whereas later attempts used pairs ofwords, vocabulary usage, sentence structure, func-tion words, and so on. Most studies show the authorwas James Madison.

Several resources give an overview of stylometrymethods [14], [28], and describe the state of thefield as it relates to computer science and computerlinguistics [11] or digital forensics [5]. ArtificialIntelligence has been embraced in the field ofstylometry, leading to more robust classifiers usingmachine learning and other AI techniques [9], [27].There has also been some work on circumvent-ing attempts at authorship attribution [12], [23],using stylometry to deanonymize conference re-views [16], and looking at stylometry as a method

5http://en.wikipedia.org/wiki/Faux Faulkner contest

of communication security [4], but these worksdo not deal with malicious attempts to circum-vent a specific method. Some research has lookedat imitation of authors. Somers [25] comparedthe work of Gilbert Adair’s literary imitation ofLewis Carroll’s Alice in Wonderland, and foundmixed results. Other work looks into the impactof stylometry on pseudonymity [23] and authorsegmentation [1].

Most authorship recognition methods are builton the assumption that authors do not make anyintentional changes to hide their writing style.These methods fail to detect authorship when thisassumption does not hold [2], [10]. The accuracy ofdetecting authorship decreases to random guessingin the case of adversarial attacks.

Kacmarcik and Gamon explored detecting ob-fuscation by first determining the most effectivefunction words for discriminating between textwritten by Hamilton and Madison, then modifyingthe feature vectors to make the documents appearauthored by the same person. The obfuscationwas then detected with a technique proposed byKoppel and Scher, “unmasking,” that uses a seriesof SVM classifiers where each iteration of clas-sification removes the most heavily weighted fea-tures. The hypothesis they put forward (validatedby both Koppel and Scher [13] and Kacmarcikand Gamon [12]) is that as features are removed,the classifier’s accuracy will slowly decline whencomparing two texts from different authors, butaccuracy will quickly drop off when the same isdone for two texts by the same author (whereone has been modified). It is the quick decline inaccuracy that shows there is a deeper similaritybetween the two authors and indicates the unknowndocument has most likely been modified.

However, the above work has some significantlimitations. The experiments were performed onmodified feature vectors, not on modified docu-ments or original documents designed with obfus-cation in mind. Further, the experiments were lim-ited to only two authors, Hamilton and Madison,and on the Federalist Papers data set. It is unclearwhether the results generalize to actual documents,larger author sets and modern data sets.

We analyzed the differences between the controland deceptive passages on a feature-by-featurebasis and used this analysis to determine whichfeatures authors often modify when hiding theirstyle, designing a classifier that works on actual,modern documents with modified text, not justfeature vectors.

III. DIFFERENCE FROM REGULAR AUTHORSHIPRECOGNITION

Deception detection is challenging becausethough authorship recognition is a well-studiedproblem, none of the current algorithms are robustenough to detect stylistic deception and performclose to random chance if the author changes hisusual writing style. This problem is quite dif-ferent from distinguishing one author’s samplesfrom others. In supervised authorship recognition,a classifier is trained on the sample documents ofdifferent authors to build a model that is specificto each author. In deception detection, we traineda classifier on regular and deceptive documentsto model the generic characteristic of regular anddeceptive documents.

Our classifier is trained on the Extended-Brennan-Greenstadt dataset where participantsspent 30 minutes to an hour on average to writedocuments in a style different from their own.Our test set also consists of imitated documentsfrom the Ernest Hemingway and William Faulknerimitation contests. We investigated the effect oflong-term deception using obfuscated documentsfrom a deceptive blog. The skill levels of theparticipants in these datasets are varied. In theBrennan-Greenstadt dataset, the participants werenot professional writers and they had a pre-specified topic to develop a different writing style,but authors in the imitation contests and fictionalblog were mostly professional writers and hadenough time and chose a topic of their choiceto express themselves in a different voice otherthan their own. The fact that the classifier, trainedon the Extended-Brennan-Greenstadt dataset, candetect deception in the test sets—though at loweraccuracy—generalizes the underlying similarityamong different kinds of deception.

IV. ANALYTIC APPROACH

Our goal is to determine whether an author hastried to hide his writing style in a written document.In traditional authorship recognition, authorship ofa document is determined using linguistic featuresof an author’s writing style. In deceptive writing,when an author is deliberately hiding his regularwriting style, authorship attribution fails becausethe deceptive document lacks stylistic similaritywith the author’s regular writing style. Thoughrecognizing correct authorship of a deceptive doc-ument is hard, our goal is to see if it is possibleto discriminate deceptive documents from regulardocuments.

To detect adversarial writing, we need to identifya set of discriminating features that distinguishdeceptive writing from regular writing. After de-termining these features, supervised learning tech-niques can be used to train and generate classifiersto classify new writing samples.

A. Feature selection

The performance of stylometry methods dependson the combination of the selected features andanalytical techniques. We explored three featuresets to identify stylistic deception.

Writeprints feature set: Zheng et al. pro-posed the Writeprints features that can represent anauthor’s writing style in relatively short documents,especially in online messages [30]. These “kitchensink” features are not unique to this work, butrather represent a superset of the features used inthe stylometry literature. We used a partial set ofthe Writeprints features, shown in Table I.

Our adaptation of the Writeprints features con-sists of three kinds of features: lexical, syntactic,and content specific. The features are describedbelow:

Lexical features: These features include bothcharacter-based and word-based features. Thesefeatures represent an author’s lexicon-related writ-ing style: his vocabulary and character choice.The feature set includes total characters, specialcharacter usage, and several word-level featuressuch as total words, characters per word, frequencyof large words, unique words.

Syntactic features: Each author organizes sen-tences differently. Syntactic features represent anauthor’s sentence-level style. These features in-clude frequency of function words, punctuation andparts-of-speech (POS) tagging. We use the list offunction words from LIWC 2007 [19].

Content Specific features: Content specific fea-tures refer to keywords for a specific topic. Thesehave been found to improve performance of au-thorship recognition in a known context [1]. Forexample, in the spam context, spammers use wordslike “online banking” and “paypal;” whereas sci-entific articles are likely to use words related to“research” and “data.”

Our corpus contains articles from a variety ofcontexts. It includes documents from business andacademic contexts, for example school essays andreports for work. As our articles are not from aspecific context, instead of using words of anyparticular context we use the most frequent wordn-grams as content-specific features. As we areinterested in content-independent analytics, we alsoperformed experiments where these features wereremoved from the feature set.

Table I: Writeprints feature set

Category Quantity DescriptionCharacter related 90 Total characters, percent-

age of digits, percentageof letters, percentage ofuppercase letters, etc. andfrequency of character un-igram, most common bi-grams and tri-grams

Digits, specialcharacters,punctuations

39 Frequency of digits (0-9), special characters(e.g.,%,&, *) and punctuations

Word related 156 Total words, number ofcharacters per word, fre-quency of large words,etc. Most frequent worduni-/bi-/ tri-grams

Function wordsand parts-of-speech

422 frequency of functionwords and parts-of-speech

Lying-detection feature set: Our feature setincludes features that were known to be effectivein detecting lying type deception in computer me-diated communications and typed documents [3],

[8]. These features are:1) Quantity (number of syllables, number of

words, number of sentences),2) Vocabulary Complexity (number of big

words, number of syllables per word),3) Grammatical Complexity (number of short

sentences, number of long sentences, Flesh-Kincaid grade level, average number ofwords per sentence, sentence complexity,number of conjunctions),

4) Uncertainty (Number of words express cer-tainty, number of tentative words, modalverbs)

5) Specificity and Expressiveness (rate of adjec-tives and adverbs, number of affective terms),

6) Verbal Non-immediacy (self-references,number of first, second and third personpronoun usage).

We use the list of certainty, tentative and affectiveterms from LIWC 2007 [19].

9-feature set (authorship-attribution fea-tures): This minimal feature set consists of thenine features that were used in the neural net-work experiments in Brennan’s 2009 paper [2].The features are: number of unique words, com-plexity, Gunning-Fog readability index, charactercount without whitespace, character count withwhitespace, average syllables per word, sentencecount, average sentence length, and Flesch-Kincaidreadability score.

B. Classification

We represent each document as (~x, y) where~x ∈ Rn is a vector of n features and y ∈{Regular, Imitation, Obfuscation} is the typeof the document. In our study, n = 9 for 9-features, n = 20 for lying-detection features andn = 707 for the Writeprints features. For classifica-tion, we used Support Vector Machine (SVM) withSequential Minimal Optimization (SMO) [20] im-plemented in the WEKA tool [7] with a polynomialkernel. We tested our dataset with other classifiersin the WEKA tool such as k-Nearest Neighbor,Naive Bayes, J48 Decision Tree, Logistic Regres-sion and SVM with RBF kernel. We chose tofocus on the SMO SVM as it outperformed other

classifiers in most of the test cases. The exceptionto this is the Lying-detection feature set, in whicha J48 Decision Tree6 outperformed SVMs. J48 isthe JAVA implementation of C4.5 algorithm forconstructing decision tree [22]. It is notable thatthe original work using these features also used adecision tree [3].

V. DATA COLLECTION

We present results on three datasets. The firstone is the Extended-Brennan-Greenstadt corpuswhich contains the Brennan-Greenstadt corpus,extended with regular, obfuscated and imitatedwriting samples of the AMT workers. The seconddataset, which we call the Hemingway-FaulknerImitation corpus, contains articles from the Interna-tional Imitation Hemingway Competition and FauxFaulkner contest. The last dataset, Thomas-AminaHoax corpus, contains blog posts from “A Gay Girlin Damascus” blog, posts of Thomas MacMasterin the alternate-history Yahoo! group7 as himselfand as Amina Arraf, and writing samples of BrittaFroelicher, a graduate student at Center for SyrianStudies at St Andrews, who is also Thomas’s wife.

A. Extended-Brennan-Greenstadt corpus

We used the Brennan-Greenstadt adversarial cor-pus for this study8. This dataset consists of twotypes of writing samples, regular and adversarial,from 12 participants. The regular writing containsapproximately 5000 words of pre-existing writingsamples per author. The regular writings are formalin nature, written for business or academic pur-poses. In the adversarial writing samples, partici-pants performed two adversarial attacks: obfusca-tion and imitation. In the obfuscation attack, eachparticipant tried to hide his identity while writinga 500-word article describing his neighborhood. Inthe imitation attack, each participant tried to hidehis writing style by imitating Cormac McCarthy’swriting style in ‘The Road’ and wrote a 500-wordarticle describing a day of their life in the thirdperson.

6http://weka.sourceforge.net/doc/weka/classifiers/trees/J48.html7http://groups.yahoo.com/group/alternate-history/8This data set is publicly available at https://psal.cs.drexel.edu

We extended this corpus by collecting similarwriting samples using AMT. We created a HumanIntelligence Task (HIT) where participants wereasked to submit the three kinds of writing sampledescribed in the previous paragraph9. After collect-ing the data, we manually verified each submissionand only accepted the ones that complied with ourinstructions. 56 participants’ work was accepted.

Participants were also asked to provide their de-mographic information. According to the provideddemographic information, all the participants’ na-tive language is English and all of them have somecollege-level degree.

A total of 68 authors’ writing samples areused in this study, 12 of the authors are fromthe Brennan-Greenstadt corpus and others are theAMT workers.

B. Hemingway-Faulkner Imitation corpusThe Hemingway-Faulkner Imitation corpus con-

sists of the winning articles from the Faux FaulknerContest and International Imitation HemingwayCompetition10. The International Imitation Hem-ingway Competition is an annual writing com-petition where participants write articles by im-itating Ernest Hemingway’s writing style. In theFaux Faulkner Contest participants imitate WilliamFaulkner’s artistic style of writing, his themes, hisplots, or his characters. Each article is at most500 words long. We collected all publicly availablewinning entries of the competitions from 2000 to2005. The corpus contains sixteen 500-word ex-cerpts from different books of Ernest Hemingway,sixteen 500-word excerpts from different books ofWilliam Faulkner, 18 winning articles from TheInternational Imitation Hemingway Competitionand 15 winning articles from The Faux FaulknerContest.

In the imitation contests, participants chose dif-ferent topics and imitated from different novels ofthe original authors. Table II, III, and IV show im-itation samples. Cormac McCarthy imitation sam-ples are all of same topic but the contest articles

9https://www.cs.drexel.edu/∼sa499/amt/dragonauth index.php

10Available at http://web.archive.org/web/20051119135221/http://www.hemispheresmagazine.com/fiction/2005/hemingway.htm

are of varied topics and most of winners wereprofessional writers.

Table II: Imitation samples from the Extended-Brennan-Greenstadt dataset.

Cormac McCarthy imitation sample: 1Laying in the cold and dark of the morning, the man washuddled close. Close to himself in a bed of rest. Still asleep,an alarm went off. The man reached a cold and pallid armfrom beneath the pitiful bedspread.Cormac McCarthy imitation sample: 2She woke up with a headache. It was hard to tell if what hadhappened yesterday was real or part of her dream because itwas becoming increasingly hard to tell the two apart. The dayhad already started for most of the world, but she was juststumbling out of bed. Across the hall, toothbrush, shower.

Table III: Imitation samples from the InternationalImitation Hemingway Competition.

Hemingway imitation sample: 1At 18 you become a war hero, get drunk, and fall in love witha beautiful Red Cross nurse before breakfast. Over absinthesyou decide to go on safari and on your first big hunt you bagfour elephants, three lions, nine penguins, and are warnednever to visit the Bronx Zoo again. Later, you talk about thewar and big rivers and dysentery, and in the morning youhave an urge to go behind a tree and get it all down on paper.Hemingway imitation sample: 2He no longer dreamed of soaring stock prices and of thethousands of employees who once worked for him. He onlydreamed of money now and the lions of industry: John D.Rockefeller, Jay Gould and Cornelius Vanderbilt. They playedin the darkened boardrooms, gathering money in large piles,like the young wolves he had hired.

C. Long Term Deception: Thomas-Amina Hoaxcorpus

In 2010, a 40-year old US citizen Thomas Mac-Master opened a blog “A Gay Girl in Damacus”where he presented himself as a Syrian-Americanhomosexual woman Amina Arraf and publishedblogposts about political and social issues in Syria.Before opening the blog, he started posting asAmina Arraf in the alternate-history Yahoo! groupsince early 2006. We collected twenty 500-wordposts of Amina and Thomas from the alternate-history Yahoo! group, publicly available articleswritten by Britta Froelicher11 who was a suspect of

11One such article: http://www.joshualandis.com/blog/?p=1831

Table IV: Imitation samples from the FauxFaulkner Contest.

William Faulkner imitation sample: 1And then Varner Pshaw in the near dark not gainsaying theother but more evoking privilege come from and out of thevery eponymity of the store in which they sat and the otheragain its true I seen it and Varner again out of the neardark more like to see a mule fly and the other himself nowresigned (and more than resigned capitulate vanquished by thebovine implacable will of the other) Pshaw in final salivaryresignation transfixed each and both together on a glowingbox atop the counter.William Faulkner imitation sample: 2From a little after breakfast until almost lunch on that longtumid convectionless afternoon in a time that was unencum-bered by measure (and before you knew to call it time: whenit was just the great roiling expressionless moment knownonly elliptically and without reference to actual clocks orwatches as When We Were Very Young) Piglet, his eyesneither seeing nor not-seeing, stood motionless as if rivetedto the iron landscape from which he had equally motionlesslyemerged until he became the apotheosis of all tiny pigswearing scarves standing on two legs and doing absolutelynothing but blinking at what lay before them in the dust.

this hoax and 142 blog posts from “A Gay Girl inDamacus.” The blog posts were divided into 500-word chunks. In total we had 248 articles.

VI. EVALUATION AND RESULTS

A. Evaluation Methodology

To evaluate our approach, we perform a threeclass classification, where the three classes areRegular, Imitation, and Obfuscation, with three fea-ture sets. We use 10-fold cross-validation with Sup-port Vector Maching (SVM) and J48 Decision Treeclassifier. In k-fold cross-validation the originaldataset is randomly partitioned into k equal foldsor subsets. Then the classifier is trained on k-1subsets and tested on the remaining one subset. Thecross-validation process is then repeated k times(the folds), with each of the k folds used exactlyonce as the validation data. The final estimation ofthe classifier is the average of the k results fromthe folds.

We also split the Writeprints feature set intothree sets, lexical, syntactic, and content specificand run the three class classification using each setseparately. In this experiment, we show that non-specific features are as effective as content specific

features in detecting adversarial attacks.

B. Results

1) Can we detect stylistic deception in docu-ments?: The results show that a classifier trainedon sets of adversarial and non-adversarial docu-ments can detect deceptive documents with 96.6%accuracy on our best feature set, as is shown inTable V.

The classification was performed on theBrennan-Greenstadt dataset, Amazon MechanicalTurk dataset, and Extended-Brennan-Greenstadtdataset which combines both of the sets. On all ofthe datasets, the Writeprints features showed thebest performance in detecting adversarial attacks.With this feature set, an SVM classifier can detectimitation attacks with 85% accuracy and obfusca-tion attacks with 89.5% accuracy on the Extended-Brennan-Greenstadt dataset.

Though deception in writing style is significantlydifferent from lying, both deceptions have similarcharacteristics. With the Lying-detection featureswe can detect imitation attacks with 75.3% accu-racy and obfuscation attacks with 59.9% accuracy.

The 9-feature set, which can detect authorshipof regular documents with over 90% accuracy,performed poorly (less than 50%) in detectingadversarial attacks.

The type of machine learning method usedin classification is another important factor indetecting deception. The SVM classifier workedbest with the Writeprints features whereas theJ48 decision tree performed well with the Lying-detection features.

2) Which linguistic features indicate stylisticdeception?: To understand the effect of differentfeatures, we ranked the Writeprints features basedon their Information Gain Ratio (IGR) [21]. IGRof a feature Fi is defined as,

IGR(Fi) = (H(D)−H(D|Fi))/H(Fi),

where D is document class and H is entropy.We used WEKA to calculate IGR. The top featuresare mostly function words, as shown in Table VI.

Other than function words, some syntactic featuressuch as personal pronoun, adverbs, adjectives, andaverage word length were some of the most dis-criminating features.

In our dataset, the non-content-specific featuresperformed similar to the content-specific featuresin detecting deception, as shown in Figure 1,which suggests the possibility of generalizing thesefeatures to detect multiple forms of adversarialattacks.

Table VI: This table shows the features that dis-criminate deceptive documents from regular docu-ments. The top discriminating features according toInformation Gain Ratio are mostly function words.

Top 20 featuresImitated documents Obfuscated documentswhats alotatop nearlately upwanna theresunderneath thousandanymore oursbeside shallshe thatsherself cuzbeneath whatslike haventhe Frequency of commatill lotsher tonsonto anywaysoon plusFrequency of dot otherPersonal pronoun maybe

3) Which features do people generallychange in adversarial attacks and whichfeatures remain unchanged?: We analysedthe Extended-Brennan-Greenstadt dataset tounderstand which features people change instylistic deception. We computed change ina feature f (Cf ) in regular and adversarialdocuments using the following formula:

Cf = 100 ∗ (fadv − freg)/(freg + 1) (1)

where, fadv and freg are the average values offeature f in the adversarial documents and regulardocuments respectively. We added 1 with freg in

Table V: The table shows performance of different feature sets in detecting regular and adversarial writingsamples. The Writeprints feature set with SVM classifier provides the best performance in detectingdeception.

Dataset Feature set, Classifier Type Precision Recall F-measure Overall F-measureRegular 97.5% 98.5% 98%

Writeprints, SVM Imitation 87.2% 82.9% 85% 96.6%Obfuscation 93.2% 86.1% 89.5%

Extended-Brennan-GreenstadtRegular 95.2% 96.2% 95.7%

Lying-detection, J48 Imitation 80.6% 70.7% 75.3% 92%Obfuscation 60.3% 59.5% 59.9%

Regular 92.3% 96.8% 94.5%9-feature set, J48 Imitation 52.9% 43.9% 48% 89%

Obfuscation 61.9% 32.9% 43%Regular 96.5% 98.6% 97.5%

Writeprints, SVM Imitation 82.3% 72.9% 77.3% 95.6%Obfuscation 96.4% 79.1% 86.9%

Amazon Mechanical TurkRegular 94.2% 96.2% 95.2%

Lying-detection, J48 Imitation 71.7% 54.3% 61.8% 90.9%Obfuscation 58.5% 56.7% 57.6%

Regular 92.5% 96.3% 94.3%9-feature set, J48 Imitation 45.5% 35.7% 40% 88%

Obfuscation 45.5% 29.9% 36%Regular 94% 100% 96.9%

Writeprints, SVM Imitation 100% 83.3% 90.9% 94.7%Obfuscation 100% 50% 66.7%

Brennan-GreenstadtRegular 90% 92.9% 91.4%

Lying-detection, J48 Imitation 90.9% 83.3% 87% 85.3%Obfuscation 11.1% 8.3% 9.5%

Regular 89.4% 93.7% 91.5%9-feature set, J48 Imitation 25% 25% 25% 84%

Obfuscation 83.3% 41.7% 55.6%

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Regular Imita8on Obfusca8on

FMeasure

TypesofWri0ng

LexicalFeatures

Syntac8cFeatures

Content‐SpecificFeatures

Figure 1: Effect of different feature types in detect-ing adversarial attacks. In each case, SVM is usedfor classification.

the denominator to avoid divide-by-zero error, asfreg can be zero for some features.

Figure 2 shows the mostly changed featuresamong parts-of-speech and 9-features. In Figure 3,

we grouped similar Writeprints features togetherand added their corresponding changes to showthe overall change. In both graphs, the y-axisrepresents a list of features that have been adjustedin the passages and the x-axis represents the changein each feature.

In our experiments, participants used more ex-istential there12 and adverbs in obfuscated pas-sages. We also noticed an increase in the usageof particles and personal pronouns and decreasein the usage of noun, and Wh-pronoun. We do seehints that authors are “dumbing down” their writingstyle with shorter sentences, simpler words withless syllables, lower readability scores and higherreadability ease (Figure 2). Another interestingresult is that participants used a large number of

12Existential there (EX) is the unstressed there that triggersinversion of the inflected verb and the logical subject of asentence [24]. For example: There/EX were lots of people intown.

‐80 ‐60 ‐40 ‐20 0 20 40 60 80 100

Avg.sentencelengthGFreadabilityindex

CardinalnumberAdjecAve

NounAveragewordlength

AveragesyllablesperwordExistenAalthere

AdverbUniquewords

VerbShortWords

ParAcleSentencecount

Personalpronoun

ImitaAon

ObfuscaAon

Figure 2: Mostly changed features among parts-of-speech and 9-features.

‐400 ‐200 0 200 400 600 800

WordbigramWordtrigram

CharacterbigramPercentuppercase

AveragewordlengthPercentdigits

CharacterunigramTotalWordsShortWordsPunctuaAon

Parts‐of‐speechCharactertrigramFuncAonwordsWordunigram

ImitaAon

ObfuscaAon

Figure 3: Overall changes in the Writeprints fea-tures.

function words to obfuscated their writings (Figure3). This suggests changing function words as onepossible way of anonymizing a document. In ourdataset, the function words that were used more inobfuscated documents than regular documents are:I, my, there, are, you. The function words that wereused less are: as, his, by, her, be.

In imitated passages, more usage of personalpronouns, particles and less usage of adjectiveswere noticed (shown in Figure 2). Participants

also used shorter sentences and words with lesssyllables which is why the imitated passages hadless readability score (Figure 2). The personalpronouns can likely be attributed to the directionof the writing survey which asked participants todescribe their day in the third person. The shortersentences are likely a result of imitating CormacMcCarthy who often uses short sentences in hisprose. Participants were found to use more functionwords than regular in this case also, although notas much as in the obfuscated documents.

Information Gain Ratio (IGR) and usage changeof a feature are positively correlated (shown inFigure 5 and Figure 4). All of the highly changedfeatures have high IGR and unchanged featureshave zero IGR. But some function words werefound to have high information gain ratio in spiteof being changed very little.

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0 20 40 60 80 100

Inform

a(on

GainRa

(o

Featurechangeintheobfuscatedtext

Figure 4: Relation between Information Gain Ratioand absolute usage change in features in the obfus-cated documents. The black line represents trend.

4) Does stylistic deception share similarcharacteristics with other deceptions?: Stylisticdeception shares similar characteristics with lying.As shown in Table V, features used to detect lyingcan detect imitation attacks with 75.3% accuracyand obfuscation attacks with 59.9% accuracy onthe Extended-Brennan-Greenstadt dataset. Thisresult indicates the similarity of linguistic changesassociated with any kind of information hiding.

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 50 100 150 200

Inform

a(on

GainRa

(o

Featurechangeintheimitatedtext

Figure 5: Relation between Information Gain Ratioand absolute usage change in features in the imi-tated documents. The black line represents trend.

5) Are some adversarial attacks moredifficult to detect than others?: In ourexperiment, obfuscation was harder to detectthan imitation. In imitation, participants followedone specific writing style, the writing style ofCormac McCarthy. Different people followeddifferent linguistic aspects in imitating him, forexample, some participants used short sentences,some used descriptive adjectives and some used aconversational format with dialogs. But the overallwriting style was limited to the style of CormacMcCarthy. Obfuscation is different than imitationas in obfuscation an author can choose to imitatemore than one authors’ writing style or develop anew style different from his own. However, whenwe include multiple imitated authors it becomescorrespondingly more difficult to detect imitationattacks.

6) Can we generalize deception detection?:We check whether our deception detection ap-proach that can detect imitation and obfuscationon the Extended-Brennan-Greenstadt can detectimitation samples from the Ernest Hemingway andWilliam Faulkner imitation contests. We performeda 10-fold cross-validation on the Hemingway-Faulkner imitation corpus. We used Writeprintsand Lying-detection features with SVM and J48classifiers respectively from the WEKA tool. Our

classifier can distinguish imitated articles fromthe original writings of Ernest Hemingway andWilliam Faulkner with 88.6% accuracy (Table VII).

Table VII: Imitated document prediction result:Hemingway-Faulkner imitation corpus. (P = Pre-cision, R= Recall and F= F-measure)

Type Lying-detection, J48 Writeprints, SVMP R F P R F

Imitation 69.7% 69.7% 69.7% 83.8% 93.9% 88.6%Regular 61.5% 61.5% 61.5% 92.6% 80.6% 86.2%

Weighted Avg. 66.1% 66.1% 66.1% 88.1% 87.5% 87.4%

We also performed an experiment where a clas-sifier trained on the Extended-Brennan-Greenstadtdataset was tested on Hemingway-Faulkner Imita-tion corpus. Only 57.1% of the imitated documentswere considered as imitation in that case. TheHemingway-Faulkner Imitation corpus is differentfrom our dataset in several ways. The participantsin the training set imitated Cormac McCarthyusing one pre-specified excerpt from ‘The Road’in writing about their day. But in the imitationcontests, participants imitated two different authorswithout any topic constraint. Also the contest win-ners were found to be more successful than themechanical turkers in imitating, as shown in TableVIII. To see how often a classifier can be fooledinto predicting imitated document as written bythe original authors, we trained an SMO SVMclassifier with the Writeprints features using theoriginal writing excerpts of Cormac McCarthy,Ernest Hemingway, William Faulkner and testedthe classifier with the imitated documents. In thistest, we classified imitated documents into threeclasses: Cormac McCarthy, Ernest Hemingway,William Faulkner. The result shows that the contestwinners can imitate Ernest Hemingway in 84.27%cases, and William Faulkner in 66.67%, whereasthe turkers were successful in 47.05% cases inimitating Cormac McCarthy.

C. Detecting long term deception

Detecting long term deception is similar to de-tecting fiction as deception. Fiction and elaboratedeception have different linguistic characteristics

Table VIII: This table shows the success rate(Precision) of participants in imitating differentauthors. Imitation contest winners were more suc-cessful in imitating than the AMT participants.

Author name Imitationsuccessrate

Writer’s skill

CormacMcCarthy

47.05% Not professional writers

ErnestHemingway

84.21% Writing contest winners

William Faulkner 66.67% Writing contest winners

than short-term on-the-spur deception, as in thelong-term deception the author has sufficient timeand topic to write descriptively and edit sufficientlyto make it appear as a truthful document. This iswhy a different approach is required to detect long-term hoaxes and deception. Regular authorshiprecognition can be helpful to find inconsistenciesin writing and to discover real authorship of thedeceptive documents.

To test our method on long-term deception,we used the Thomas-Amina Hoax corpus. Weperformed an authorship attribution test on theposts he created as himself and as Amina in thealternate-history Yahoo! group. We found that heconsistently maintained a separate writing style asAmina in the Yahoo! group as none of the Amina’sposts were attributed to Thomas in the authorshipattribution test. Thomas’s writing style as Aminawas different than his regular writing style. Theuse of upper-case letters and modal verbs 13 werenoticeable in Amina’s posts, whereas Thomas usedlonger sentences and more adjective and adverbs.Table IX lists the top Writeprints features thatdiscriminate Thomas and Amina’s posts in theYahoo! group.

Moreover, all of the posts Thomas wrote asAmina and as himself and posts of Britta wereconsidered as regular when tested on an SVMclassifier which was trained with the Extended-Brennan-Greenstadt corpus. Deception classifica-tion of the posts from “A Gay Girl in Damascus”

13Modal verbs are verbs that do not take an -s ending in thethird person singular present, i.e. can, could, may, might, ought.

Table IX: The top features according to Infor-mation Gain Ratio that discriminate Thomas andAmina.

Feature Information Gain RatioPercent upper case 0.758t 0.620than 0.620all 0.516only 0.516tha 0.501though 0.501Frequency of * 0.432less 0.433can 0.433not 0.433a 0.423that 0.423Modal verb 0.423that 0.423them 0.373even 0.372doing 0.361

also did not show any indication of masking. Inour test, only 14% of the blog posts were con-sidered as deceptive which is less than the errorrate, suggesting a random effect. 13 blog postswere classified as obfuscated documents, 22 wereclassified as imitated document. Table X shows that57.14% of the deceptive documents were attributedto Amina during authorship attribution.

Table X: Blog post classification.

Thomas AminaImitation 11 11Obfuscation 4 9

But maintaining an alternate writing style con-sistently for a long time is hard, which was evidentin the Thomas-Amina case. When people startedquestioning Amina’s existence, Thomas and hiswife Britta were suspected as possible writers ofthe blog based on various pieces of evidence, forexample, Thomas’s address was used in Amina’saccount, and photos from Britta’s picasa albumwere used in Amina’s blog. In the end, Thomasadmitted that he was “Amina.” Regular authorshiprecognition also supports this fact. More than halfof the blog posts (54.03%) were attributed to

Thomas during authorship attribution with an SVMclassifier and the Writeprints feature set. Only 10posts were attributed to Britta and the rest wereattributed to “Amina.” Native language detection,age and other demographics analysis are otherpossible ways to detect this form of deception,which are not explored in this paper.

VII. DISCUSSION

In this research, we showed that two kinds ofadversarial attacks—imitation and obfuscation—can be detected with high accuracy using a largefeature set. However, the results of cross-validationon the Extended-Brennan-Greenstadt corpus areconsiderably better than the results on the imitationcontest participants (96.6% vs. 87.4%).

There are several potential explanations for thisdiscrepancy. The first is that the original datasetonly handles a single imitated author, Cormac Mc-Carthy, and the imitation classifier may be overfitto detect imitations of his style. This suggeststhat training on a combination of different authorimitations will produce a stronger general classifierand that an imitation classifier should be trained onthe target of the attack, if known.

The second potential explanation is the skilland incentive differential between the AMT par-ticipants and the contest winners. We includedall reasonable attempts by the mechanical turkparticipants, but only the top three contestants peryear were included in the contest data set. Weoffered our participants $5. A Faulker contest win-ner receives airfare for two to Memphis on UnitedAirlines, five days free car rental, two free passesto the Faulkner Conference, and a complimentaryroom for five days and nights at an Oxford motel.First prize in the Heminway contest was round-triptickets and dinner for two at Harry’s in Florence,Italy.

A third potential explanation is that part of thesuccess on the original data set is due to topicdetection, not deception detection. In the Extended-Brennan-Greenstadt corpus, regular documents arefrom a variety of documents, obfuscation docu-ments are a description of a neighborhood, and im-itation documents provide a third-person narrative

of a day.Our analysis of the feature set shows that the

non-content specific features have the same accu-racy as of the content-specific features (Figure 1).Also, as most top ranked discriminating featuresare function words, even by ignoring contextualsimilarity of the documents, it is possible to detectadversarial documents with sufficient accuracy.

While it is true that content features may in-dicate authorship or deception, we do not believethis is the case here. Our non-deceptive writingsamples consist of multiple documents per author,yet our authorship recognition techniques identifythem properly with high levels of accuracy. Thedifferent content features there did not dissuadethe standard authorship recognition techniques andwe do not believe they greatly alter the outcomeof the deception analysis. Furthermore, previouslinguistic research has shown that the frequenciesof common function words are content neutral andindicative of personal writing style [15].

What the “A Gay Girl in Damascus” resultsshow is that obfuscation is difficult to maintain inthe long term. While Tom’s posts as Amina werenot found to be deceptive by our classifier, we showthat traditional authorship attribution techniqueswork in this case.

Implications for Future Analyses: The currentstate of the art seems to provide a perfect balancebetween privacy and security. Authors who aredeceptive in their writing style are difficult toidentify, however their deception itself is oftendetectable. Further, the detection approach worksbest in cases where the author is trying fraudulentlypresent themselves as another author.

However, while we are currently unable to un-mask the original author of short term deceptions,further analyses might be able to do so, especiallyonce a deception classifier is able to partition thesets. On the other hand, the Extended-Brennan-Greenstadt data set used contains basic attacks byindividuals relying solely on intuition (they haveno formal training or background in authorshipattribution) and the results on the more skilledcontest winners are less extensive.

We are currently working on a software appli-

cation to facilitate stylometry experiments and aidusers in hiding their writing style. The softwarewill point out features that are identifying to usersand thus provide a mechanism for performingadaptive countermeasures against stylometry. Thistool may be useful for those who need longerterm anonymity or authors who need to maintaina consistent narrative voice. Even though ThomasMacMaster proved extremely skilled in hiding hiswriting style, half his posts were still identifiableas him rather than the fictional Amina.

In addition, these adaptive attacks may be able tohide the features that indicate deception, especiallythose in our top 20. It is also possible that attemptsto change these features will result in changes thatare still indicative of deception, particularly in thecase of imitations. The fact is that most peopledo not have enough command of language toconvincingly imitate the great masters of literatureand the differences in style should be detectableusing an appropriate feature set. A broader setof imitated authors is needed to determine whichauthors are easier or harder to imitate. Despite ourability to communicate, language is learned on anindividual basis resulting in an individual writingstyle [11].

Implications for Adversarial Learning: Ma-chine learning is often used in security problemsfrom spam detection, to intrusion detection, to mal-ware analysis. In these situations, the adversarialnature of the problem means that the adversary canoften manipulate the classifier to produce lowerquality or sometimes entirely ineffective results.In the case of adversarial writing, we show thatusing a broader feature set causes the manipulationitself to be detectable. This approach may be usefulin other areas of adversarial learning to increaseaccuracy by screening out adversarial inputs.

VIII. CONCLUSION

Stylometry is necessary to determine authentic-ity of a document to prevent deception, hoaxes andfrauds. In this work, we showed that manual coun-termeasures against stylometry can be detectedusing second-order effects. That is, while it maybe impossible to detect the author of a document

whose authorship has been obfuscated, the ob-fuscation itself is detectable using a large featureset that is content-independent. Using InformationGain Ratio, we showed that the most effective fea-tures for detecting deceptive writing are functionwords. We analysed a long-term deception andshowed that regular authorship recognition is moreeffective than deception detection to find indicationof stylistic deception in this case.

IX. ACKNOWLEDGEMENT

We are thankful to the Intel Science and Tech-nology Center (ISTC) for Secure Computing andDARPA (grant N10AP20014) for supporting thiswork. We also thank Ahmed Abbasi for clarifyingthe Writeprints feature set and our colleagues AylinCaliskan, Andrew McDonald, Ariel Stolerman andanonymous reviewers for their helpful commentson the paper.

REFERENCES

[1] Ahmed Abbasi and Hsinchun Chen. Writeprints: Astylometric approach to identity-level identificationand similarity detection in cyberspace. ACM Trans.Inf. Syst., 26(2):1–29, 2008.

[2] M. Brennan and R. Greenstadt. Practical attacksagainst authorship recognition techniques. In Pro-ceedings of the Twenty-First Conference on Inno-vative Applications of Artificial Intelligence (IAAI),Pasadena, CA, 2009.

[3] J. Burgoon, J. Blair, T. Qin, and J. Nunamaker.Detecting deception through linguistic analysis.Intelligence and Security Informatics, pages 958–958, 2010.

[4] K Calix, M Connors, D Levy, H Manzar, G MCabe,and S Westcott. Stylometry for e-mail authoridentification and authentication. Proceedings ofCSIS Research Day, Pace University, 2008.

[5] Carole E. Chaski. Who’s at the keyboard: Author-ship attribution in digital evidence investigations.In 8th Biennial Conference on Forensic Linguis-tics/Language and Law, 2005.

[6] M.G. Frank, M.A. Menasco, and M. O’Sullivan.Human Behavior and Deception Detection.

[7] M. Hall, E. Frank, G. Holmes, B. Pfahringer,P. Reutemann, and I.H. Witten. The weka datamining software: an update. ACM SIGKDD Explo-rations Newsletter, 11(1):10–18, 2009.

[8] J.T. Hancock, L.E. Curry, S. Goorha, and M. Wood-worth. On lying and being lied to: A linguis-tic analysis of deception in computer-mediatedcommunication. Discourse Processes, 45(1):1–23,2008.

[9] D.I. Holmes and R.S. Forsyth. The federalistrevisited: New directions in authorship attribution.Literary and Linguistic Computing, 10:111–127,1995.

[10] P. Juola and D. Vescovi. Empirical evaluation ofauthorship obfuscation using JGAAP. In Proceed-ings of the 3rd ACM workshop on Artificial Intel-ligence and Security, pages 14–18. ACM, 2010.

[11] Patrick Juola. Authorship attribution. Foundationsand Trends in information Retrieval, 1(3):233–334,2008.

[12] Gary Kacmarcik and Michael Gamon. Obfus-cating document stylometry to preserve authoranonymity. In Proceedings of the COLING/ACLon Main conference poster sessions, pages 444–451, Morristown, NJ, USA, 2006. Association forComputational Linguistics.

[13] Moshe Koppel, Jonathan Schler, Shlomo Argamon,and Eran Messeri. Authorship attribution withthousands of candidate authors. In Proceedings ofthe 29th annual international ACM SIGIR confer-ence on Research and development in informationretrieval, SIGIR ’06, pages 659–660, New York,NY, USA, 2006. ACM.

[14] M.B. Malyutov. Information transfer and com-binatories. Lecture Notes in Computer Science,4123(3), 2006.

[15] F. Mosteller and D. Wallace. Inference and dis-puted authorship: The federalist. 1964.

[16] Mihir Nanavati, Nathan Taylor, William Aiello, andAndrew Warfield. Herbert westdeanonymizer. In6th Usenix Workshop on Hot Topics in Security(HotSec), 2011.

[17] A. Narayanan, H. Paskov, N. Gong, J. Bethencourt,E. Stefanov, R. Shin, and D. Song. On thefeasibility of internet-scale author identification.In Proceedings of the 33rd conference on IEEESymposium on Security and Privacy. IEEE, 2012.

[18] Michael P. Oakes. Ant colony optimisation for sty-lometry: The federalist papers. Proceedings of the5th International Conference on Recent Advancesin Soft Computing, pages 86–91, 2004.

[19] J.W. Pennebaker, R.J. Booth, and M.E. Francis.Linguistic inquiry and word count (LIWC2007).Austin, TX: LIWC (www. liwc. net), 2007.

[20] J.C. Platt. Sequential minimal optimization: A fastalgorithm for training support vector machines. Ad-vances in Kernel MethodsSupport Vector Learning,208(MSR-TR-98-14):1–21, 1998.

[21] J.R. Quinlan. Induction of decision trees. Machinelearning, 1(1):81–106, 1986.

[22] J.R. Quinlan. C4. 5: programs for machine learn-ing. Morgan kaufmann, 1993.

[23] Josyula R. Rao and Pankaj Rohatgi. Canpseudonymity really guarantee privacy? InSSYM’00: Proceedings of the 9th conference onUSENIX Security Symposium, Berkeley, CA, USA,2000. USENIX Association.

[24] B. Santorini. Part-of-speech tagging guidelines forthe Penn Treebank Project (3rd revision). 1990.

[25] Harold Somers and Fiona Tweedie. Authorshipattribution and pastiche. Computers and the Hu-manities, 37:407–429, 2003.

[26] M. Steller and JC Yuille. Recent developments instatement analysis. Credibility assessment, pages135–154, 1989.

[27] Fiona J. Tweedie, S. Singh, and D.I. Holmes.Neural network applications in stylometry: Thefederalist papers. Computers and the Humanities,30(1):1–10, 1996.

[28] Uzlem Uzuner and Boris Katz. A comparativestudy of language models for book and authorrecognition. In IJCNLP, page 969, 2005.

[29] James Wayman, Nicholas Orlans, Qian Hu,Fred Goodman, Azar Ulrich, and ValorieValencia. Technology assessment for thestate of the art biometrics excellence roadmap.http://www.biometriccoe.gov/SABER/index.htm,March 2009.

[30] R. Zheng, J. Li, H. Chen, and Z. Huang. Aframework of authorship identification for onlinemessages: Writing style features and classificationtechniques. Journal American Society for Informa-tion Science and Technology, 57(3):378–393, 2006.