The world is y0ur$: Geolocation-based wordlist generation with wordsmith

THE WORLD IS Y0UR$:

GEOLOCATION-BASED WORDLIST

GENERATION WITH WORDSMITH

S A N J I V K AWA | TO M P O RT E R

@ h a c k e r j i v | @ p o r t e r h a u 5

❯ whoami

2

Sanjiv Kawa@hackerjiv

S R . P E N E T R A T I O N T E S T E R

P S C / N C C G R O U P

• Roots in dev and IT

• Penetration testing

• Binary analysis and exploit dev

• Canadian

❯ su porterhau5

3

Tom Porter@porterhau5

S R . S E C U R I T Y C O N S U L T A N T

F U S I O N X R E D T E A M

• Flow data analytics

• Penetration testing

• Red teaming

• BloodHound extensions

What is Wordsmith?

4

Custom wordlist generation Crack hashes / password attacks

Tailored for your target

Geo-location data Modular and extensible

Username generation

Wordsmith v1

5

Wordsmith v1: Geo-location Data Collected

6

Major league sports teams

Colleges and universities

Common names

Area codesZip codes

Streets and roads

Landmarks

Cities, towns, etc

Wordsmith v1: Additional Features

7

CeWL IntegrationBasic mangling (whitespace, specials, split on space)

Specify minimum

character lengthTo lowercase[a-z]

Wordsmith v1: Things we learned

8

Feedback from the community was incredible. Thank you!

Top three requests:

1. More countries need to be available (v1 was US only)2. Needs to be a way to introduce more/your own data3. Limited to the English language

Wordsmith v2

9

New CLI design

Multi-language

(13 so far! – UTF-8)

Introduced religions

Generate usernames

Modular framework allows for user contribution and extensibility

Geo-location data sets for over 230 countries!

Data Sources

Coverage: World

Data types: Population, Religion, Languages, etc

10

www.cia.gov/library/publications/the-world-

factbook/geos/print_[aa-zz].html

Coverage: 13 languages (hunspell)

Data Sources

11

Coverage: US

Data Types: Sports teams, colleges

Coverage: World

Data Types: Landmarks and archeological sites

Coverage: World

Data Types: Religious texts

Data Sources

12

Coverage: World

Data Types: Roads, Cities, Counties

Coverage: US

Data Types: Popular first names. Last names

Coverage: US

Data Types: Area Codes, Zip Codes

How to get Wordsmith

13

❯ git clone https://github.com/skahwah/wordsmith.git

❯ cd wordsmith

❯ bundle install # (optional for CeWL integration)

❯ ruby wordsmith.rb

wordsmith v2.0.7Written by: Sanjiv "Trashcan Head" Kawa & Tom "Pain Train" PorterTwitter: @hackerjiv & @porterhau5

[*] Hello new wordsmither!

[*] This script will remove the data/ directory in the current working directory. Enter 'y' to continue: y

[*] Just need to unpack some files (Running: tar -xf data.tar.xz)

[*] Unpack completed!

[*] CeWL found: /usr/bin/cewl

Files

14

❯ ls -l

-rw-r--r-- 1 user staff 3159 Oct 1 22:57 CHANGELOG.mddrwxr-xr-x 2 user staff 4096 Oct 1 22:57 data-rw-r--r-- 1 user staff 50602888 Oct 1 22:57 data.tar.xz-rw-r--r-- 1 user staff 116 Oct 1 22:57 Gemfile-rw-r--r-- 1 user staff 1393 Oct 1 22:57 LICENSE-rw-r--r-- 1 user staff 7514 Oct 1 22:57 README.md-rwxr-xr-x 1 user staff 31081 Oct 1 22:57 wordsmith.rb

• View README first, or check out –E option (examples)

• wordsmith.rb: primary ruby script

• data.tar.xz (~50 MB): compressed archive of data

• data/ (~250 MB): data arranged in hierarchy

Boundaries & Attributes

15

Boundaries (-I <input>)

• Areas of the world to get

words for

• 249 countries and

territories

• States/Provinces

• Cities

• Custom regions

Attributes (ex: -r -l)

• Types of words to grab:

• Cities

• Colleges

• Landmarks

• Languages

• Names

• Roads

• Religions

• and more…

❯ ruby wordsmith.rb –I usa –r –l

Structure

16

❯ ls data/

abw afg ago aia ala alb and are arg arm ... wlf wsm yem zaf zmb zwe

ISO ALPHA-3 Country Codes

❯ ls data/usa

ak al ar az ca cia.txt co ct dc ... tx usa.yaml ut va vt wa wi wv wy

States, Provinces, Counties, Municipalities

❯ ls data/usa/nc

areacodes.txt charlotte cities.txt colleges.txt counties.txt ...

Cities, Counties

❯ ls data/usa/nc/charlotte

sports.txt

Attributes (sports, colleges, roads, etc.) are .txt files

Boundaries and Input

17

❯ ruby wordsmith.rb –I usa [options]

❯ ruby wordsmith.rb –I usa-nc [options]

❯ ruby wordsmith.rb –I usa-nc-charlotte [options]

❯ ruby wordsmith.rb –I usa,can [options]

❯ ruby wordsmith.rb –I usa-sd,usa-nd,usa-co [options]

-I for specifying input boundaries

Can supply one or many boundaries

❯ ruby wordsmith.rb –I 10 [options]

Providing a number (ex: 10) will select N most populous countries

Regions

18

❯ ruby wordsmith.rb –I europe [options]

❯ grep europe data/regions.csv

europe,"Continent of Europe",ala alb and arm aut aze bel bgr bih blr checyp cze deu dnk esp est fin fra fro gbr geo ggy gib grc hrv hun imn irlisl ita jey kaz lie ltu lux lva mco mda mkd mlt mne nld nor pol prt rourus sjm smr srb svk svn swe tur ukr vat

regions.csv contains custom grouping of boundaries

Can see regions with -R option:

❯ ruby wordsmith.rb –R

Alias: newenglandDescription: US - New EnglandMembers: usa-ct usa-me usa-ma usa-nh usa-ri usa-vt

Alias: plainsDescription: US - PlainsMembers: usa-ia usa-ks usa-mn usa-mo usa-ne usa-nd usa-sd

Alias: greatlakesDescription: US - Great LakesMembers: usa-il usa-in usa-mi usa-oh usa-wi

Attributes

19

❯ ruby wordsmith.rb –I europe [options]

❯ ruby wordsmith.rb –h

Main Arguments:-I, --input <input> Comma-delimited list of inputs

Input Options:-a, --all Grab all options-b, --other Grab other miscellaneous attributes-e, --cia Grab demographics compiled by the CIA-c, --cities Grab all city names-f, --colleges Grab all college sports-l, --landmarks Grab all landmarks-v, --language Grab the most popular language(s)-N, --all-names Grab all first names and last names-G, --first-names Grab all first names-L, --last-names Grab all last names-F, --female-fnames Grab all female first names-M, --male-fnames Grab all male first names-p, --phone Grab all area codes-r, --roads Grab all road names-g, --religion Grab the most popular relgious text(s)-t, --teams Grab all major sports teams-u, --counties Grab all counties-z, --zip Grab all zip codes

Attribute Examples

20

❯ ruby wordsmith.rb –I usa-sd -z57001570025700357004...

Grab all zip codes for South Dakota

❯ ruby wordsmith.rb –I gbr-eng –r –c -lAb KettlebyAbberleyAbbertonAbbess Roding...

Grab all roads, cities, and landmarks for England, GBR

❯ ruby wordsmith.rb –I asia -aAbasAbatanAbbegAbejao...

Grab all attributes for Asia

Child Nodes

21

❯ ruby wordsmith.rb –I gbr –C

Format:boundary-name : attribute1 attribute2 attribute3 etc.

gbr : cities counties landmarks roads cia|-- gbr-sco : cities counties roads|-- gbr-wal : cities counties roads|-- gbr-eng : cities counties roads| |-- gbr-eng-su : cities counties roads| |-- gbr-eng-ch : cities counties roads| |-- gbr-eng-ex : cities roads| |-- gbr-eng-nt : cities counties roads| |-- gbr-eng-sk : cities roads| |-- gbr-eng-ca : cities counties roads| |-- gbr-eng-bu : cities counties roads| |-- gbr-eng-sx| | |-- gbr-eng-sx-east_sussex : cities counties roads| | |-- gbr-eng-sx-west_sussex : cities counties roads...

See the child nodes (-C) and their attributes of a given boundary

Country Metadata

22

❯ ls -l data/jpn/-rw-r--r-- 1 user staff 32002 Aug 30 19:16 cia.txt-rw-r--r-- 1 user staff 13184 Sep 9 2016 cities.txt-rw-r--r-- 1 user staff 5608 Sep 9 2016 counties.txt-rw-r--r-- 1 user staff 107 Aug 30 19:36 jpn.yaml-rw-r--r-- 1 user staff 113672 Oct 1 21:10 landmarks.txt-rw-r--r-- 1 user staff 871994 Sep 9 2016 roads.txt

❯ cat data/jpn/jpn.yaml

config:

population: 126,702,133

language_1: Japanese

religion_1: Shintoismreligion_2: Buddhism

The World Factbook:

PopulationOfficial languagesMost popular religions

Most populous countries (ex: -I 25)Official languages (-v, --language)Most popular religions (-g, --religion)

Religions

23

❯ wc -l data/religion/*

28168 douay-rheims-parsed.txt97682 king-james-bible-book-verse.txt20190 king-james-bible-parsed.txt42876 niv-bible-parsed-spanish.txt34202 niv-bible-parsed.txt7872 quran-parsed-eng.txt

❯ cat king-james-bible-book-verse.txt

The First Book of Moses: Called GenesisGenesis1:11:1GenesisJohn3:163:16John...

❯ cat king-james-bible-parsed.txt ...JesuiteJesusJetherJethethJethro...

(-g, --religion)

Identified the most common religions

• KJV Bible

• NIV Bible

• Douay Rheims

• Quran

~ 200 countries are covered

Languages

24

❯ head –n 5 language-frequency.txt

83:English38:French29:Spanish26:Arabic11:Russian

❯ wc -l data/languages/*.txt

457097 arabic.txt47866 bahasa.txt110750 bengali.txt115485 cedict.txt466544 english.txt72038 french.txt585844 german.txt338534 hebrew.txt15990 hindi.txt95152 italian.txt47866 malay.txt340235 portuguese.txt379324 russian.txt798915 spanish.txt371169 turkish.txt

(-v, --language)

Identified the most common languages

~ 195 countries are covered

Modular Design

25

❯ ls data/usa/mn/

areacodes.txt colleges.txt fnames.txt landmarks.txt sports.txtcities.txt counties.txt lakes.txt roads.txt zipcodes.txt

❯ cat data/usa/mn/lakes.txt

AaronAbbeyAcornAdelman's Pond...

❯ ruby wordsmith.rb –I usa-mn –b

AaronAbbeyAcornAdelman's Pond...

Modular design:- Easily extensible- Introduce your own .txt files (grab with –b option)- Contribute and help build the project

Output Options

26


<Input options snipped>

Output Options:-o, --output FILE The filename for writing output-q, --quiet Don't show words, use with -o option

-k, --min-length LEN Minimum length of word to include-n, --max-length LEN Maximum length of word to include-D, --complexity Words meet Windows default complexity-j, --lowercase Convert all words to lowercase-w, --specials Add words with special chars removed-x, --spaces Add words with spaces removed-y, --split Split words by space and add-m, --mangle Add all permutations (-w, -x, -y)

-P, --prepend-phones Prepend state area codes to each word-A, --append-phones Append state area codes to each word-X, --prepend-zips Prepend zip codes to each word-Z, --append-zips Append zip codes to each word-W, --prepend-wordlist FILE Prepend words in FILE to each word-Y, --append-wordlist FILE Append words in FILE to each word

Tweaking Output

27

❯ ruby wordsmith.rb –I usa-dc –r

Pennsylvania Ave.

Name of a road generated for D.C.

Mangle (-m): split words, remove specials, remove spaces

❯ ruby wordsmith.rb –I usa-dc –r -m

Pennsylvania Ave.Pennsylvania AvePennsylvaniaAve.AvePennsylvaniaAve.PennsylvaniaAve

❯ ruby wordsmith.rb –I usa-dc –r –m –k 8

Pennsylvania Ave.Pennsylvania AvePennsylvaniaPennsylvaniaAve.PennsylvaniaAve

Min Length (-k): specify minimum char length

Tweaking Output

28

❯ ruby wordsmith.rb –I usa-dc –r –m –D

Pennsylvania Ave.Pennsylvania AvePennsylvaniaAve.

Windows Default complexity (-D): 8 char min, 3/4 cases

❯ ruby wordsmith.rb –I usa-sd –a –q –o SD.txt

cities in ./data/usa/sd: 390colleges in ./data/usa/sd: 37counties in ./data/usa/sd: 66landmarks in ./data/usa/sd: 16fnames in ./data/usa/sd: 2319areacodes in ./data/usa/sd: 1roads in ./data/usa/sd: 15569zipcodes in ./data/usa/sd: 394religions: 145786languages: 1107300

[*] 1252939 words written to: /opt/wordsmith/SD.txt

Quiet output (-q), write results to file (-o sd.txt)

Prepending & Appending

29

• Prepend or Append:• Zip codes (-X,-Z)

• Area codes (-P,-A)

• User-supplied wordlist (-W,-Y)

https://arstechnica.com/tech-policy/2016/08/if-youre-an-alleged-drug-dealer-dont-use-asshole209-as-a-password/

Prepending & Appending

30

❯ cat years.txt

1717!20172017!

years.txt: file I created with words I want to append

❯ ruby wordsmith.rb –I usa-sd –f -m –Y years.txt

...AugustanaAugustana17Augustana17!Augustana2017Augustana2017!BlackHillsBlackHills17BlackHills17!BlackHills2017BlackHills2017!...

Grab colleges (-f), mangle (-m), then append custom wordlist (-Y)

Names

31

❯ cat data/usa/fnames.txtJamesJohnRobertMichaelMary...

❯ cat data/usa/lnames.txtSmithJohnsonWilliamsBrownJones...

• Most common baby names in each state since 1910

• -G: most common first names • -L: most common last names• -N: all names

Username Generation

32


<other options snipped>

Username Generation Options:--filn FirstInitialLastName (bsmith)--fnln FirstNameLastName (bobsmith)--fnli FirstNameLastInitial (bobs)--lnfi LastNameFirstInitial (smithb)--lnfn LastNameFirstName (smithbob)--fidln FirstInitial.LastName (b.smith)--fndln FirstName.LastName (bob.smith)--truncate LEN Truncate username at LEN number of chars (bobsmi)--max-users LEN Max number of usernames to generate--name-depth LEN Num of first/last names to iterate over

(default:100, 0 will get all)

• Generate different username formats

• Use --max-users and --name-depth to handle speed & volume

Username Generation

33

❯ ruby wordsmith.rb –I usa --fnlnJamesSmithJamesJohnsonJamesWilliamsJamesBrownJamesJonesJamesGarciaJamesMiller...

First name Last Name

❯ ruby wordsmith.rb –I usa --fndlnJames.SmithJames.JohnsonJames.WilliamsJames.BrownJames.JonesJames.GarciaJames.Miller...

First name (dot) Last Name

Username Generation

34

❯ ruby wordsmith.rb –I usa –filn –-truncate 8...aDavisaRodriguaMartineaHernandaGonzaleaWilsonaAnderso...

Truncate down to 8 characters

❯ ruby wordsmith.rb –I usa –lnfn –q usernames in ./data/usa: 10000

❯ ruby wordsmith.rb –I usa –lnfn –q --name-depth 250usernames in ./data/usa: 62500

❯ ruby wordsmith.rb –I usa –lnfn –q --name-depth 1000usernames in ./data/usa: 1000000

Adjust --name-depth to generate more usernames

Ireland – Interesting Password Recoveries

36

• Cork1234

• Carlow123

• Dublin1234

• Seapoint1916

• Artane2016

• Templeroan2009

• Donegal56

• ParkLodge30!

• Portishead01

• Tipperary2

• Larkfield18

• Wolseley2014

• Farriers40

• 5RotheAbbey

Multinational Organization Results

37

• Organization has offices in USA, Australia and Canada

• Unable to disclose total number of hashes

Wordlist Hashcat

run time

Number of

passwords recovered

Top 10k

(10k words)

4 sec

Rockyou

(14.4m words)

30 mins

AUS, CAN, USA Wordlist

(7.3m words)

13 mins

256

476

241

ruby wordsmith.rb -I aus,can,usa -a -j -q -m -o aus-can-usa-all-lowercase-q-m.txt

Multinational – Interesting Password Recoveries

38

Australia:

• Bayswater2017

• Primavera001

• Padstow123!

• Queenslander2015

• Razorback1965

• Parramatta16

• Sydney201%

Canada

• !Matthew2222

• Canada1984

• Vancouver186

USA

• Bernie424!

• ColoradoSprings3!

• ChicagoCubs2016

• BostonCeltics29

• Anakin2005s

• Denean1973

• Cubbie221!

• Metrocenter11

KrbGuess using USA Usernames

39

❯ ruby wordsmith.rb -I usa --filn --name-depth 10000 -q -o filn-usa-260k.txt

usernames in ./data/usa: 260000

❯ java -jar krbguess.jar --realm corp.trevorforget.com --dict filn-usa-260k.txt --server 10.10.10.10 --output corp-krbguess-1.log

KrbGuess v0.21 by Patrik Karlsson <[email protected]>====================================================[INF] Found user: [email protected][INF] Found user: [email protected][INF] Found user: [email protected][INF] Found user: [email protected][INF] Found user: [email protected][INF] Found user: [email protected][INF] Found user: [email protected][INF] Found (locked/disabled) user: [email protected]...<snipped>...[INF] Finished guessing 260000 usernames in 469 seconds

❯ cat corp-krbguess-1.log | grep -i found | grep -v disabled | wc -l505

• Collecting and collating this data required the development of some parsers

Parsers

40

❯ git clone https://github.com/skahwah/wordsmith_parsers.git

❯ ls

LICENSE cia-parsers landmark-parser osm-parsersREADME.md census-parsers names-parsers religion-parsers

https://github.com/skahwah/wordsmith_parsers

Future Work

41

• Data!– Diving deeper into OpenStreetMap

– Popular song lyrics (h/t @pfizzell)

– Got ideas? We’d love to hear them!

• Skills– GIS

– Multiple language speakers

– Obscure website hunting & scraping

• Design– Lookups based on coordinates

– API? (h/t @pfizzell)

Thank you!

42

Sanjiv Kawa@hackerjiv

S R . P E N E T R A T I O N T E S T E R

P S C / N C C G R O U P

Tom Porter@porterhau5

S R . S E C U R I T Y C O N S U L T A N T

F U S I O N X R E D T E A M

https://github.com/skahwah/wordsmith

Software

The world is y0ur$: Geolocation-based wordlist generation with wordsmith