Using TF-IDF for Lateral Keyword Research

#pubcon

http://ash.nallawalla.com

@ashnallawalla

Lateral Keywords for Writers(When the Google Keyword Planner isn’t enough)

Presented by: Ash NallawallaSEO Strategist, Suncorp Insurance

#pubcon


@ashnallawalla

2

About Ash

• SEO consultant, currently at Suncorp Insurance (eight brands)

• Moderator at Webmasterworld forums

• Previously in enterprise SEO roles, notably NAB, ANZ Bank, Ubank, Optus and Yellow Pages

#pubcon


@ashnallawalla

KEYWORD RESEARCH BASICS

No longer enough

#pubcon


@ashnallawalla

4

Everybody’s doing it

• Most of us use the Google Keyword Planner to get a feel for the most searched terms.

• Our competitors do that too.

#pubcon


@ashnallawalla

5

Search volume alone isn’t enough

• But we need a starting point.

#pubcon


@ashnallawalla

6

Give researched keywords to writers

• A keyword matrix ensures a good spread of keywords across the site and saves the writer from guessing keywords.

#pubcon


@ashnallawalla

7

Search intent is important

• Intent can be Navigational, Informational, Commercial, Transactional.

• Yes, check out some tools.

#pubcon


@ashnallawalla

CHECK OUT THE COMPETITION

So who is winning in your niche?

#pubcon


@ashnallawalla

9

Start with a ranking check• Use your preferred rank-checking tool to see who is

ranking for each keyword.• We want to check which company’s content is

consistently coming up on Page 1 for a number of similar keywords.

#pubcon


@ashnallawalla

10

Count ranking keywords

• First get the count of keywords that rank.

#pubcon


@ashnallawalla

11

Derive the mean position

• Get the “average” position for each company.

#pubcon


@ashnallawalla

12

Invert the mean position

• “Inverting” means deducting the rank from 10, so that a higher number denotes a higher rank.

#pubcon


@ashnallawalla

13

Derive a “score”

• Score (say Allianz) = (C3*C5)+(D3*D5)• Score = (6.1x21)+(4.0x3) = 141

#pubcon


@ashnallawalla

14

Ranking spreadsheet

• “Visibility” is important, but what is your way to measure it?

• Which competitor is more visible?

#pubcon


@ashnallawalla

15

The content writer’s dilemma

• The spreadsheet shows the “winners”, not the “losers”. We can see who is using the most searched phrases.

• Others are using the same tools.• So what content are they using

that you are not using?• (Note: Ranking involves many other factors

and this is also about Selling!)

#pubcon


@ashnallawalla

DEEP DIVE – TERM FREQUENCY

Looking for that lightbulb moment?

#pubcon


@ashnallawalla

17

Hat Tip to Eric Enge

• See his articles in Moz:– Just Google “Eric Enge TF-IDF” for the URLs.

(click the image below if you have the PPT)

https://moz.com/blog/using-term-frequency-analysis-to-measure-content-quality

#pubcon


@ashnallawalla

18

Inverse Document Frequency

• Inverse Document Frequency – a measure of the “rareness” of a term, so we weigh down the stop words and scale up the rare ones.

• Refer to Eric’s second article for more details.

https://moz.com/blog/inverse-document-frequency-and-the-importance-of-uniqueness

#pubcon


@ashnallawalla

19

TF-IDF example

• Say, a document with 100 words contains the term “cat” 3 times.

• The TF is 3/100 x 0.5 + 0.5 = 0.515• Google has, say, 30 trillion pages and the word “cat”

appears in 1.7 billion pages.• The IDF is log(30,000,000,000,000 /1,700,000,000)

or log(730,2.718281828) = 6.593044535• The TF-IDF (or TF*IDF) weight is the product of:

0.515 x 6.593044535 = 3.395417935

#pubcon


@ashnallawalla

20

Term Frequency – Two ways to measure

• Term Frequency – how frequently the term appears in a document (incl. stop words)

Or

If Raw Term Count > 0, TF = 1+log10(Raw Term Count)If Raw Term Count = 0, TF = 0

#pubcon


@ashnallawalla

21

Getting back to Term Frequency…

• Search for your keyword.• Visit the page/s of the highest

ranking company and the next four top rankers.

• Note their URLs.• Note the URL of your own page.• Do a Term Frequency analysis

and, perhapsInverse Document Frequency analysis (TF-IDF).

#pubcon


@ashnallawalla

22

Get n-grams

• Use one of the old “keyword density” tools to get 1-word, 2-word, and 3-word pairs from your site and the five competitors.

• Collate, de-dupe n-grams and place in Eric’s spreadsheet.

#pubcon


@ashnallawalla

23

TF – one worksheet per keyword

• Six sets of n-grams on the left and de-duped list in grey zone.

Eric’s spreadsheet

#pubcon


@ashnallawalla

24

TF – close-up

• Get a count of each word or phrase used by the top five pages and by yours

#pubcon


@ashnallawalla

25

TF – close-up

• Next, do the TF number crunching, i.e.

#pubcon


@ashnallawalla

26

TF – close-up

• Use conditional formatting to pick a range of TF values and compare your TF column with the average TF of the competitors.

• You will now see “significant” words to consider.

#pubcon


@ashnallawalla

27

The extracted words

• The pages I was analysing did not contain some “obvious” words – this is the beauty of this technique.

#pubcon


@ashnallawalla

28

The future?

• Working on a web version

• Takes minutes, not hours.

Sliders

Gems

Beta: http://www.lateralkeywords.com

#pubcon


@ashnallawalla

29

Summary

• Keyword research requires more than the Google tool. Do lateral keyword research.

• Do consider Term Frequency at least. Also look into Inverse Document Frequency.

• Download full PPT from:http://www.trainsem.com/pubcon

Ash Nallawalla• Twitter: @ashnallawalla• Email: [email protected]• Web: http://ash.nallawalla.com

http://www.trainsem.com/pubcon

http://ash.nallawalla.com/

Marketing

Using TF-IDF for Lateral Keyword Research