31
GOOGLE ADWORDS Optimizing Online Advertising 15.071x – The Analytics Edge TM

Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

  • Upload
    others

  • View
    3

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

GOOGLE ADWORDS Optimizing Online Advertising

15071x ndash The Analytics Edge

TM

Google Inc

bull Provides products and servicesrelated to the Internetbull Mission ldquohellip to organize the

worldrsquos information and make ituniversally accessible andusefulrdquo

bull Most widely known for itssearch engine(wwwgooglecom)bull User enters a query Google

returns links to websites thatbest fit query

15071x ndash Google AdWords Linear Optimization Recitation 2

TM

History of Google

bull 1996 ndash Sergei Brin and Larry Page graduate students atStanford working on a research projectbull How to measure importance of any webpage using links on

the internet

bull 1998 ndash Incorporated Google as a company and receivedfirst funding database of 60 million webpages

bull 2004 ndash Initial Public Offering

bull 2007 ndash Google acquires YouTube and other companiesbull 2013 ndash More than 1 billion unique monthly visitors

15071x ndash Google AdWords Linear Optimization Recitation 3

Googlersquos Business Model

bull Google search engine is free to use so how doesGoogle make money

bull Answer online advertising

15071x ndash Google AdWords Linear Optimization Recitation 4

Example of Sponsored Ads

15071x ndash Google AdWords Linear Optimization Recitation

Screen shot copy Google All rights reserved This content is excluded from our Creative Commons license For more information see httpsocwmiteduhelpfaq-fair-use

5

       

Google Advertising - AdWords

bull Why do companies advertise on Googlebull Google receives heavy trafficbull Search pages are formatted in a very clean waybull Companies can choose which types of queries their ads

will be displayed for better targeting

bull 97 of Googlersquos revenues come from AdWords

15071x ndash Google AdWords Linear Optimization Recitation 6

How does Advertising on Google work

1 Advertisers place bids for different queries in an auction

2 Based on bids and quality score (fit of advertiser and ad to the queries) Google decides price-per-click of each advertiser and each query

3 Google then decides how often to display each ad for each query

15071x ndash Google AdWords Linear Optimization Recitation 7

Price-per-click (PPC)

bull For each query Google decides each advertiserrsquosprice-per-click (PPC)bull How much advertiser pays Google when user clicks ad

for that query

bull Each advertiser also specifies a budgetbull Each time user clicks on advertiserrsquos ad budget is

depleted by PPC amount

15071x ndash Google AdWords Linear Optimization Recitation 8

Example of price-per-click

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Budget -T--Mobile-- verizon ATampT $170

T-Mobile $100

Verizon $160

15071x ndash Google AdWords Linear Optimization Recitation

TM

Click-through Rate (CTR)

bull Advertiser only pays Google if the user clicks on thead

bull The probability that a user clicks on an advertiserrsquos adis the click-through rate (CTR)bull Can also think of as ldquoclicks per userrdquo

15071x ndash Google AdWords Linear Optimization Recitation 10

Example of click-through rate

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 11

Average Price Per Display

bull Average amount that an advertiser pays each time itsad is shown is PPC x CTR

15071x ndash Google AdWords Linear Optimization Recitation 12

How average price per display works

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

Suppose 10 users search for ldquobest LTE networkrdquo

15071x ndash Google AdWords Linear Optimization Recitation 3

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 2: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Google Inc

bull Provides products and servicesrelated to the Internetbull Mission ldquohellip to organize the

worldrsquos information and make ituniversally accessible andusefulrdquo

bull Most widely known for itssearch engine(wwwgooglecom)bull User enters a query Google

returns links to websites thatbest fit query

15071x ndash Google AdWords Linear Optimization Recitation 2

TM

History of Google

bull 1996 ndash Sergei Brin and Larry Page graduate students atStanford working on a research projectbull How to measure importance of any webpage using links on

the internet

bull 1998 ndash Incorporated Google as a company and receivedfirst funding database of 60 million webpages

bull 2004 ndash Initial Public Offering

bull 2007 ndash Google acquires YouTube and other companiesbull 2013 ndash More than 1 billion unique monthly visitors

15071x ndash Google AdWords Linear Optimization Recitation 3

Googlersquos Business Model

bull Google search engine is free to use so how doesGoogle make money

bull Answer online advertising

15071x ndash Google AdWords Linear Optimization Recitation 4

Example of Sponsored Ads

15071x ndash Google AdWords Linear Optimization Recitation

Screen shot copy Google All rights reserved This content is excluded from our Creative Commons license For more information see httpsocwmiteduhelpfaq-fair-use

5

       

Google Advertising - AdWords

bull Why do companies advertise on Googlebull Google receives heavy trafficbull Search pages are formatted in a very clean waybull Companies can choose which types of queries their ads

will be displayed for better targeting

bull 97 of Googlersquos revenues come from AdWords

15071x ndash Google AdWords Linear Optimization Recitation 6

How does Advertising on Google work

1 Advertisers place bids for different queries in an auction

2 Based on bids and quality score (fit of advertiser and ad to the queries) Google decides price-per-click of each advertiser and each query

3 Google then decides how often to display each ad for each query

15071x ndash Google AdWords Linear Optimization Recitation 7

Price-per-click (PPC)

bull For each query Google decides each advertiserrsquosprice-per-click (PPC)bull How much advertiser pays Google when user clicks ad

for that query

bull Each advertiser also specifies a budgetbull Each time user clicks on advertiserrsquos ad budget is

depleted by PPC amount

15071x ndash Google AdWords Linear Optimization Recitation 8

Example of price-per-click

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Budget -T--Mobile-- verizon ATampT $170

T-Mobile $100

Verizon $160

15071x ndash Google AdWords Linear Optimization Recitation

TM

Click-through Rate (CTR)

bull Advertiser only pays Google if the user clicks on thead

bull The probability that a user clicks on an advertiserrsquos adis the click-through rate (CTR)bull Can also think of as ldquoclicks per userrdquo

15071x ndash Google AdWords Linear Optimization Recitation 10

Example of click-through rate

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 11

Average Price Per Display

bull Average amount that an advertiser pays each time itsad is shown is PPC x CTR

15071x ndash Google AdWords Linear Optimization Recitation 12

How average price per display works

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

Suppose 10 users search for ldquobest LTE networkrdquo

15071x ndash Google AdWords Linear Optimization Recitation 3

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 3: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

History of Google

bull 1996 ndash Sergei Brin and Larry Page graduate students atStanford working on a research projectbull How to measure importance of any webpage using links on

the internet

bull 1998 ndash Incorporated Google as a company and receivedfirst funding database of 60 million webpages

bull 2004 ndash Initial Public Offering

bull 2007 ndash Google acquires YouTube and other companiesbull 2013 ndash More than 1 billion unique monthly visitors

15071x ndash Google AdWords Linear Optimization Recitation 3

Googlersquos Business Model

bull Google search engine is free to use so how doesGoogle make money

bull Answer online advertising

15071x ndash Google AdWords Linear Optimization Recitation 4

Example of Sponsored Ads

15071x ndash Google AdWords Linear Optimization Recitation

Screen shot copy Google All rights reserved This content is excluded from our Creative Commons license For more information see httpsocwmiteduhelpfaq-fair-use

5

       

Google Advertising - AdWords

bull Why do companies advertise on Googlebull Google receives heavy trafficbull Search pages are formatted in a very clean waybull Companies can choose which types of queries their ads

will be displayed for better targeting

bull 97 of Googlersquos revenues come from AdWords

15071x ndash Google AdWords Linear Optimization Recitation 6

How does Advertising on Google work

1 Advertisers place bids for different queries in an auction

2 Based on bids and quality score (fit of advertiser and ad to the queries) Google decides price-per-click of each advertiser and each query

3 Google then decides how often to display each ad for each query

15071x ndash Google AdWords Linear Optimization Recitation 7

Price-per-click (PPC)

bull For each query Google decides each advertiserrsquosprice-per-click (PPC)bull How much advertiser pays Google when user clicks ad

for that query

bull Each advertiser also specifies a budgetbull Each time user clicks on advertiserrsquos ad budget is

depleted by PPC amount

15071x ndash Google AdWords Linear Optimization Recitation 8

Example of price-per-click

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Budget -T--Mobile-- verizon ATampT $170

T-Mobile $100

Verizon $160

15071x ndash Google AdWords Linear Optimization Recitation

TM

Click-through Rate (CTR)

bull Advertiser only pays Google if the user clicks on thead

bull The probability that a user clicks on an advertiserrsquos adis the click-through rate (CTR)bull Can also think of as ldquoclicks per userrdquo

15071x ndash Google AdWords Linear Optimization Recitation 10

Example of click-through rate

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 11

Average Price Per Display

bull Average amount that an advertiser pays each time itsad is shown is PPC x CTR

15071x ndash Google AdWords Linear Optimization Recitation 12

How average price per display works

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

Suppose 10 users search for ldquobest LTE networkrdquo

15071x ndash Google AdWords Linear Optimization Recitation 3

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 4: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Googlersquos Business Model

bull Google search engine is free to use so how doesGoogle make money

bull Answer online advertising

15071x ndash Google AdWords Linear Optimization Recitation 4

Example of Sponsored Ads

15071x ndash Google AdWords Linear Optimization Recitation

Screen shot copy Google All rights reserved This content is excluded from our Creative Commons license For more information see httpsocwmiteduhelpfaq-fair-use

5

       

Google Advertising - AdWords

bull Why do companies advertise on Googlebull Google receives heavy trafficbull Search pages are formatted in a very clean waybull Companies can choose which types of queries their ads

will be displayed for better targeting

bull 97 of Googlersquos revenues come from AdWords

15071x ndash Google AdWords Linear Optimization Recitation 6

How does Advertising on Google work

1 Advertisers place bids for different queries in an auction

2 Based on bids and quality score (fit of advertiser and ad to the queries) Google decides price-per-click of each advertiser and each query

3 Google then decides how often to display each ad for each query

15071x ndash Google AdWords Linear Optimization Recitation 7

Price-per-click (PPC)

bull For each query Google decides each advertiserrsquosprice-per-click (PPC)bull How much advertiser pays Google when user clicks ad

for that query

bull Each advertiser also specifies a budgetbull Each time user clicks on advertiserrsquos ad budget is

depleted by PPC amount

15071x ndash Google AdWords Linear Optimization Recitation 8

Example of price-per-click

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Budget -T--Mobile-- verizon ATampT $170

T-Mobile $100

Verizon $160

15071x ndash Google AdWords Linear Optimization Recitation

TM

Click-through Rate (CTR)

bull Advertiser only pays Google if the user clicks on thead

bull The probability that a user clicks on an advertiserrsquos adis the click-through rate (CTR)bull Can also think of as ldquoclicks per userrdquo

15071x ndash Google AdWords Linear Optimization Recitation 10

Example of click-through rate

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 11

Average Price Per Display

bull Average amount that an advertiser pays each time itsad is shown is PPC x CTR

15071x ndash Google AdWords Linear Optimization Recitation 12

How average price per display works

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

Suppose 10 users search for ldquobest LTE networkrdquo

15071x ndash Google AdWords Linear Optimization Recitation 3

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 5: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Example of Sponsored Ads

15071x ndash Google AdWords Linear Optimization Recitation

Screen shot copy Google All rights reserved This content is excluded from our Creative Commons license For more information see httpsocwmiteduhelpfaq-fair-use

5

       

Google Advertising - AdWords

bull Why do companies advertise on Googlebull Google receives heavy trafficbull Search pages are formatted in a very clean waybull Companies can choose which types of queries their ads

will be displayed for better targeting

bull 97 of Googlersquos revenues come from AdWords

15071x ndash Google AdWords Linear Optimization Recitation 6

How does Advertising on Google work

1 Advertisers place bids for different queries in an auction

2 Based on bids and quality score (fit of advertiser and ad to the queries) Google decides price-per-click of each advertiser and each query

3 Google then decides how often to display each ad for each query

15071x ndash Google AdWords Linear Optimization Recitation 7

Price-per-click (PPC)

bull For each query Google decides each advertiserrsquosprice-per-click (PPC)bull How much advertiser pays Google when user clicks ad

for that query

bull Each advertiser also specifies a budgetbull Each time user clicks on advertiserrsquos ad budget is

depleted by PPC amount

15071x ndash Google AdWords Linear Optimization Recitation 8

Example of price-per-click

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Budget -T--Mobile-- verizon ATampT $170

T-Mobile $100

Verizon $160

15071x ndash Google AdWords Linear Optimization Recitation

TM

Click-through Rate (CTR)

bull Advertiser only pays Google if the user clicks on thead

bull The probability that a user clicks on an advertiserrsquos adis the click-through rate (CTR)bull Can also think of as ldquoclicks per userrdquo

15071x ndash Google AdWords Linear Optimization Recitation 10

Example of click-through rate

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 11

Average Price Per Display

bull Average amount that an advertiser pays each time itsad is shown is PPC x CTR

15071x ndash Google AdWords Linear Optimization Recitation 12

How average price per display works

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

Suppose 10 users search for ldquobest LTE networkrdquo

15071x ndash Google AdWords Linear Optimization Recitation 3

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 6: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

       

Google Advertising - AdWords

bull Why do companies advertise on Googlebull Google receives heavy trafficbull Search pages are formatted in a very clean waybull Companies can choose which types of queries their ads

will be displayed for better targeting

bull 97 of Googlersquos revenues come from AdWords

15071x ndash Google AdWords Linear Optimization Recitation 6

How does Advertising on Google work

1 Advertisers place bids for different queries in an auction

2 Based on bids and quality score (fit of advertiser and ad to the queries) Google decides price-per-click of each advertiser and each query

3 Google then decides how often to display each ad for each query

15071x ndash Google AdWords Linear Optimization Recitation 7

Price-per-click (PPC)

bull For each query Google decides each advertiserrsquosprice-per-click (PPC)bull How much advertiser pays Google when user clicks ad

for that query

bull Each advertiser also specifies a budgetbull Each time user clicks on advertiserrsquos ad budget is

depleted by PPC amount

15071x ndash Google AdWords Linear Optimization Recitation 8

Example of price-per-click

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Budget -T--Mobile-- verizon ATampT $170

T-Mobile $100

Verizon $160

15071x ndash Google AdWords Linear Optimization Recitation

TM

Click-through Rate (CTR)

bull Advertiser only pays Google if the user clicks on thead

bull The probability that a user clicks on an advertiserrsquos adis the click-through rate (CTR)bull Can also think of as ldquoclicks per userrdquo

15071x ndash Google AdWords Linear Optimization Recitation 10

Example of click-through rate

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 11

Average Price Per Display

bull Average amount that an advertiser pays each time itsad is shown is PPC x CTR

15071x ndash Google AdWords Linear Optimization Recitation 12

How average price per display works

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

Suppose 10 users search for ldquobest LTE networkrdquo

15071x ndash Google AdWords Linear Optimization Recitation 3

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 7: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

How does Advertising on Google work

1 Advertisers place bids for different queries in an auction

2 Based on bids and quality score (fit of advertiser and ad to the queries) Google decides price-per-click of each advertiser and each query

3 Google then decides how often to display each ad for each query

15071x ndash Google AdWords Linear Optimization Recitation 7

Price-per-click (PPC)

bull For each query Google decides each advertiserrsquosprice-per-click (PPC)bull How much advertiser pays Google when user clicks ad

for that query

bull Each advertiser also specifies a budgetbull Each time user clicks on advertiserrsquos ad budget is

depleted by PPC amount

15071x ndash Google AdWords Linear Optimization Recitation 8

Example of price-per-click

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Budget -T--Mobile-- verizon ATampT $170

T-Mobile $100

Verizon $160

15071x ndash Google AdWords Linear Optimization Recitation

TM

Click-through Rate (CTR)

bull Advertiser only pays Google if the user clicks on thead

bull The probability that a user clicks on an advertiserrsquos adis the click-through rate (CTR)bull Can also think of as ldquoclicks per userrdquo

15071x ndash Google AdWords Linear Optimization Recitation 10

Example of click-through rate

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 11

Average Price Per Display

bull Average amount that an advertiser pays each time itsad is shown is PPC x CTR

15071x ndash Google AdWords Linear Optimization Recitation 12

How average price per display works

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

Suppose 10 users search for ldquobest LTE networkrdquo

15071x ndash Google AdWords Linear Optimization Recitation 3

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 8: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Price-per-click (PPC)

bull For each query Google decides each advertiserrsquosprice-per-click (PPC)bull How much advertiser pays Google when user clicks ad

for that query

bull Each advertiser also specifies a budgetbull Each time user clicks on advertiserrsquos ad budget is

depleted by PPC amount

15071x ndash Google AdWords Linear Optimization Recitation 8

Example of price-per-click

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Budget -T--Mobile-- verizon ATampT $170

T-Mobile $100

Verizon $160

15071x ndash Google AdWords Linear Optimization Recitation

TM

Click-through Rate (CTR)

bull Advertiser only pays Google if the user clicks on thead

bull The probability that a user clicks on an advertiserrsquos adis the click-through rate (CTR)bull Can also think of as ldquoclicks per userrdquo

15071x ndash Google AdWords Linear Optimization Recitation 10

Example of click-through rate

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 11

Average Price Per Display

bull Average amount that an advertiser pays each time itsad is shown is PPC x CTR

15071x ndash Google AdWords Linear Optimization Recitation 12

How average price per display works

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

Suppose 10 users search for ldquobest LTE networkrdquo

15071x ndash Google AdWords Linear Optimization Recitation 3

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 9: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Example of price-per-click

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Budget -T--Mobile-- verizon ATampT $170

T-Mobile $100

Verizon $160

15071x ndash Google AdWords Linear Optimization Recitation

TM

Click-through Rate (CTR)

bull Advertiser only pays Google if the user clicks on thead

bull The probability that a user clicks on an advertiserrsquos adis the click-through rate (CTR)bull Can also think of as ldquoclicks per userrdquo

15071x ndash Google AdWords Linear Optimization Recitation 10

Example of click-through rate

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 11

Average Price Per Display

bull Average amount that an advertiser pays each time itsad is shown is PPC x CTR

15071x ndash Google AdWords Linear Optimization Recitation 12

How average price per display works

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

Suppose 10 users search for ldquobest LTE networkrdquo

15071x ndash Google AdWords Linear Optimization Recitation 3

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 10: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Click-through Rate (CTR)

bull Advertiser only pays Google if the user clicks on thead

bull The probability that a user clicks on an advertiserrsquos adis the click-through rate (CTR)bull Can also think of as ldquoclicks per userrdquo

15071x ndash Google AdWords Linear Optimization Recitation 10

Example of click-through rate

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 11

Average Price Per Display

bull Average amount that an advertiser pays each time itsad is shown is PPC x CTR

15071x ndash Google AdWords Linear Optimization Recitation 12

How average price per display works

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

Suppose 10 users search for ldquobest LTE networkrdquo

15071x ndash Google AdWords Linear Optimization Recitation 3

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 11: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Example of click-through rate

Advertiser Query 1 (ldquo4G LTErdquo)

Query 2 (ldquolargest LTErdquo)

Query 3 (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 11

Average Price Per Display

bull Average amount that an advertiser pays each time itsad is shown is PPC x CTR

15071x ndash Google AdWords Linear Optimization Recitation 12

How average price per display works

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

Suppose 10 users search for ldquobest LTE networkrdquo

15071x ndash Google AdWords Linear Optimization Recitation 3

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 12: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Average Price Per Display

bull Average amount that an advertiser pays each time itsad is shown is PPC x CTR

15071x ndash Google AdWords Linear Optimization Recitation 12

How average price per display works

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

Suppose 10 users search for ldquobest LTE networkrdquo

15071x ndash Google AdWords Linear Optimization Recitation 3

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 13: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

How average price per display works

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

ldquobest LTE networkrdquo ldquobest LTE networkrdquo

Suppose 10 users search for ldquobest LTE networkrdquo

15071x ndash Google AdWords Linear Optimization Recitation 3

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 14: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

Google decides to display Verizonrsquos ad

verizon

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 15: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

How average price per display works

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

verizon

CTR of Verizon and ldquobest LTE networkrdquo is 02 so only 2 users click on the ad

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 16: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

How average price per display works

verizon Verizon pays PPC for each user

2 clicksverizon $25 per click

= $50

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 17: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

How average price per display works

verizon Verizon pays on average per user display

$50verizon

divide 10 displays

= $5 per display

15071x ndash Google AdWords Linear Optimization Recitation

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 18: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

How average price per display works

verizon This is exactly the PPC multiplied by the CTR

verizon $25 per click

020 clicks per user

= $5 per userdisplay

15071x ndash Google AdWords Linear Optimization Recitation

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 19: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Average price per display for example

Advertiser Query 1 PPC (ldquo4G LTErdquo)

Query 2 PPC (ldquolargest LTErdquo)

Query 3 PPC (ldquobest LTE networkrdquo)

ATampT $5 $5 $20

T-Mobile $10 $5 $20

Verizon $5 $20 $25

Advertiser Query 1 CTR (ldquo4G LTErdquo)

Query 2 CTR (ldquolargest LTErdquo)

Query 3 CTR (ldquobest LTE networkrdquo)

ATampT 010 010 008

T-Mobile 010 015 010

Verizon 010 020 020

15071x ndash Google AdWords Linear Optimization Recitation 19

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 20: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Average price per display for example

Advertiser Query 1 APPD (ldquo4G LTErdquo)

Query 2 APPD (ldquolargest LTErdquo)

Query 3 APPD (ldquobest LTE networkrdquo)

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

15071x ndash Google AdWords Linear Optimization Recitation 20

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 21: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Query estimates

bull Google does not control how many times a query will be requested ndash driven by users

bull For each query Google has estimate of number of times query will be requested over a given day

15071x ndash Google AdWords Linear Optimization Recitation 21

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 22: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Example of query estimates

Query Est of Requests

ldquo4G LTErdquo 140

ldquolargest LTErdquo 80

ldquobest LTE networkrdquo 80

15071x ndash Google AdWords Linear Optimization Recitation 22

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 23: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Googlersquos problem

bull How many times to display each ad for each query to maximize revenue

15071x ndash Google AdWords Linear Optimization Recitation 23

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 24: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Googlersquos problem

bull Objectivebull Maximize revenue

bull Decisionbull For each advertiser and query number of times ad will

be displayed for that querybull Constraints

bull Average amount paid by each advertiser cannot exceedbudget

bull Total ads for given query cannot exceed estimatednumber of requests for that query

15071x ndash Google AdWords Linear Optimization Recitation 24

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 25: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Problem data

Advertiser Avg $ Query 1 Ad Display

Avg $ Query 2 Ad Display

Avg $ Query 3 Ad Display

ATampT $050 $050 $160

T-Mobile $100 $075 $200

Verizon $050 $400 $500

ATampT $170 Q1 (ldquo4G LTErdquo) 140

T-Mobile $100 Q2 (ldquolargest LTErdquo) 80

Verizon $160 Q3 (ldquobest LTE networkrdquo) 80

Advertiser Budget Query Est of Requests

15071x ndash Google AdWords Linear Optimization Recitation 24

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 26: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Modeling the problem

bull Decision variables

bull Revenue to Google under ad strategy

bull Amount advertiser ATampT pays in ad strategy

bull Number of times ad strategy uses query 2

15071x ndash Google AdWords Linear Optimization Recitation 25

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 27: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Extensions to the problem

bull Slatespositionsbull Personalizationbull Other issues

bull Estimating CTRsbull How should advertisers bid

15071x ndash Google AdWords Linear Optimization Recitation 26

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 28: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Slatespositions

bull Search result page has space for more than one adbull Slate combination of adsbull Many possible slates which ones to display

15071x ndash Google AdWords Linear Optimization Recitation 27

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 29: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

Personalization

bull In addition to the query Google can use otherinformation to decide which ad to displaybull IP addressgeographic locationbull Previous Google searchesbrowser activity on Google

bull How do we account for this

15071x ndash Google AdWords Linear Optimization Recitation 28

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 30: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

AdWords at Googlersquos scale

bull We studied a small instance of the ad allocationproblembull 3 bidders 3 queries

bull We saw how an optimization solution increasesrevenue by 16 over ldquocommon-senserdquo solution

bull In reality problem is much largerbull Hundreds to thousands of bidders over $40 billionbull Gains from optimization at this scale become enormous

15071x ndash Google AdWords Linear Optimization Recitation 29

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms

Page 31: Optimizing Online Advertising 15.071x – The Analytics Edge...Google Advertising - AdWords • Why do companies advertise on Google? • Google receives heavy traffic • Search pages

MIT OpenCourseWare httpsocwmitedu

15071 Analytics Edge Spring 2017

For information about citing these materials or our Terms of Use visit httpsocwmiteduterms