Upload
luke-otterblad
View
120
Download
1
Tags:
Embed Size (px)
DESCRIPTION
aka, smoking gun rule inference
Citation preview
AKA identifies unique coupons given different names in the SnipSnap coupon database using a combination
of k-means clustering and "smoking gun" feature based rule inference.
Github: https://github.com/snipsnap/aka-service/Email: [email protected]
Step 1: Matches – same value, description text and activity dates
Matches – pairs are shown ,but many more than 2 items are matched into groups
More Examples…Different Barcodes – Same Coupon
The above two were matched into a group. The coupon below was also in the same set of American Eagle but NOT put into the same group even though it has some similarity….
How does it work?• https://github.com/snipsnap/aka-service• run via the command line• $ python aka.py -db_pswd your_password -
store McDonald’sid face_value offer_details start_date expiriation_date
988767 Free With the purchase of an Egg McMuffin 2013-09-03 2013-10-31
989829 FREE Egg McMuffin with the purchase of an Egg McMuffin 2013-09-03 2013-10-31
997447 Free Egg McMuffin with the purchase of an Egg McMuffin 2013-09-03 2013-10-31
Active Coupons for a Store as a Graph• When the aka-service is started, for a particular store each active coupon is converted to dictionary format
and face value and details based features are converted to the python version of a graph and normalized with some language processing.
• Item - > Features
{"CouponA": {[‘free’, ‘with’, ‘the’, ‘purchase’, ‘of’, ‘an’, ‘egg’, ‘mcmuffin’] {"CouponB": {[‘free’, ‘with’, ‘the’, ‘purchase’, ‘of’, ‘an’, ‘egg’, ‘mcmuffin’]
• Features -> Item
{“egg":["CouponA","CouponB"], “mcmuffin": ["CouponA","CouponB"], “free": ["CouponA","CouponB"], “with":["CouponA","CouponB"], “the": ["CouponA","CouponB"]} "purchase": ["CouponA","CouponB"] “of": ["CouponA","CouponB"]} “an": ["CouponA","CouponB"]}
Despite different text AKA identifies all of these as the same item
id face_value offer_details start_date expiriation_date aka_guid
988767 Free With the purchase of an Egg McMuffin 2013-09-03 2013-10-31
de5086f0-35bc-11e3-8da3-005056c00008
989829 FREE Egg McMuffin with the purchase of an Egg McMuffin 2013-09-03 2013-10-31
de5086f0-35bc-11e3-8da3-005056c00008
997447 Free Egg McMuffin with the purchase of an Egg McMuffin 2013-09-03 2013-10-31
de5086f0-35bc-11e3-8da3-005056c00008
Free is treated as a value keyword (along with % and $ descriptions)
But, words and value alone don’t create the match.
Expiry date also matters
Coupon with No Barcode connected to the same offer with a barcode
Same offer value (free mini candle) and same data range (September 9-October 6, 2013)
Matching picture and computer images
A change in degree…but the same coupon
Smoking Gun Features
• A Smoking gun feature for a coupon is a piece of information that identifies it as being the same real world item as another coupon (with near certainty).
• There are two sources of such identification in the database. The first is a barcode_id. Multiple coupons that have the same barcode_id are indeed the same physical coupon. The second is a promo_code.
• Two coupons that have the same promo_code are the same coupon 95% + of the time. (Some stores like Dunkin Donuts don’t use unique codes…but more on that later)
More Matches
Above two coupons are matched, and are also NOT matched with the below coupon despite having an extremely similar description and validity:
The code in the upper right hand corner (9152 versus 9992 –the smoking gun) helps significantly in separating them into a different identification.
Two coupons Not matched, even though they have the same description and similar
text
(they are valid at different times)
Finding smoother images
I experimented with using the number of recorded features as an indicator of picture quality – but that didn’t have much correlation. What did work was using the picture with the highest number of redemptions within an aka group
Better images
The Dollar Store $1 Off coupon problem – likely to be many of those
These four were originally matched. But I had to introduce the notion of a confidence percentage. This is largely because AKA weights the value of an item more heavily than the details words describing the offer (for most stores they have few items that are the same price)
More equal prices, but with high confidence set
Trouble Spots: AKA identifies same offer due to assumed smoking gun, but while there is the same
barcode there is a different expiry.
Ignoring PLU for Dunkin Donuts (and other publishers that duplicate promocodes) and going with 99% confidence does the trick.
There’s Exceptions to every rule
• Coupons are no different• In the settings.yaml (pictured above) you can define exceptions to
global rules. • What pop_smoking_gun tells aka is that for Dunkin’ Donuts the
global rules of promo_code and barcode_id does not apply– for Dunkin Donuts’ they don’t create PLU codes as unique to an offer.
Another example
Ignoring PLU for Dunkin Donuts (and other publishers that duplicate promocodes) and going with 99% confidence does the trick.
But knowing the store “rules” also helps correct errors (if they stick to unique codes)
http://c346897.r97.cf1.rackcdn.com/cd0faf92-f85e-11e2-9f66-40406c9e1e47.jpg
http://c346897.r97.cf1.rackcdn.com/d32b578e-fd2a-11e2-9be6-40406c9e1e47.jpg
Mechanical Turk expiry: 10/17/2012 Mechanical Turk expiry: 10/7/2012
Since Bed Bath & Beyond id’s and promocodes indicate the same item aka can reconcile the mistake
AKA- never misinterpret a store's coupon rules again
ids sharable descrption_text Aka_guid
987120 1 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008
987271 1 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008
988484 1 save 20% on your entire purchase bath body works f139439
75926f4f-328f-11e3-a3cd-005056c00008
989519 1 save 20% on your entire purchase bath body works 9522
75926f4f-328f-11e3-a3cd-005056c00008
989774 1 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008
990040 0 save 20% on your entire purchase bath body works f139492
75926f4f-328f-11e3-a3cd-005056c00008
990943 1 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008
992970 1 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008
992998 0 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008
994314 1 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008
10 coupons all identified as the same item with some marked sharable and some not. Suppose a publisher had submitted coupon 990040 to not be shareable……
AKA- never misinterpret a store's coupon rules again
ids sharable descrption_text Aka_guid aka_sharable
987120 1 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008 0
987271 1 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008 0
988484 1 save 20% on your entire purchase bath body works f139439
75926f4f-328f-11e3-a3cd-005056c00008 0
989519 1 save 20% on your entire purchase bath body works 9522
75926f4f-328f-11e3-a3cd-005056c00008 0
989774 1 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008 0
990040 0 save 20% on your entire purchase bath body works f139492
75926f4f-328f-11e3-a3cd-005056c00008 0
990943 1 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008 0
992970 1 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008 0
992998 0 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008 0
994314 1 save 20% on your entire purchase bath body works 75926f4f-328f-11e3-a3cd-005056c00008 0
An easy feature could be to treat a single not sharable within an aka group as a “presidential” vote and switch all to not sharable. This can also work for items tagged as manufacturer coupons. You’d basically only need 1 tag from Mechanical Turk (or a from classifier).
Exact Matches
http://c346897.r97.cf1.rackcdn.com/5a621136-1511-11e3-a7d0-40406c9e1e47-thumb.jpghttp://c346897.r97.cf1.rackcdn.com/34605f52-1515-11e3-8576-40406c9e1e47-thumb.jpg
Kroger’s matches
Kroger’s requires the highest confidence of any store, as many of their coupons are different only by a single word. These will match (incorrectly) without a high confidence set. Listed below is a sample false match made by AKA:
Same item in the database twice for Macy’s
http://c346897.r97.cf1.rackcdn.com/59667340-1588-11e3-a8e3-40406c9e1e47-thumb.jpghttp://c346897.r97.cf1.rackcdn.com/ac4dc266-1588-11e3-a7d0-40406c9e1e47-thumb.jpg
Same item again
http://c346897.r97.cf1.rackcdn.com/25ddfb40-13d2-11e3-998b-40406c9e1e47-thumb.jpghttp://c346897.r97.cf1.rackcdn.com/25ddfb40-13d2-11e3-998b-40406c9e1e47-thumb.jpg
Rougher Image Connected with a better version at McDonalds
Does a big mac by any other name, still taste like a big mac?
Digital and print match
More Matches
Better coupon picture identification
Occasional data entry errors can lead to bad reconciliation
aka_guid id barcode_id alt_barcode_id
face_value offer_details
2719bf74-40b6-11e3-86dd-22000a91806d
421909 138859 0$5.00 Off $25.00
Save $5.00 On Your Purchase Of $25.00 Or More
2719bf74-40b6-11e3-86dd-22000a91806d
539197 46299 0 Save $1.00 On Any Aveeno Product
2719bf74-40b6-11e3-86dd-22000a91806d
560927 138859 0 Save $1.00 On any
2719bf74-40b6-11e3-86dd-22000a91806d
595323 138859 0 20% Off 1 Regular Priced Item
Here the 99% reliable barcode_id is idenified with 3 different items (for Toys R Us)
These three items were matched via barcode which I can only assume is some type of data entry error. The difference is that for every other toys”r”us coupon the smoking gun rules are valid. These items barcodes are recorded incorrectly
But it is an isolated error
Background for entity resolution (aka collective reconciliation, de-duping)
• Chapter 20 of Beautiful Data “Connecting Data” by Toby Segaran (who I think likely wrote the chapter while working on the YouTube reconciliation).
• Indrajit Bhattacharya’s PhD dissertation, which you can find at: http://www.lib.umd.edu/drum/handle/1903/4241
• About me: Father of 2 lovely daughters with my wife Emma. Programmer, Statistician, Pot Limit Omaha and Mixed Game poker semi-professional (though I don’t get much time for poker nowadays). I'm located in historic Northfield, MN where I share an office with my Jack Russell Terrier, Kirby.
• Email: [email protected].
Questions