Are bot operators eating your lunch?

Agenda

Bots 101The growing bot problemHow bots are eating your lunch!Hayneedle case studySelection criteria for a bot detection solutionQ & A

Good BotsSearch Engine Crawling Power APIsCheck system connectivity and status

Bad BotsSteal contentScan for vulnerabilitiesPerform fraudetc.

The Basics of Bots

A “Bot” is an automated program that runs on the internet

Traffic Distribution by Type, 2016

High Profile Web Scraping in the Ecommerce Industry

QVC is an American television home shopping network and online ecommerce site.

Aggressive price and inventory scraping by shopping aggregator app resulted in the following repercussions for QVC

● Two day website outage● Loss of $2M in revenue● Highly publicized lawsuit● Damage to QVC Brand

Traffic by Size, Ecommerce Sites, 2014 vs 2015

Small and medium ecommerce sites saw about a 100% increase in bad bots between 2014 and 2015

Majority of Bots are Advanced Persistent Bots (APBs)

APBs have one or more of the following abilities:

AdvancedMimick human behaviorLoad JavaScriptLoad external resourcesSupport cookiesBrowser automation (Selenium, PhantomJS)

Persistent Dynamic IP rotationDistribute attacks across IP addressesHide behind anonymous and peer-to-peer proxies

2016 Distil Bad Bot Report

Why the Massive Increase in APBs?

Online data has increased in valuePricing information, product availability, product descriptions, and vendor reviews are changing daily and highly valuable to competitors

Anyone can get in the gameCheap or free virtual servers, bandwidth, easy-to-use tools, and scrapers for hire

Bots no longer tied to IP addressesBots cycle through random IP addresses Bots hide behind anonymous proxies Consumer IPs now infected with bot traffic too

Loading Assets & Bots Mimicking Humans

% of bots able to load external assets (e.g. JavaScript) % of bots able to mimic human behavior

These bots skew marketing tools such as (Google Analytics, A/B testing,

conversion tracking, etc.)These bots fly under the radar of most

security tools

That Majority of Bad Bots Now Use Multiple IP Addresses

Bots which dynamically rotate IP addresses, or distribute attacks are significantly harder to detect and mitigate

Bad Bots Cause the Majority of Website Problems

19% of Traffic Causes the Following Problems

How Bots Eat Your Lunch

How bots are eating your lunch!

How Bots Eat Your Lunch

LOST PROFITS

Decreased Customer Loyalty

Reduce Findability

Lost Cross/Upsell Opportunities

Decreased Customer Satisfaction

Increased Costs

Increased Fraud

Bots and Competitive Data Mining

Duplicating your Product PortfolioBots can easily gather product and supplier listsfor replication elsewhere

Undermining your PricesBots monitor your prices, ensuring competitorscan undercut with lower price listings

Availability TrackingIdentifying when your supply has been exhausted provides competitors a unique opportunity to raise the price of their goods.

Negative SEO Attacks Damage Relevancy

Bots steal content, product lists, and prices for duplication elsewhere on the Internet

Duplicated content reduces your company’s uniqueness and thus quality score

SEO damage may result, especially if○ Your prices are undercut○ The content is repurposed on a more popular site

Duplicate Content Results in Diminished SEO

Common hacking tools like network mappers and vulnerability scanners are automated programs

Once a victim’s network has been mapped, automated vulnerability scanning can be used to find security flaws that can be exploited

These bots let hackers scale their operations

Vulnerability Scanning and Target Exploitation

Bots Make Large Scale Account Takeover Possible

Over 1 billion usernames, passwords combinations exist in the wild

Bot operators create bots to test millions of username/password combinations from breaches at other websites to find the credentials that work on your site

Newly compromised accounts are then used for various forms of fraud/theft

Automated Stolen Credit Card Testing Enables Fraud

“Carding” uses micro-transactions on stolen credit cards against e-commerce sites to test their validity

Carding results in poor user experiences and lots of expensive chargebacks

Bots Plant Malicious Links in Fake Comments

Comment spam is frequently used to redirect users to malicious websites

Malicious Site

Hayneedle Case Study

Hayneedle Case Study

About Hayneedle

Leading online retailer for indoor and outdoor home furnishings and decor

1,000s of top brands - including Hayneedle exclusive designs - and millions of products for every space, style, and budget

Hayneedle Bad Bot Challenges

Bad Bot Challenges Business Impact

Competitive price scraping Competitors attempt to undercut pricing

Automated CVV guessing games

Fraudsters use stolen credit cards in carding attacksTime investigating and reporting the problem

Bot traffic competing with real customers Web performance and the user experience

Skewed analytics Conversion funnel optimizationA/B testing

Inefficient DIY approach“Battle-of-the-bots” ate up 20% of team resourcesOnly 30% effective (at best)Quality of life issues

Hayneedle Bad Bot Challenges

Constant game of bad bot “Whack-a-mole”

Log file analysis and performance monitoring

Agent-string analysis

IP blocking

Traffic redirects

Tarpits

...but the bad bots keep changing their identities, scripts, and IP addresses

Hayneedle Bot Selection Criteria

Bot Detection and Mitigation Solution Requirements

No impact on human visitors

“Self tuning” for defending against emerging and unknown threats

Crowd-sourced threat intelligence model

Seamlessly co-exist with existing solutions(CDN, WAF, etc.)

No “black boxes”

Traffic Overview Report

On August 7th bad bot traffic:

● Spiked ~10x ● Was 4x human traffic

Total Traffic vs CAPTCHAs Served

CAPTCHAs served was 73% of overall traffic served that day!

CAPTCHA Failed Attempts and Solved

Out of 17,000,000+ CAPTCHAs served, only 78 were solved

How to Manage Transactional Traffic

Best Practices and Lessons Learned

Monitor (don’t CAPTCHA) traffic on your checkout and account subdomainsReview Threats by OrganizationUnderstand the rationale of scrapersSelectively Block nefarious organizations

Blocking Nefarious Organizations

Can probably block traffic coming from this data center, especially when 70% of the traffic is from Automated Browsers and/or Known Violators

Hayneedle Results with Distil Networks

Eliminated competitive data miningIntercepting bot traffic with negligible false positivesClean analytics for funnel optimization and A/B testingDistil is a key piece of our fraud detection and prevention suite of toolsUpstream HTTP Errors Report highlighted an issue with our CDN providerWeb infrastructure dedicated to serving humans Boosted team morale!

The Only Easy and Accurate Way to Protect Web Applications from Bad Bots, API Abuse, and Fraud

Browser ValidationDetects all known browser automation tools, such as Selenium and Phantom JS

Protects against browser spoofing by validating each incoming request as self reported

Advanced Bot Detection Increases Accuracy

Behavioral Modeling and Machine LearningMachine-learning algorithms pinpoint behavioral anomalies specific to your site’s unique traffic patterns

Self optimizing algorithms improve bot detection and mitigation without manual configuration

Sticky Bot Tracking With No Impact On Real Users

Device FingerprintingFingerprints stick to the bot even if it attempts to reconnect from random IP addresses or hide behind an anonymous proxy or peer-to-peer network

Tracks distributed attacks that would normally fly under the radar

Without Distil

With Distil

Without Impacting Users Sharing the Same IPAvoids blocking residential users or organizations that might share the same NAT as the bot or botnet

Threat Intelligence From All Distil-Protected Sites

Known Violators DatabaseReal-time updates from the world’s largest Known Violators Database, which is based on the collective intelligence of all Distil-protected sites

Distil customers are automatically protected against new threats discovered anywhere on the network

Automated Attackers Leverage APIs as an Attack Vector

Web Applications

API Endpoints

When blocked from a website, Bots frequently use APIs as an attack vector

APIs tend to have access to the same content, but without as many security controls

○ Install on virtualized or bare metal appliance(s)○ High availability configurations with failover

monitoring○ Heartbeat up to Distil Cloud ○ Deploys in days

Flexible Deployment Options

○ Automatically compresses and optimizes content for faster delivery

○ 17 global datacenters automatically fail over when a primary location goes offline

○ Automatically increases infrastructure and bandwidth to accommodate spikes

○ Deploys in hours

Physical or Virtual Appliances

Content Delivery Network

Dedicated Analyst Team

Fully Managed Service (aka High Touch Services)

So, is a bot

eating your

lunch?