Distil Networks 2017 Bad Bot Report: 6 High Risk Lessons for Website Defenders

IT & DATA MANAGEMENT RESEARCH,

INDUSTRY ANALYSIS & CONSULTING

David Monahan

Research Director, Enterprise Management Associates

Twitter: @SecurityMonahan

Distil Networks 2017 Bad Bot Report:

6 High Risk Lessons

for Website Defenders

Rami Essaid

CEO, Distil Networks

Twitter: @ramiessaid



Featured Speakers

David Monahan, Research Director, Risk & Security Management, EMA

David has over 20 years of IT security experience and has organized and

managed both physical and information security programs, including Security and

Network Operations (SOCs and NOCs) for organizations ranging from Fortune

100 companies to local government and small public and private companies.

Rami Essaid, CEO, Distil Networks

Rami is the CEO and co-founder of Distil Networks, the first easy and accurate

way to identify and police malicious website traffic, blocking 99.9% of bad bots

without impacting legitimate users.

With over 15 years in telecommunications, network security, and cloud

infrastructure management, Rami continues to advise enterprise companies

around the world, helping them embrace the cloud to improve their scalability and

reliability while maintaining a high level of security. Follow Rami at @RamiEssaid

Slide 2 © 2017 Enterprise Management Associates, Inc.



Logistics for Today’s Webinar

Slide 3 © 2017 Enterprise Management Associates, Inc.Slide 3 © 2016 Enterprise Management Associates, Inc.

An archived version of the event recording will be

available at www.enterprisemanagement.com

• Log questions in the chat panel located on the lower

left-hand corner of your screen

• Questions will be addressed during the Q&A session

of the event

QUESTIONS

EVENT RECORDING

A PDF of the speaker slides will be distributed

to all attendees

PDF SLIDES

http://www.enterprisemanagement.com/



David Monahan

Research Director of Security and Risk Management

Enterprise Management Associates

[email protected]

@SecurityMonahan

Bad Bot Report:

Six Risky Lessons


mailto:[email protected]



Bot- a.k.a. “Internet Bot,” “Internet

Robot,” or “Web Robot”

Automated systems using various programs to perform

relatively simple, repetitive tasks on behalf of their owners



Bots are Part of Internet Life

The web, e-commerce, and bots are here to stay

Good bots are used by all major web presence

companies:

Facebook, Google, Microsoft, Yahoo, etc.

Used to index/manage websites, measure app

performance, and other maintenance tasks

Bad bots are used by nefarious organizations

worldwide

Bad bots are created, not born

Free cloud accounts

Compromised systems








The Good, the Bad, and the Ugly About Bots

Bots are estimated to be between 40% and 55%

of total Internet traffic

Bad bots are estimated to be between 19% and

31% of Internet traffic

Bot control is voluntary without additional

technology

Robots.txt is the only “integrated” protection method in

html

Require “tests” or thorough vetting to stop



INDUSTRY ANALYSIS & CONSULTINGSlide 8

When Bots Attack (Application and API Flaws)

Token Cracking

Carding

Ad Fraud

Fingerprinting

Scalping Obtain

Expediting

Credential Cracking

Credential Stuffing

CAPTCHA Bypass

Card Cracking

© 2017 Enterprise Management Associates, Inc.

More at OWASP Automated Threat Handbook

Scraping

Cashing Out

Sniping

Vulnerability Scanning

(Distributes) Denial of Service

Footprinting

Skewing

Spamming

Account Creation

Account Aggregation

https://www.owasp.org/images/3/33/Automated-threat-handbook.pdf



Why Bots can be Tough for Applications to

Detect Bots masquerade as users

Page browsing

Mouse movement and clicks

Adaptive content presentation/responses

Bots masquerading as other devices

Lies that it is a mobile device

Lies about its browser engine/version

Lies about its OS

Application APIs deliver micro-services,

exposing numerous interfaces to the

Internet

Net effect: provides opportunity to attack each

micro-service






Techniques to Stop Application Attacks

Better application coding practices

Input filtering

Safer functions

HIPS (Human Interactive Proofs)

(re)CAPTCHA

Hidden fields

HOPS (Human Observation Proofs)

Mouse movement

Page movement (selection rate, usage patterns)

Clicks

Web Application Firewall

Bots or bad programming – life lesson




Attacks Against Business Logic


Exploit various facets of operation rather than programming

flaws

Require a greater understanding of operation than

programming

No single part of the application or normal Internet filtering has

enough visibility/context

Business logic attacks are not trivial in their consequences and are

successful on even the largest organizations. A few of the large organizations

that fell victim to business logic flaws are Facebook, Nokia, and Vimeo.

http://techcrunch.com/2015/02/12/this-bug-wouldve-let-anyone-delete-your-photos-from-facebook/

http://avsecurity.in/2013/06/nokia-insecure-direct-object-reference/

https://hackerone.com/reports/42587



Modification of authentication flags and privilege escalations

Business constraint exploitation/modification or business logic

bypass to generate fraudulent transactions

Requested parameter modification

Developer’s cookie tampering and business process/logic

bypass

Exploiting clients’ side business routines embedded in

JavaScript, Flash, or Silverlight

Identity or profile extraction

LDAP parameter identification and critical infrastructure

access




Attacks Against Business Logic

Examples



Thoughts

Cloud and IoT have done for bots what Paypal and

cryptocurrency has done for ransomware

Bad bots are at epidemic proportions and will continue

expanding if left unchecked

Bot activity will continue to become more invasive and

burdensome to application delivery

Bot sophistication is increasing. Machine learning and AI

will do for bots what they did for malware detection

Automation of Internet attacks will likely have the same

impacts on the hacking industry that it has on other

production line manufacturing (bots replacing humans)13 © 2017 Enterprise Management Associates, Inc.





Defeating Advanced Bots

Continuous monitoring and prevention are necessary: but with WHAT?!

Must “see” full context

• API and business logic awareness is crucial

• Advanced fingerprinting (sees through the lies)

More than IP, OS, browser, reputation

Pull data from client, not rely on push

• Adaptive learning (unsupervised machine learning)

• Behavioral analysis

• Enhanced API authentication

• Dynamic rate limiting to protect API scraping

• Browser validation




Next Steps




• Learn more about bots!

• Take your time in evaluating solutions

• Ask the right questions

(Check out the paper)



Distil Networks 2017 Bad Bot Report:

6 High Risk Lessons


Rami Essaid

CEO, Distil Networks

Twitter: @ramiessaid

2017 Bad Bot Report Methodology

Study based on anonymized data

Hundreds of billions of bot requests

Thousands of domains

Plus 17 global data centers

Key FindingsKey Findings

Bad Bot, Good Bot, and Human Traffic, 2016

Good

Bots

Humans

Bad Bots

19.9% of Web Traffic Causes Problems

The Four Key Website Attributes that Attract Bad Bots

Signup and Login

Payment Processor

Web Forms

Pricing Information

Proprietary Content

The Four Attributes By the Numbers

The Bad Bot Landscape

Size Matters: The Bigger The Site, The Bigger the Bad Bot Problem

Largest sites most

attractive to bad bots

Bad bot traffic on large

sites up 36.43% YOY

Small and tiny sites have

more bots than humans

*Websites grouped by Alexa rank

More Bad Bots Than Good on Large and Medium Sites

Small and tiny sites

have more good

bots than bad bots

37.5% more bad

bots than good on

large sites

Uncle Sam’s Bot Army

More bad bots come from the US than

the rest of the world...combined

The US originates 5 times more bad bot

traffic than The Netherlands (2nd Place)

Countries with the Highest “Bad Bot GDP”

Dominica has 3,348 bad bots per

internet user

Seychelles ranked third, which is

also the alleged home of the owner

of BitTorrent site Pirate Bay

US only 5th on bad bot GDP list

with 446

Bad Bots Lie About Their Identity

75.9% of bad bots claim they are

Chrome, Internet Explorer,

Firefox, or Safari

38.61% of bad bots claim they

are Chrome

More bad bots claim to be Safari

Mobile than Safari OSX for the

first time

8% of bad bots claim to be good

bots like search engine crawlers

More Bad Bots Claim to Be Mobile

The amount of bad bots claiming to be

mobile browsers jumped 42.78% in 2016

Mobile: The Undefended Frontier

9.4% of bad bot traffic

originates from mobile ISPs

T-Mobile and AT&T Wireless

top US based Mobile ISPs for

bad bot traffic

China Mobile third on the list

Data Centers are the Biggest Threat

Two out of three bad bots come from a data center

Amazon AWS is responsible for 4x the amount of bad

bot traffic as second place (OVH SAS)

Bad Bots Know What They Want

You’ve Been Scraped

OWASP AUTOMATED THREAT: SCRAPING

Scraper bot sophistication

What Gets Scraped?

Data Scraping Price Scraping

AggregatorsCompetitive Intel

Bad Bots Love Login Pages

OWASP AUTOMATED THREATS:

CREDENTIAL CRACKING, CREDENTIAL STUFFING

Account takeover bot sophistication

How Credential Stuffing Works

Credential stuffing exploits our

propensity to reuse passwords

across multiple sites.

Protecting Your Login Page Is Not Enough

Account Based Fraud

OWASP AUTOMATED THREATS:

CARDING, CARD CRACKING, CASHING OUT

Account exploitation bot sophistication

Account Takeover Attacks: Why?

Financial fraud

Targets are accounts at financial or e-commerce services that store users’ banking details. The attackers perform unauthorized withdrawal from bank accounts or fraudulent transactions using the credit/debit cards on file.

This includes virtual currency such as bitcoin, in-game currency, and rewards programs. This is all worth real money.

Spam

Spam can appear in any service feature that accepts user-generated content, including discussion forums, direct messages, and reviews/ratings, degrading platform integrity and brand reputation.

Phishing

Attackers can assume a compromised user’s identity and launch phishing attacks on others in his/her social circle to steal their credentials, personal information, or sensitive data.

Spamming Bots Are Annoying

OWASP AUTOMATED THREAT: SPAMMING

Spamming bot sophistication

Application Denial of Service

OWASP AUTOMATED THREAT: DENIAL OF

SERVICE

Denial of service bot

sophistication

What’s the Difference Between Application Denial of Service and DDoS?

Application Denial of Service

Attacks the application directly

Hard to spot because it won’t show up

as an anomaly on your firewall and may

not impact load balancer

DDoS

Attacks the ISP hosting your application

Easier to spot because it floods

upstream infrastructure to point where

packets never arrive at the web server

All Your Web Analytics Are Wrong

OWASP AUTOMATED THREAT: SKEWING

Sophistication level of bots

that skew analytics

Skewed Conversion Tracking

“The number of conversions were

greatly deflated because of bad bot

traffic. Now that we’re filtering bad

bot traffic out, we’re able to see

what the real data is and make

decisions based on real visitors.”

Marty Boos

CIO, StubHub

Advice for Web Security Professionals

Geofence Your Website from Offending Countries

*Measuring customer block requests by geographical region

China and Russia

accounted for 79.9%

of country-specific

block requests

Dominica, Netherlands,

and Seychelles all

generate more than a

thousand bad bots per

internet user

Only Allow Browsers on Your Site

25% of bad bots are simple scripts running

in the command line interface

If you block users that aren’t connecting

with browsers, you will prevent simple bad

bots from attacking your site

Block Old User Agents and Browsers

9.45% of bad bots claim to be

browser versions that are 5

years old or older

Blocking old browsers and user

agents will stop bad bots from

reaching your site

The top 10 Oldest Self-Reported Browsers by Bad Bots, 2016

Mobile is a Growing Bad Bot Attack Vector

Rate-limit mobile traffic

Consider carefully when IP

blocking within mobile because

it blocks too many real users

Try to generate tokens, in a

secure way, to identify and

rate-limit users

Having a login, data, pricing information,

payment processing, and/or forms means you

have bad bots

Take action, don’t just ignore the problem

Don’t do it yourself, because you’ll be stuck in an

endless cycle of IP whack-a-mole

Understand the problem, read the OWASP

handbook on automated threats

Don’t Ignore the Problem

What to Look for in a Bot Mitigation Solution

Blocks all automated threats including scraping,

account takeover, spamming, and payment

processor fraud

Uses hi-definition digital fingerprints to ID bad bots,

not just IPs

Enables geofencing from offending nations and ISP

fencing from offending ISPs

Detects scripts, headless browsers, and browser

automation that imitates legitimate users

Applies behavioral analysis using machine learning

Protects APIs

Advanced Persistent Bots

APBs

75%

Basic scripts running

in command line

Headless browsers,

more human-like

Browser automation,

most human-like

https://resources.distilnetworks.com/whitepapers/2017-bad-bot-report

Download the Report

QUESTIONS….COMMENTS

?I N F O @ D I S T I L N E T W O R K S . C O M

OR CALL US ON

1.866.423.0606

www.distilnetworks.com

Thank You for Participating!

To learn more about Distil Networks, visit us at:

http://www.distilnetworks.com

Or contact us at: 415-423-0831

Technology

Distil Networks 2017 Bad Bot Report: 6 High Risk Lessons for Website Defenders