The Bot Challenge - info.signalsciences.com

The Bot Challenge:Effective Detection, Analysis,and Management

Over 50% of all web requests are auto-generated, but not all are beneficial. From benign search engine crawlers to malicious account takeover and API abuse, there are numerous types of bots to be aware of that could cost your company.

Ebook

Ebook2 +www.fastly.com | www.signalsciences.com

What You Need to KnowThis ebook explains:

• The potential cost of malicious bots

• How good and bad bots differ

• How to effectively determine bot intent

• Tactics to stop malicious bot traffic from negatively impacting your business

The Scope and Cost of the Bot Threat

Bot Intent: Not All Bots are Malicious

Bad Bot Attack Indicators

Effective Bot Mitigation

Table of Contents

3

4

6

9


The Scope and Costof the Bot ThreatIn an application security context, a bot is software that automates web requests with various goals. Bots are used to perform tasks without human intervention, such as scanning and indexing website content or testing stolen credit card numbers when deployed with nefarious intent. Bots can be either helpful or harmful, and in the context of this paper, “bot attack” always refers to an attacker with a fraudulent goal.

Threat actors leverage automated bot attacks on websites and APIs to retrieve data for their own use or abuse application logic flows, such as authentication or payment processes. In the world of application development and deployment, bots present both security challenges and opportunities to understand how “good” bots are utilizing resources.

Unfortunately, bad bot-generated web requests aimed at your organization’s web apps and APIs are now the norm. Successful organizations plan to mitigate bot traffic to reduce unnecessary server instances and bandwidth usage, allowing legitimate customers to use applications that drive an organization’s business.

The bot-generated traffic targeting your apps and APIs will only increase as threat actors continue to leverage web attack methods like account takeover for their own goals.

Bots are now responsible for over 50% of all web traffic.

43% of all login attempts originate from malicious botnets2.

In the U.S. alone, threat actors will cause more than $12 billion1 in losses by next year. Direct financial losses are only part of the picture: fraud can lead to a negative customer experience, which damages your brand and reputation.

LoginAttempts 43%

WebTraffic 50%

https://nilsonreport.com/constant_contact_promo.php?id_promo=16


Bot Intent:Not All Bots Are MaliciousOperations and security teams should not indiscriminately block all bot traffic. Blocking all bot traffic from your website prevents legitimate customers and benign bots from accessing sites and can negatively impact your organization’s bottom line.

Benign bot traffic can improve organic search engine results, share e-commerce product information with partners, or syndicate content for distribution on third-party sites.

Good Bot Examples

Monitoring Bots

Monitoring bots check the health, availability, and responsiveness of websites. These bots provide real-time status and performance information about web services.

Search Engine Bots

Google, Bing, and other search engines use bots to index web content, which improves search engine results for users. Also known as web spiders, search engine bots automatically scan websites, categorize content, analyze harvested data, and assess technical SEO metrics (e.g. keyword density, broken links, HTML code validation). While search engine bots generally are helpful in furthering business goals such as expanding the exposure of site content to a wider audience, there is some risk in that the requests they generate can potentially overwhelm a site with too many requests in a given time period.

Bad Bot Examples

While benign bots have altruistic goals, others have malicious intent. Bad bots run code repeatedly and generate automated web traffic. They unleash floods of web requests against the login and other key transactional web pages of retailers, banks, or any organization that stores valuable personal or financial data and makes it accessible via a web application or API.

Here are common scenarios in which a threat actor leverages bots to achieve a fraudulent goal.

Content Scraping

Web scraping bots automatically gather and copy data from other websites. They can disguise themselves as innocuous search engine crawlers as they scan content, but these search bot imposters steal content without the knowledge nor permission of the website owner.

In contrast, legitimate search engine bots declare themselves using user agent strings (e.g. robots.txt, googlebot). Google or Bing use bot crawlers to index content for the primary purpose of improving search engine results for end-users. Scraped web content is a diverse category that includes written copy, images, HTML/CSS code, metadata, and e-commerce data.


The attacker repurposes this content in exploitative ways:

• Republishing copyrighted television shows or paywalled news articles

• Duplicating blog posts to steal SEO value and organic traffic

• Gathering product pricing or inventory data to gain a competitive advantage

• Compiling contact information to sell to other businesses as sales targets

• Stealing HTML code to build a fake branded website as part of a phishing scheme

Account Takeover (ATO)

When breaches occur, a potential outcome is large dumps of user credentials. Threat actors can purchase username and password dumps on the Dark Web and use automated bots to rapidly test the usernames and passwords in the authentication flows for major retail and financial websites. This process is known as “credential stuffing.” Once valid credentials are found, they’re then used against other sites to take over website accounts and lock out legitimate users. Attackers can also take personally identifiable information (PII) and stored payment methods from those accounts to commit further fraud.

Form Submission Abuse

Bots abuse forms on public-facing websites for a variety of malicious reasons. They might perform SQLi, scanning, or post unwanted content through these forms.

API Abuse

Automated bots probe APIs in an attempt to extract sensitive data like PII or credit card numbers. For example, adversaries use bots against public-facing APIs by canning spoof XFF header information to execute account takeover attacks.


Bad bot operators share a common goal: accessing valuable data that can be used for further fraud. This section reviews common attack types for the financial and retail e-commerce verticals. It also digs deeper into API abuse, which is important to understand since many companies share business and customer data via API.

Finance: A Prime Target for Account Takeover (ATO)Bots perpetuate ATO attempts against the websites of banks, credit unions, or online investment companies. To prevent them from succeeding, financial services companies must monitor key authentication events, which requires visibility into where ATO happens—at the point of account creation and login. Key web transactions to track success and failure rates for specific event types include:

• Account login

• Account creation

• Password reset

Block ATO Attempts With a Request Threshold-Based Approach

Financial companies should baseline their expected web request traffic volume for key authentication events by day, week, or month or a customer time frame specific to their business.

Defining a baseline for what is a “normal” or expected volume of request traffic over a specific time frame provides a guidepost for what is not

Bad Bot Attack Indicators

normal. When authentication events spike above an expected threshold, alerts can notify stakeholders and automated blocking can be put in place to prevent these abnormal request from volumes reaching an app or API endpoint.

Monitoring web request activity in this manner empowers DevOps and security teams to quickly identify malicious activity that causes requests to spike. Two key traffic trends to watch are:

• An unexpected increase in login attempts. This indicates account takeover attempts are occurring.

• A higher than normal amount of password resets. This signifies that customers are locked out of their accounts or malicious actors are attempting to reset passwords so they can commandeer legitimate customer accounts.

Account Linking and Cashing Out

After successfully acquiring banking or financial account credentials, attackers can link those accounts to their own and initiate fund transfers. Financial organizations should monitor two key indicators of cashing out:

• Account linking activity: when account linking unexpectedly increases, it can indicate attackers are preparing to siphon funds to third-party accounts.

• Funds transfer activity: this usually follows spikes in account linking and represents the payoff for attackers.


Retail: Fertile Ground for Automated Fraud

Organizations that operate an online retail presence often allow customers to transact using stored value units to purchase merchandise or services. Two common examples of this model are loyalty programs and discount deal sites. Attackers can use a combination of approaches in order to complete fraudulent purchases.

Retail Account Takeover

After acquiring stolen username and password pairs, they can manipulate an account for their own gain by:

• Changing the address so merchandise is delivered to a different address.

• Changing the email address to lock out a valid account owner and reset the account password.

• Using brute force methods to determine a card verification value (CVV), which retailers require valid to complete purchases.

• Creating fake accounts to test stolen credit cards with fraudulent transactions, no matter how small the value.

Authentication and Purchase Flows Are Primary Targets4

90% of websites with login pages

experience bot attacks like credential stuffing or credential cracking.

80% of sites with sign-up or application form pages are victims of malicious bot activity aimed at creating fraudulent new accounts.

E-Commerce Bot Attack Tactics

• Gift card cracking occurs when bot operators attempt to brute force an API that enables users to check their gift card balances—the end goal is to validate gift card numbers. Abnormally high requests against the gift card API and failures from a single IP can indicate a brute force attempt in progress.

• Inventory scraping bots visit product pages, perform searches, and scrape site data. These can be identified using pre-identified bad IP ranges from various sources like SANS. Signal Sciences customers benefit from Network Learning Exchange (NLX) that identifies suspicious traffic from sources that have been confirmed to be malicious and protects our entire customer base against subsequent attacks originating from identified IP addresses.

• Scalping bots are deployed against a purchase flow in an attempt to buy discounted or limited edition items. The fraudsters use stolen credit or stored value cards to purchase merchandise and then sell it elsewhere at a premium or at high volume. An indicator to monitor is higher than expected “add to cart” activity from a single IP.


API Abuse: Service Disruption, Data Leakage, or Account Lockout

APIs function as the backbone of modern web, cloud, and mobile applications, so it’s no surprise that attackers utilize bots to mimic legitimate API consumers. Gartner estimates that by 2022, API abuses will be the most frequent attack vector for enterprise web application data breaches3. Clearly, API security must be part of any strategic security plan.

APIs transfer a variety of data as organizations carry out their business operations. Financial services companies serve both customers and third-parties who need access to customer-specific data. Retailers share inventory and pricing data with resale and distribution partners. Healthcare organizations publish patient test results and request insurance claim information via API.

Detecting and blocking malicious requests is key to preventing attackers from abusing APIs and causing service disruption, data leakage, or account lockouts. Defeating API abuse requires visibility into where and how attackers are attempting to manipulate your application’s business logic, including authentication events. In order to surface those real-time insights, you’ll need to instrument and monitor your application for key application transaction events.

In addition to account takeovers, account linking and fund transfers, other examples of automated API misuse include:

• Fake web portal account creation: occurs when a bot operator seeks to manipulate an API to create large numbers of bot-controlled accounts.

• Data aggregation: bot-generated requests harvest an organization’s data and aggregates it with that of others for monetary gain without permission.

• Data scraping: automated harvesting of proprietary data via the data owner’s API.


Effective Bot MitigationStopping bad bots requires a security solution that provides the necessary visibility into various actions that bot operators are likely to take with automated web requests. This is achieved by examining and decisioning on web request context such as inspecting specific attributes in HTTP request headers, responses, or other information associated with incoming web requests. Signal Sciences (now part of Fastly), observes and examines the context of incoming automated traffic to identify bots.

Bot Detection Close x

Dec 3, 11:40 AM - Dec 3, 1:05 PM PST

Bad Bot

Bot User Agent

search-bad-bot

Scraper Bot

SEO Bot

9k

62

0

0

0

1.6k

768

011:45 12 PM 12:15 12:30 12:45

This context is determined by rules, lists, and classifications of many different types of bots, and can be displayed on the Signal Sciences management console. Lists allow you to parameterize site rules with business data you’ve collected, including:

• IP addresses of bot request sources

• User agents

• Form submission types

• Country of origin

• Wildcard values, and more

With rate-limiting rules enabled, Signal Sciences detects and blocks high-volume malicious bot requests.

This bot-generated web request analysis enables customers to create parameters and predefined signals that filter out bad bots from authentic users. Moreover, custom signals combined with other customer-created rules can identify automated login attempts, which are often leveraged in account takeovers.

Advanced Rate Limiting

Advanced rate limiting is a key means Signal Sciences provides customers to detect and block high-volumes of automated malicious web requests that would otherwise negatively impact application and API performance. It can stop bot-generated web requests that could result in abusive actions including:

• Brute force attacks

• Application & API abuse or denial of service

• Website content scraping

• Malicious credit card validation

• Login form attempts

• Password reset attempts

• Credential stuffing

• Web form abuse

• Gift card balance checking

Leveraging award-winning app and API web protection technology, advanced rate limiting provides intelligent controls to reduce the number of requests directed at these and other key web application functions.


However it’s not enough to stop the bad bot-generated traffic. A superior bot mitigation solution that utilizes rate limiting should empower security stakeholders to understand bot-generated traffic in order to more proactively manage the resulting blocked IP lists and provide the means to have full control over what web requests to respond to in the future. An advanced rate limiting offering should enable developers and security teams to:

• Easily setup application-specific rules to prevent app and API abuse.

• Define custom conditions to block abusive requests.

• Quickly identify and respond to a real-time list of IPs that have been rate limited.

• Gain valuable insights into traffic targeting applications or API endpoints.

• Reduce infrastructure costs by eliminating unpredictable traffic spikes and attacks.

Signal Sciences makes it easy to create application-specific rate limiting rules. One-click actions enable further control over automated volumetric web requests.


Custom Rules Reliably Block Bot Traffic

Signal Sciences can detect and stop application logic attacks, app feature abuse and misuse, account takeover attempts, and more. The intuitive rules builder in the management console enables customers to define their own signals with inputs selected from drop-down menus, including user agent, path, method, scheme, post or query parameter, request cookies, and more.

When those signals are tripped, automated actions occur and can include blocking a web request, alerting your teams, or other appropriate action. All of this can also be done programmatically via Signal Sciences’ API.

Because Signal Sciences can block anomalous bot-generated requests, customers are able to further sort and categorize signals to:

• Analyze bot behavior on a granular level with custom dashboards that uplevel key metrics around blocked bot-generated traffic.

• Prevent future bot requests based on changing request patterns and their associated web request context.

Signal Sciences rule builder enables security and DevOps teams to easily create rules that utilize custom parameters to detect and block malicious bot-generated web requests.


Bot Protection for Web Apps & APIs An effective web app and API protection solution provides actionable intelligence and tools necessary to easily and consistently block bot-generated web requests. Signal Sciences, now part of Fastly, can detect and defeat bad bots with easy configuration and no performance overhead on the apps or APIs. This deep context and the ability to immediately block and mitigate bot activity are key to actively protecting any organization’s web layer assets against automated attacks.

Signal Sciences unified management console empowers customers to see and analyze bot traffic across their entire application and API footprint. In this example, bot-specific signals determine and visually classify web requests as malicious or benign.


1 The Nilson Report - Issue 1164, November 2018

2 “The Internet is Mostly Bots” - Adrienne LaFrance, The Atlantic

3 How to Build an Effective API Security Strategy - Gartner Research

4 “Don’t Treat Your Customer Like a Criminal” — Tricia Phillips, Gartner Research

Endnotes

https://nilsonreport.com/constant_contact_promo.php?id_promo=16

https://www.theatlantic.com/technology/archive/2017/01/bots-bots-bots/515043/

https://www.theatlantic.com/technology/archive/2017/01/bots-bots-bots/515043/

https://www.gartner.com/en/documents/3697317/don-t-treat-your-customer-like-a-criminal

Please visit www.signalsciences.com to learn more about how our platform can help you.

https://www.signalsciences.com

Documents

The Bot Challenge - info.signalsciences.com