ExtraHop IT Operational Intelligence White Paper · The ExtraHop IT Operational Intelligence Platform: How It Works and Real-World Results Table of Contents Introduction !! ! ! !

T he E x t raH op IT O per a t iona l I n te l l i genc e P la t fo r m: H ow I t Wor ks and R ea l -Wor ld R esu l t s

WHITE PAPER The ExtraHop IT Operational Intelligence Platform: How It Works and Real-World Results By Tyson Supasatit Technical Marketing Manager Abstract

ExtraHop accelerates IT transformation with real-time IT operations analytics. The ExtraHop platform equips all IT teams with correlated, cross-tier visibility so they can answer the question, “What is happening in my environment right now?” With this operational intelligence, organizations in all industries have built a sustainable competitive advantage by running their IT more efficiently and with greater agility. This white paper explains the technology that powers the ExtraHop platform and how IT organizations use ExtraHop to accomplish critical IT tasks and add significant value to the business.


Table of Contents Introduction 3 Wire Data Can Transform IT Operations 3

Wire Data: Unlocking the Potential of Data on the Wire 3

How ExtraHop Works 4

Simple, Non-Invasive Deployment 5 Full-Stream Reassembly and Full-Content Analysis 5 Streaming Datastore and Intelligent Alerting Engine 6 Open, Extensible, and Shareable Platform 6

ExtraHop in Action 7

End-User Intelligence 7 Proactive Remediation 7 Infrastructure Optimization 8 Application Optimization 9 IT Decision Management 9 IT and Business Intelligence 10 Security and Compliance 11

Conclusion 12


Introduction ExtraHop enables companies to achieve a sustainable competitive advantage through more proactive and agile IT Operations. Organizations that have adopted the ExtraHop operational intelligence platform have transformed their IT Operations so that they are making informed decisions regarding IT infrastructure, answering questions that impact millions of dollars in revenue, and preventing problems instead of reacting to them. In short, the ExtraHop platform is helping these IT organizations become a strategic asset to the business.

Wire Data Can Transform IT Operations Technology is not a panacea, but the right set of solutions is essential to help IT Operations respond faster to changing business needs. Most IT organizations purchase monitoring tools to meet narrow departmental requirements, not according to a strategic, overarching plan. This behavior results in an ad hoc accumulation of niche products that exist in siloes, not the cohesive IT operational intelligence framework that will equip these organizations to accelerate IT maturity. ExtraHop is part of a set of next-generation technologies that together equip IT teams with holistic operational intelligence. Fundamentally, there are only four key sources of data available for IT operations management. Each data source is necessary, although the role and importance of each is evolving.

• Machine data, including logging provided by vendors, SNMP, and WMI. This information about system internals helps IT teams identify overburdened machines, plan capacity, and perform forensic analysis of past events. New distributed log-file analysis solutions enable IT organizations to address a broader set of tasks, including answering business-related questions.

• Agent data from byte-code instrumentation, call-stack sampling, and custom logging. Code diagnostic tools have traditionally been the purview of Development and QA teams, helping to identify hotspots or errors in the software code. New SaaS vendors have dramatically simplified the deployment of agent-based products.

• External data from synthetic transactions and service checks. This data enables IT teams to test common transactions from locations around the globe.

• Wire data, which has traditionally included NetFlow, HTTP traffic analysis, and packet capture. The information available off the wire historically has been used for measuring, mapping, and forensics. ExtraHop unlocks the tremendous potential of real-time wire data, opening up vastly greater opportunities and serving as the lynchpin of IT operational intelligence.

Wire Data: Unlocking the Potential of Data on the Wire The information needed for operational intelligence has always existed on the wire, but previously was not available in real time or in a way that was meaningful to all IT teams. The ExtraHop platform introduces revolutionary new high-speed packet processing capabilities that make it possible—for the first time—to fully analyze the wealth of data passing over the wire in real time and present it in a way that makes sense for network engineers, security professionals, DBAs, storage administrators, application architects, application owners, and others. ExtraHop extracts this real-time wire data without the use of agents.


ExtraHop provides value to all IT teams, equipping them with real-time operational intelligence needed to answer the question, “What’s happening in my IT environment right now?”

The traditional approach to obtaining visibility across all the tiers of an IT environment would be to pull as many discrete metrics as possible from each tier and then try to make sense of the collected data with analysis and reporting servers. This bottom-up approach provides information that is often hours old, uncorrelated, and frequently unreliable because of poor integration between various tools. Worse still, these legacy tools become more expensive and require more effort to manage as the environment grows in complexity, leaving IT organizations paying more and getting less in return. ExtraHop takes a radically different approach, using wire data as the source for cross-tier insight. The network is the common element that ties all components of the application delivery chain together, even as those components become more numerous and distributed. Each component communicates with others using transport and application protocols. These protocols definitively describe what is happening in the IT environment. The networking adage, “packets don’t lie,” applies here. Moreover, these protocols seldom change, making the network the ideal instrumentation point in increasingly heterogeneous and fluid environments.

How ExtraHop Works The ExtraHop platform performs full-stream reassembly and full-content analysis of network traffic to extract IT and business insights. ExtraHop analyzes application transactions continuously and in real time, at speeds up to a sustained 20Gbps. An open and extensible platform, ExtraHop enables IT teams to define and implement new metrics within minutes, and integrates seamlessly with manager of managers (MOM) systems and other next-generation monitoring products such as Keynote, New Relic, SevOne, and Splunk.


Simple, Non-Invasive Deployment The ExtraHop platform is a completely passive, out-of-line network appliance that is easy to deploy and manage. Deployed using a network tap, SPAN, or other data-access technology, ExtraHop analyzes every application transaction, not just a sample portion of network traffic as with synthetic transactions. Where a tap or SPAN are not available, ExtraHop offers a high-speed packet forwarder that can be packaged in automated configuration utilities such as Chef. As soon as traffic is detected by the platform, ExtraHop’s Context and Correlation Engine automatically discovers and classifies devices, both physical and virtual, and determines relationships between devices based on MAC addresses, IP addresses, naming protocols, and other heuristic elements. As the IT environment changes—with new software builds and upgraded infrastructure components, for example—ExtraHop automatically detects and adjusts to those changes.

For distributed environments, the ExtraHop Central Manager delivers a consolidated view of wire data from multiple ExtraHop appliances, enabling organizations to gain visibility into the communications of hundreds of thousands of devices across datacenters and branch offices. IT administrators can easily update the platform firmware remotely, making the ExtraHop platform an ideal choice for deployment within physically isolated, or lights-out, datacenters. Full-Stream Reassembly and Full-Content Analysis While other products only inspect L4 headers, only the ExtraHop Context and Correlation Engine performs full-stream reassembly continuously in real time. This advanced approach reassembles multiple packets into a stream and reconstructs transactions, flows, and sessions—a prerequisite for true application fluency. ExtraHop is purpose-built for production on-premises and cloud environments, supporting real-world traffic patterns such as IP fragments, out-of-order segments, and microbursts. When packet loss occurs on the monitoring link, ExtraHop resynchronizes and recovers. Because it was built to take full advantage of multicore processing, the ExtraHop Context and Correlation Engine is able to perform full-stream reassembly at a sustained 20Gbps. Through full-stream reassembly, the ExtraHop Context and Correlation Engine can analyze the full content of transaction payloads (not to be confused with packet payloads) and extract crucial details such as the specific URI included in a HTTP 500 Error, slow stored procedures in a database, or the location of a corrupt file in network-attached storage. ExtraHop offers protocol modules for web applications, NoSQL and relational databases, network-attached storage (NAS) and storage-area networks (SANs), directory services, and industry-specific protocols for financial and telecommunications verticals.

The ExtraHop Context and Correlation Engine is built for massively scalable transaction analysis—up to a sustained 20Gbps.


Streaming Datastore and Intelligent Alerting Engine The ExtraHop Context and Correlation Engine includes a high-speed, streaming datastore that records and retrieves performance and health metrics in real time. Optimized for time-sequenced telemetry, the datastore writes to and reads from underlying block devices directly, translating into reliably superior recording and retrieval speeds without the tuning and management required by a relational database. The streaming datastore powers an intelligent alerting engine that helps IT teams prevent small issues from growing into larger problems. IT teams can configure the default alerts and create new alerts for behaviors and events such as network activity, webserver and database errors, payload length, slow transactions, and expiring SSL certificates.

Open, Extensible, and Shareable Platform ExtraHop is a platform for IT Operations innovation, equipping IT organizations to quickly meet new requirements for visibility and insight. ExtraHop offers generous options for integration with existing IT management toolsets, including policy-based logging of events that are only available through analysis of wire data. Best of all, the innovative extensions for the ExtraHop platform can be easily bundled, shared, and improved upon through the ExtraHop community.

• Open – ExtraHop works with other management and monitoring solutions using both push and pull integration. For push integration, syslog export enables IT teams to send policy-based, event-driven metrics to any IT management console, custom Big Data analysis store, SIEM product, or third-party management tool such as Keynote, SevOne, or Splunk. For pull integration, IT teams can use SDK documentation to access the same API that is used by the ExtraHop web interface. This API provides immediate access to any metric in the ExtraHop datastore.

• Extensible – ExtraHop provides a programmatic interface to its Context and Correlation Engine that IT teams can use to define and implement new custom metrics in minutes. Application Inspection Triggers (AI Triggers) make it possible to rapidly answer questions such as “How many duplicate orders are occurring and whom do they affect?” “Which client types are affected by this new update?” “What users are accessing this sensitive storage file?” and “What are the front-end web requests that are associated with these slow SQL queries?”

• Shareable – What makes ExtraHop a true platform is the ability to package and share extensions. IT teams can package together dashboards, alerts, geomaps, dynamic groups, and AI Triggers and then share them within the organization or with the wider ExtraHop user community. These solution bundles can be downloaded and extended to meet particular IT management tasks or application monitoring requirements. In this way, IT teams benefit from community-driven enhancements by quickly implementing and building on the innovation of others.

“ExtraHop gives us the intelligence we need to continually increase efficiency and sustain a competitive advantage.” Drew Garner Director of Cloud Architecture Concur


ExtraHop in Action Companies from a wide range of industries are using ExtraHop to transform how they run IT. The following examples provide a glimpse into what is possible with the IT and business insights delivered through the ExtraHop platform. End-User Intelligence Unlike monitoring products that only show what users are doing and experiencing on the frontend, ExtraHop can correlate user activity and experience to performance in the backend IT infrastructure. In other words, ExtraHop does not just show what users are experiencing, it also explains why. One telecommunications service provider used ExtraHop to identify the specific users whose devices were adversely affected by a firmware update. Traditionally, service providers would rely on tools that show which systems are communicating and when. Only ExtraHop enables IT teams to see what is actually being said between systems. In the case of the service provider, ExtraHop reconstructed and analyzed the contents of all Diameter transactions, including attribute-value pairs (AVPs) such as customer IDs and handset type. With this information, the service provider could easily isolate which subscribers were affected by the firmware update and work with the handset manufacturer to develop a fix.

A large research hospital had spent weeks trying to isolate the cause of extremely slow Citrix logins every morning around 8:30 a.m. With ExtraHop, the hospital identified severe contention at the storage tier—a single doctor was pulling down 2GB of photos stored in his My Pictures folder every time he logged in. By deleting the My Pictures folder from user profiles, the IT team at the research hospital solved the problem, helping to earn goodwill from users and paving the way for an expansion of the hospital’s VDI deployment. Proactive Remediation In an ideal world, everything is tested and works perfectly when deployed to production. Reality works much differently, requiring IT Operations team to maintain real-time visibility into the performance of production applications. ExtraHop provides trend-based early-warning alerts for the entire production environment so that IT teams can proactively identify and fix problems fast. Prior to deploying ExtraHop, Alaska Airlines’ IT team had no way of monitoring the real-time performance of their Informix database. This database underlies Alaska Airlines’ weights and balances application, which must calculate weight distribution on planes before they are cleared for takeoff. The IT team could not continuously run profilers on the database in production because of the high overhead required. With ExtraHop, Alaska Airlines monitors the performance of its critical Informix database continuously with zero overhead. By reconstructing and analyzing all transactions, ExtraHop provides the IT team with real-time database performance metrics, including details such as errors, methods, and users.

“ExtraHop has proven itself to be very valuable to Alaska Airlines—and no other solution in our environment has been able to analyze Informix the way that ExtraHop does. It has enabled us to quickly and accurately diagnose several issues that would have been impractical or impossible to pin down previously.” Kris Kutchera VP of Information Technology Alaska Air Group


With ExtraHop, IT organizations can monitor the performance of databases in production, including details such as methods, without running any database profilers, which can add onerous system overhead.

Infrastructure Optimization Oftentimes, IT Operations teams do not root out inefficiency from their infrastructure because no one is complaining and there are other urgent projects waiting. ExtraHop makes it easier to identify inefficient activity as well as poor performance that users quietly tolerate. Detailed metrics from ExtraHop also help IT teams to determine the optimal settings for application delivery controllers (ADCs), WAN optimizers, and network-attached storage given the unique requirements of their applications.

By assembling the TCP state machines for every endpoint, ExtraHop can monitor sophisticated TCP metrics such as PAWS-dropped SYNs, receive-window throttles, retransmission timeouts, and Nagle delays.

At one company, an Operations team member was using ExtraHop to find SQL queries that were good candidates for caching. In the course of his investigation, he saw that CIFS traffic comprised 70 percent of network bandwidth. This seemed odd, so he drilled into the CIFS transaction details and found some familiar file names in the list—files associated with the company’s homegrown logging system! A bug in the log archive script was causing five million files to be copied across the network unnecessarily. The network team was unfamiliar with the logging system and had assumed that this traffic growth was organic. In fact, they were preparing a forklift upgrade of the network infrastructure to handle the increase. However, with the archive script fixed, network utilization dropped by an astounding 70 percent, which helped the company defer hundreds of thousands of dollars in capital expense. Legacy network-monitoring tools would not have helped in this case. Only ExtraHop, with its ability to analyze L7 application-level details, is able to distinguish CIFS traffic and list the filenames for each transaction.


Healthcare services provider MedSolutions used ExtraHop to identify a misconfiguration in their F5 BIG-IP that was adding network latency for users in the corporate office. ExtraHop showed a high number of retransmission timeouts (RTOs) on LAN segments behind the corporate load balancers—behavior that was obvious looking at TCP analysis in ExtraHop, but would have required careful investigation with a packet sniffer to reveal otherwise. The IT team found the F5 BIG-IP was misconfigured with a TCP profile for a wide-area network instead of local-area network. In addition to RTOs, ExtraHop tracks sophisticated TCP metrics such as Nagle delays and tinygrams, which help network teams and system administrators to determine which congestion control algorithms to turn on.

Application Optimization ExtraHop supports the entire application management lifecycle, providing architects, developers, testers, and operations teams with a way to measure how updates and configurations affect performance. With consistent and trusted data from ExtraHop, stakeholders can work together more effectively to ensure fast and smooth rollouts. ExtraHop also provides operational intelligence for packaged applications, enabling IT teams monitor performance across all tiers of the application delivery chain. A large outdoor equipment retailer rolled out mobile point-of-sale (POS) devices in preparation for the holiday shopping season. The company estimated that by reducing lines at checkout counters, they would recoup nearly one million dollars in lost sales. However, store managers complained that performance for these mobile devices was so slow that they were useless, with product scans taking from 30 seconds to one minute. Using SSL analysis in ExtraHop, the IT Operations team discovered the third-party mobile POS software was performing 15 SSL handshakes per transaction. The vendor provided a fix so that the application used recognized SSL tokens, reducing transaction times to less than one second—faster even than traditional POS terminals. IT Decision Management ExtraHop provides IT organizations with the insight they need to make decisions about capacity planning, application migrations, decommissioning legacy systems, and infrastructure changes.

Practice Fusion, a provider of web-based electronic medical record (EMR) solutions, used ExtraHop to migrate a portion of their web application from physical to virtual infrastructure. This particular workload was customized to run a particular HP server platform, and previous attempts to virtualize the workload had failed because the software encountered race conditions and similar problems. With ExtraHop, the IT team at Practice Fusion measured baseline performance for the application on the dedicated HP servers and on a parallel virtual infrastructure. ExtraHop showed that performance was slightly better on the virtual infrastructure, helping Practice Fusion to avoid spending $75,000 purchasing new hardware and revalidating the software for the new platform.

“With ExtraHop, MedSolutions has a real-time, holistic view of all of our applications and infrastructure. This operational intelligence enables us to quickly answer questions and take action to improve performance and efficiency.” Satish Dave CIO MedSolutions


Concur relies on more than 1,000 database instances to power its SaaS expense reporting solution. So when the R&D Operations team wanted to dramatically expand the cache in front of the database, finding the best SQL workloads to migrate to the cache would have been next to impossible using database profilers. Instead, Concur used ExtraHop to analyze database transactions for the entire infrastructure and determine the total weight for each SQL query by calculating the number of times that query was run by the time required to return the data. This information helped Concur to justify expanding its cache from 13,000 hits per day to more than 500 million hits per day, which in turn resulted in a 20 percent improvement in application performance. IT and Business Intelligence The wire data that flows through IT environments contains a wealth of information that is valuable to the business. ExtraHop enables IT organizations to tap this valuable business data to help drive additional revenue and analyze customer behavior and pricing trends. A large financial services firm knew its system was duplicating orders, but could not find the source of the problem or discover which accounts were affected and how frequently. The IT team used ExtraHop to analyze the XML payload and extract details specified by the Orbital payment protocol, including user, merchant ID, account number, and order ID (see below). Through syslog export, the IT team set up ExtraHop to automatically forward this specific information to their Splunk deployment for search and analysis.

ExtraHop enables IT teams to easily mine the full transaction payload and extract metrics that are relevant to the business, such as account numbers and order IDs.

“We set up ExtraHop in our staging environment so that the engineering team can see the impact of new code against our baseline performance. With visibility across all tiers of the environment, we can see whether a performance problem is due to infrastructure, misconfiguration, or possibly a code-level issue.” John Hluboky VP of Technical Operations Practice Fusion


A major online advertising platform used ExtraHop to identify customers who had exhausted their prepaid keyword accounts. Before using ExtraHop, the IT Operations team had no visibility into the cause for HTTP 500 errors returned by their own API. ExtraHop enabled the team to examine the HTTP payload for these transactions and see what was causing the errors. In many cases, the application was returning an HTTP 500 message when the prepaid limit was reached, not because of an actual server problem. By proactively identifying which large clients needed to replenish their account balances, the online advertising platform is able to collect revenue that would otherwise be lost. Security and Compliance Because ExtraHop provides detailed metrics for every transaction passing over the wire, it provides security teams with valuable information about who is accessing which systems and how they are doing so. For example, IT teams can use ExtraHop to see which clients are accessing the database using root or system administrator accounts. ExtraHop also facilitates compliance audits by providing audit teams with detailed reports showing database and storage activity that is in violation of policy, including unauthorized access to specific directories and files. The IT Operations team at an online retailer was trying to stop an attacker that was extracting data from the database through SQL injection. Using ExtraHop, the IT team isolated the web requests that resulted in HTTP 500 errors and database responses in excess of 5MB. The IT team then used ExtraHop to analyze the web requests and find both the IP address of the attacker and the database vulnerability they were trying to exploit. This information enabled the IT team to quickly block connections from the attacker and patch the database. Many IT organizations use ExtraHop to defeat repeated brute-force FTP hacking attempts from overseas IP addresses. These IT teams set an alert in ExtraHop that fires when a specific client fails three FTP login attempts within 30 seconds and triggers a Fail2Ban action for that particular client IP address.

With ExtraHop, IT teams can easily track all SSL certificate expirations and RSA key sizes.


© 2013 ExtraHop Networks, Inc. All rights reserved.

Conclusion Operations teams stand at the intersection of IT and the business. Increasingly, business success will depend on how quickly and how well these IT Operations teams respond to new demands. ExtraHop delivers the greatest results in companies that believe how they run IT matters. Organizations across a wide variety of industries—including telecommunications, financial services, retail, healthcare, and government—use ExtraHop to build sustainable advantages over their competition. By running IT operations better, these companies can respond faster to new requirements, roll out innovative new services faster, provide superior user experiences, and quickly gather business insights. Using ExtraHop’s visibility and insight, Operations, Development, Security, and other teams are working together to continually improve security, performance, and availability. At the same time, these IT teams are cutting costs through a more elegant, scalable, and flexible framework for IT operational intelligence.

ExtraHop provides the real-time operational intelligence required to make IT more agile and proactive. The world’s best-run IT organizations use ExtraHop to manage more than half a million devices and monitor over a trillion transactions daily, including Adobe, Alaska Airlines, Concur, Expedia, and Microsoft. ExtraHop Networks, Inc. 520 Pike Street, Suite 1700 Seattle, WA 98101 USA www.extrahop.com [email protected] T 877-333-9872 F 206-274-6393 Customer Support [email protected] 877-333-9872 (US) +44 (0)845 5199150 (EMEA)

Documents

ExtraHop IT Operational Intelligence White Paper · The ExtraHop IT Operational Intelligence Platform: How It Works and Real-World Results Table of Contents Introduction !! ! ! !