Converting Hype into Value Final - IBM Big Data & Analytics Hub · 2013-10-18 · Converting Hype into Value With Big Data and Analytics Colin White, BI Research ... The objective

08 Fall

Converting Hype into Value With Big Data and Analytics Colin White, BI Research October 2013 Sponsored by IBM

Converting Hype into Value With Big Data and Analytics

Copyright 2013 BI Research, All Rights Reserved. 1

INTRODUCTION There is a tremendous amount of interest in big data and there is substantial business value to be gained from big data projects. Distinguishing true business value from vendor marketing hype, however, is sometimes difficult. Another problem is that vendors often focus on big data technologies and fail to emphasize the business benefits and use cases for big data. In addition to big data technologies, it is equally important to consider the analytics and business insights derived from big data that can be used to improve business decision making, optimize business processes and increase business innovation. It is the insights gained by analyzing big data that provide the real business value for organizations.

The objective of this paper is to extract reality from the hype and to help you understand the business benefits of big data and the significant role analytics play in providing this value. Along the way, the paper will also discuss briefly how IBM (the sponsor of this paper) approaches big data and analytics.

THE ROLE OF BIG DATA AND ANALYTICS Big data is most often associated with the ability to extend the analytic environment to leverage large volumes of new types of data, such as sensor data and social computing data from Web sites like Twitter and Facebook. Although analyzing new types of data increases intelligence about business operations and improves the overall decision-making process, the scope of big data is broader than this.

Big data represents the opportunity for organizations to gain from a new wave of analytics innovation that supports new types of data and extends the use of analytics to a broader set of users while also at the same time offering a richer and user-friendly set of analytic capabilities. Big data advances, for example, help organizations grow the use of predictive techniques by business users. Marketing departments can now use predictive analyses to extend traditional analytics about past and present customer behavior with the capability to predict more easily possible future behavior and outcomes. This extends marketing beyond simply analyzing large customer segments to personal one-on-one marketing.

Big data can be used by many different parts of the organization, including front-office sales and marketing, middle-office finance and human resources, back-office procurement and supply chain, and research and development. It can also be used by a broad set of industries and government organizations. To begin with, these organizations are focusing their initial big data projects on areas that provide the biggest and fastest return on investment (ROI) – examples here include more focused customer marketing and call center optimization, improved fraud detection and risk management, and supply chain optimization.

IBM is also focusing its efforts on important key areas where big data and analytics solutions offer the best ROI. These include the ability to:

• Acquire, grow and retain customers

• Optimize business operations and reduce fraud

• Manage risk

Business insight provides the real value for big data and analytics

The scope of big data is more than just accessing and processing new types of data

Big data can be used by a broad set of business areas, industries and government organizations



• Transform financial processes

• Improve IT economics

Business organizations, and vendors such as IBM, are also focusing their big data development efforts on creating new business models and on building analytic applications that could not be supported before because of

• incomplete or limited accessible data for generating the required analytics,

• the high hardware and software costs involved, or

• technology limitations, such as poor performance or inadequate analytic capabilities.

Big data technologies not only extend the power and breadth of analytics, but also enable analytic solutions to be deployed at a lower cost and with better performance. The availability of low-cost systems such as Hadoop appliances and cloud-based solutions demonstrates this trend in the industry. Of course it is important to consider the total cost of ownership of a solution, and not just its initial cost.

The direction of vendors to improve the usability, functionality, price/performance and speed of analytic processing is a key success factor for big data and analytics projects. IT is already struggling to keep pace with increasing data volumes and the growing demand for analytics, and current IT infrastructure and technologies simply will not be able to support and exploit the value of big data unless the existing infrastructure can be enhanced and extended with new technologies, tools and practices in a cost effective manner.

Big data is not a single technology, but a set of related and overlapping technologies. There is no one single solution that can satisfy everybody’s big data management and analytic needs. Instead, vendors offer a menu of different choices that enable customers to deploy analytic systems that are optimized to suit certain business use cases and workloads. This optimization may involve improving performance, reducing costs, enabling new sources of data to be explored and captured, and/or new types analytics to be produced.

To overcome the bottlenecks and boundaries of the current IT infrastructure, vendors tackling big data solutions offer advances in both analytics and data management. The key advances in each of these areas are discussed in more detail below.

ADVANCES IN ANALYTICS There have been many advances in the analytics area over recent years, many of them associated with big data. These advances can best be explained using the diagram shown in Figure 1.

Analytic processing creates and delivers four different types of intelligence that aid organizations in making more effective decisions, optimizing business processes and in improving business innovation. These four types are descriptive intelligence,

Big data technologies also enable analytic solutions to be deployed at a lower cost and with better performance

Big data involves a set of related and overlapping technologies



diagnostic intelligence, predictive intelligence and prescriptive intelligence.1 Figure 1 shows these four types and the types of analyses they support. It also gives examples of the way this intelligence is delivered and visualized by business users.

Big data offers increased business value for each of the types of intelligence. For traditional descriptive and diagnostic intelligence used by most organizations today, big data helps enhance existing data warehouse and analytic solutions to take advantage of new and evolving analytic approaches and new sources of data. With predictive and prescriptive intelligence, big data helps organizations enhance existing projects and also extend the analytics environment to enable more sophisticated analytics to be used by a broader user audience and to be applied to a larger set of business problems and requirements.

The analytic advances in the areas of intelligence shown in Figure 1 can be broken down into four main categories: new and improved analytic techniques, enhanced data navigation and visualization, automated decision management and agile deployment.

New and improved analytic techniques are used to uncover patterns in both existing and new types of data. These techniques fall into three groups. The first group includes techniques that aid business users in analyzing new types of data, for example, text, images, video or documents from office and content management systems. The second group enhances techniques for predictive modeling and analysis. The third and last group includes prebuilt function libraries containing advanced statistical and analytic functions. These libraries allow business users to exploit the value of the advanced functions without having to know how to code them or how the underlying algorithms are implemented. The libraries may be supplied by vendors, be available under open source license, or may be written by the 1 Note that IBM uses a fifth category, cognitive intelligence, which it uses to associate technologies such as IBM Watson with the ability to base decisions and actions on what has been learned from prior experiences.

Figure 1. Types of intelligence and analyses

Big data helps organizations extend the analytics environment to enable more sophisticated analytics to be used by a broader user audience

Analytic advances: new and improved analytic techniques



customer. The functions often run in the underlying data management system, which enables them to take advantage of the parallel computing capabilities of that system to improve performance. IBM’s direction here has been to build out a business analytics portfolio, which includes a range of analytics capabilities from Cognos reporting and analysis to SPSS predictive analysis and IBM Watson cognitive capabilities. One particular area of focus is improved ease of use so that complex analytic techniques are hidden from the user or embedded in business processes.

Enhanced data navigation and visualization helps business users uncover insights while exploring and navigating through large volumes and varieties of data. Examples of capabilities include faceted navigation, tree maps, text and special visualizations. Key objectives are to provide consumer-like user interaction and experience, to support a wide skill base ranging from business managers to data scientists, and to support not only desktop environments, but also mobile devices. For data navigation, IBM’s InfoSphere Data Explorer (which is based on technology acquired from Vivisimo) enables the searching and navigation of both structured and multi-structured data. Another key product is the IBM SPSS Analytic Catalyst. This browser-based tool sifts through large data sets and visually presents key variables and statistically interesting relationships in an easy to understand visual format that does not require sophisticated analytic skills to interpret. IBM also provides free analytic software and new visualizations via the analyticszone.com Web site.

Decision management predictive models and business rules alert people about exceptions they need to focus on. They can also be used to automate decisions and take real-time actions. Fraud detection is a good use case here. Based on prior fraudulent activity, predictive models and rules can be created that can be used by business processes to check credit card transactions for potential fraud. If the results indicate that the transaction is potentially fraudulent it can be rejected, or referred to a person for manual evaluation. The predictive models and rules can be refined over time as the system gains more knowledge about fraud by analyzing data from multiple customer channels. IBM’s solution in this area is IBM SPSS Decision Management. This product can be used in conjunction with IBM InfoSphere Streams (described below) for making automated and real-time decisions and actions on large volumes of streaming data.

Agile deployment identifies capabilities that are being added to the analytics environment to provide faster time to business value. Increasing the sophistication of analytic processing and providing access to new sources and types of data may improve decision making and action taking, but these advances can only provide real business benefits if they can be deployed in a timely manner. The prebuilt function libraries, enhanced data navigation and visualization, and automated decision management discussed above all improve usability and enable organizations to become more agile in deploying analytic solutions.

Another important requirement here is ability to reuse and deploy analytic solutions across multiple target systems including on-premises platforms and cloud-based environments. The industry direction toward supporting cloud computing is important because it allows analytic solutions to be deployed rapidly and without incurring the upfront costs and implementation efforts involved in upgrading the IT infrastructure to support new analytic and data management approaches. Cloud computing is particularly well suited to analyzing data that is already resident in the

Analytic advances: enhanced data navigation and visualization

Analytic advances: decision management

Analytic advances: agile deployment



cloud, for example, web store data, social computing data, and data from cloud-based operational business systems. The IBM direction in this area is to deliver analytics offerings via the IBM SmartCloud.

The cloud environment is not suited to all big data and analytics projects, but this important direction now gives organizations the choice of multiple deployment options: on-premises or in the cloud or a combination of the two (often called a hybrid cloud environment). The appropriate option can then be selected based on the ability of the platform to support project requirements and budgets.

A somewhat different, but equally important, direction in the area of agility is the move by vendors to incorporate many of the advances outlined in this paper into prepackaged business domain and industry specific analytic applications and templates. This trend speeds up the deployment process and offers another deployment option for organizations wishing to exploit the business value of big data. An IBM example is its Next Best Action Signature Solution, which uses real-time analysis to anticipate customer behavior and deliver the most appropriate recommendation for any given customer at the right time across different channels. This helps promote more personalized customer interactions and longer-term customer relationships.

ADVANCES IN DATA MANAGEMENT Big data analytic projects can only be successful if the underlying data management platform can provide the required performance, scalability, reliability, security and governance. As with the analytics environment, vendors are offering several deployment options for big data projects. Three important options to note are: analytic relational database platforms, non-relational systems and stream processing systems.

Analytic Relational Database Platforms These platforms consist of packaged hardware and software systems that have been optimized to improve the price/performance of analytic processing. They make possible what could not be supported before for cost reasons, or because the analytics could not be delivered in a timely manner. These platforms are often also enhanced to handle a wider variety of data types, for example, multi-structured data, such as web logs or sensor data. This allows business users to blend and analyze different types of data from both internal and external data sources.

The improved price/performance offered by these platforms also enables organizations to keep more detailed data online for longer periods of time, which reduces the need to aggregate or subset the data – this in turn can help improve the accuracy of the analytical results.

Many of the relational database systems used by these platforms have also been enhanced to support a number of different data storage options that can be selected based on workload requirements. Examples here include columnar and row-based data stores and in-memory computing. IBM solutions here include the IBM DB2 with BLU Acceleration relational database system and the IBM PureData Systems integrated hardware and software appliances.

Prepackaged analytic solutions also improve agility

Data management advances: analytic relational database platforms provide improved price/ performance



Non-Relational Systems These systems are not new and there are many different types of non-relational systems ranging from high-performance file systems and document management systems to graph databases.

One area of focus at the moment from an analytics perspective is the Hadoop distributed computing environment. The objective of Hadoop is somewhat similar to analytic relational database platforms – improved price/performance – but Hadoop is more likely to be used for managing, transforming and analyzing large volumes of multi-structured data by batch applications, whereas analytic relational database platforms are geared toward a higher-percentage of structured data and both pre-planned and ad hoc processing.

Hadoop is being used in a variety of ways, for example:

• As a data refinery for transforming data and delivering the transformed data to downstream systems (such as a data warehouse)

• For analyzing web and social media

• As a queryable data archive

IBM’s solutions in this area include IBM InfoSphere BigInsights and the IBM PureData System for Hadoop hardware and software appliance.

Stream Processing Systems These systems analyze both structured and multi-structured data as it flows from sensor devices and through and between different IT systems. It is a unique approach because it can filter and analyze data in motion without the need to persist it in an enterprise data warehouse first. This is particularly useful for real-time decision making and action taking and also in situations where it is not cost-effective or required to persist large volumes of detailed data in an enterprise data warehouse. IBM’s solution in this area is IBM InfoSphere Streams.

It is important to emphasize that the three advances outlined above should not be used to build separate data silos, but should instead be used to create a cohesive data system that extends the existing data management capabilities of an organization. As will be seen below, many big data and analytics solutions require a combination of different data management technologies.

THE BENEFITS OF BIG DATA Big data and new and evolving analytics technologies are used to enhance and extend, rather than replace, the traditional decision-making environment. Figure 2 summarizes some of the key improvements that big data helps bring to existing analytic and data warehousing systems.

The top half of Figure 2 lists the extensions provided by the analytics advances of big data. These new analytic extensions, for example, enable powerful investigative computing platforms to be deployed by data scientists to model and blend together data from a variety of different sources to look for ways of improving existing predictive models and analyses and to investigate potential new business opportunities. The results of this investigation work – updated models, new business

Data management advances: non-relational systems such as Hadoop

Data management advances: stream processing systems for analyzing in motion data

Top half of Figure 2 summarizes the analytic advances of big data



rules, new analyses and/or new data – can then be promoted back into the production decision-making environment such as an enterprise data warehouse (EDW). In some cases, the results may lead to a new built-for-purpose line-of-business (LOB) application, which can be deployed on the same investigative platform or a different platform based on requirements. Although these new LOB systems are usually analytics-driven, they are, nevertheless, operational in nature because they drive day-to-day business operations. Many of these LOB applications blur the line between what is operational and what is analytic processing. The management of online display advertising is an example of a hybrid LOB application.

The business models and rules generated by an investigative computing platform may also be run directly as a part of a business process with the assistance of a decision management capability. The direct embedding of models and rules in business processes helps speed up the decision making process, and in some cases, fully automate business decisions. Examples of applications here include fraud detection, next best customer action, product offers to customers to avoid churn, etc. These new big data analytic capabilities not only extend the power of the analyses that can be performed, they also change the way analytics applications are built and deployed. Data scientists, for example, can now use an investigative computing platform to explore data, identify different patterns, and experiment with different algorithms without the need for any of the information being stored in the traditional EDW environment.

The bottom half of the table in Figure 2 lists extensions provided by the data management advances of big data. Value here comes from improved price/performance, the capability to make close to real-time decisions, and the ability to process a much richer set of data sources.

Figure 2. What does big data add to the traditional decision-making environment?

Bottom half of Figure 2 summarizes the data management advances of big data



The additional data sources that can be now blended into the analytical environment can help answer questions that could not be answered before. For example, there are now many more channels of customer information available for analysis. Information can also be related to very different data sources – product sales could be affected by other factors such as the weather, a drought, or a natural disaster. These kinds of relationships can now be investigated more thoroughly.

Another advantage of big data is that more detailed data can be kept online for longer, which helps improve the accuracy of the results. Analytics can now be generated on a full set of detailed data, rather than on aggregated or sample data. Some retail companies, for example, are now keeping ten years of customer data online, whereas in the past they only kept two years’ worth of data before summarizing it.

CHOOSING THE RIGHT SOLUTION As discussed earlier, big data is not a single market or technology and there is no single solution that can satisfy everybody’s needs. Instead, organizations will likely use multiple analytic and data management approaches. The challenge is deciding which to use when and how to interconnect the various systems involved. The number of options can be daunting and this is one of the reasons why IT tends to focus on technology differences between products, rather than on matching technologies to different use cases.

Although big data technologies are still evolving, there are now a number of use cases and customer case studies that help identify many of the benefits that can be obtained from big data. Figure 3 summarizes six common use cases together with application examples of each.

New sources of data can be blended with existing data to gain new insights

Organizations will likely use multiple analytic and data management approaches

Figure 3. Big data use cases and application examples



Real-Time Monitoring and Analytics Analytics are increasingly being used to monitor business operations in real time and to take action if certain business events occur. Fraud detection is an example we have already discussed in this context. Models are built based on known fraudulent situations and rules generated for use by decision management software to help business processes track and check for fraudulent situations. The business value of real-time analytics is the ability to detect fraud faster, which reduces the risk of financial impact to the organization.

Another approach to real-time monitoring and analytics is to use a stream processing system to analyze data as it flows through the business. This is particularly useful for analyzing events from sensors embedded in networks, smart grids, oil wells, aircraft, and so forth. The readings from these sensors can be monitored over time and when an out-of-line situation is detected (a possible equipment failure is predicted, for example), alerts are sent and appropriate actions taken. Stream processing systems can also be used for matching and correlating data from unrelated data streams, for example, weather data and sales data.

Near-Real-Time Analytics The objective of near-real-time analytics is basically the same as for real-time analytics, to speed up the decision making process. The main difference is that the need for split second information is not as high in a near-real-time environment – some latency is acceptable. Unlike fraud detection, for example, re-routing a package in the event of a potential weather delay does not have to been done in a split second.

Near-real-time decisions are possible using high-performance analyses performed against low-latency data in a data warehouse or high-speed analytic relational database. The data warehouse or data store is updated from source systems at intervals to match the data latency requirements of the analytic processing. The low-latency data and prepackaged analytic results can also be accessed directly by a business process using a web service call. Customer next best offer or option is an example of an application that fits into this category. The actual offer made to the customer may depend on the channel the request comes through (e-mail, call center, web chat, mobile device, etc.), the type of customer service call being handled, value of the customer, churn risk, or even possibly the service-center agent handling the call. Regardless, the ability to quickly make a valuable offer to a customer helps enhance customer satisfaction and improve customer retention.

Data Refinery With growing data volumes and data sources, there is significant interest by organizations in storing, managing and transforming detailed structured and multi-structured data on a single online data platform. This platform can then be used to feed downstream decision-making systems as required. This approach reduces data management and data transformation costs, and makes more data available for analysis online. In the past, the cost to maintain large amounts of data online has often been prohibitive, and as a result companies have often had to aggregate data to reduce costs. With the advent of systems such as Hadoop it is now possible to cost-effectively maintain large amounts of data online and this is one of the fastest growing use cases for Hadoop. A Hadoop data refinery in a retail organization could,

Real-time monitoring and analytics can be used for fraud detection

Near-real-analytics can be used for customer next best offers in a call center

A data refinery is one of the fastest growing use cases for Hadoop



for example, be used to collect and manage all sales- and customer-related detailed data (point-of-sale, web, supply chain) for down stream analysis. One large retail organization maintains ten years’ worth of sales data in Hadoop and the last two years sales data in its traditional data warehouse. Additional data is then brought into the data warehouse from Hadoop as required. A key requirement here is robust and high-performance connectors between Hadoop and other systems.

Analytics Accelerator An analytics accelerator is a separate analytic platform that is used to accelerate the performance of certain analytic workloads. For example, a trading desk of a large financial organization offloads customer critical analyses from a traditional data warehouse environment to a high-performance analytic relational database platform. The performance gain reduces the analyses from hours to minutes, which gives the financial company a significant customer advantage. Analytic accelerators are not new, but the data management and analytics advances of big data now provide a range of platforms that can be used to offload certain performance-critical analytic workloads. The actual platform used will depend on the analytic workload and the types of data being managed.

New LOB Analytic Application This use case has potentially the biggest long-term business potential for big data. This is because the full power of the advances outlined in Figure 2 can be brought to bear on specific LOB problems and requirements. It enables organizations to build analytic solutions that were not previously possible and to expand the use of analytics to a broader set of business areas. As mentioned earlier, many of these solutions are new applications that are a hybrid combination of operational and analytic processing. Depending on the nature of the problem being addressed and the types of analytic processing and data involved, either a relational or non-relational system may be used to deploy the application.

The display advertising industry is an example of how analytics are being used by new industries and business areas. There are organizations in this market sector that specialize in helping companies place advertisements on various web properties. These organizations calculate the fair market value of tens of thousands of ads per second, bid for appropriate ad space, place the ads, and measure the effectiveness of ad campaigns in terms of product sales and revenue. These organizations are small and have limited IT resources for supporting the processing and analysis of huge amounts of data very rapidly. The hybrid operational/analytic applications involved would not have been possible to build prior to the innovations around big data and analytics outlined in this paper.

Investigative Computing Platform An investigative computing platform provides an analytic playground for data scientists to explore data and experiment with different analytic algorithms. The output may result in a new LOB analytic application, improved models and analytics or new types of data and analyses that can be migrated into a production decision-making environment. Companies are using these platforms to experiment with new types of data and new algorithms. Several retailers, for example, are experimenting with social computing data to determine how different types of customers use

Big data platforms can be used to offload certain performance-critical analytic workloads

Display advertising is an example of using analytics in new industries and business areas



different social computing channels, the types of social data that are valuable for measuring customer reaction and satisfaction, and so forth.

GETTING STARTED WITH BIG DATA AND ANALYTICS The deployment of big data and associated analytic technologies is an evolutionary process. Many of the first companies to deploy big data projects began by building standalone solutions. As these first companies have gained experience from these initial projects they have begun developing a longer term and more cohesive strategy for future work that can leverage all categories of analytics. Such a big data strategy requires four key things to be done if future projects are to fully exploit the business value of big data. These are:

Build a culture that understands the benefits of analytics An analytics-savvy culture starts with employees who are skilled in exploring the data, understanding its implications and applying insight. These employees need descriptive intelligence to better understand what has happened, diagnostic intelligence to understand why it happened, predictive intelligence to find patterns and see what is likely to happen, and prescriptive intelligence that helps them understand the best solutions to solve a business problem or satisfy a business objective. These employees can help the organization extend the use of analytics to a broader set of users. To drive maximum value, however, organizations also need to move toward analytics-driven business processes and practices.

Be pragmatic about governance and security There are varying degrees of risk in every organization and business decision. These risks vary by industry and business area, for example marketing versus finance. The use of big data and analytics requires a pragmatic balance between risk and providing easier access to information and analytics. Regardless, organizations must proactively identify, understand, and manage risk and embed the appropriate level of governance and security into all analytic processes. They also need to develop policies that identify when data can be discarded or archived in order to meet regulatory requirements.

Extend the existing analytics and data warehousing infrastructure to support big data and all types of analytics Organizations are recognizing the value of all forms of data from operational transaction data to social computing, mobile computing and machine sensor data. To be able to capitalize on the business value of this data, an organization must evolve its existing analytics and data management platforms to support big data projects.

Improve business and IT cooperation through new roles Companies leading the charge toward the use of big data are creating new roles to improve business and IT interaction and to highlight the importance of analytics in their organizations. Examples here include Chief Analytics Officer, Chief Data Scientist and Chief Data Officer.

CONCLUSION Big data and analytics innovations fuel a new wave of business value for organizations. True value is achieved from big data by the analytics and analytic

Four key things are required if projects are to fully exploit the business value of big data

An analytics culture

Pragmatic data governance and security

An extended data warehouse infrastructure

New business roles for enabling analytics



solutions that can be created from a hybrid of new and existing data systems and the new types of business applications they can support. These applications allow business users to employ analytics to improve the agility and competiveness of the business. Of course big data, like any technology, is not a panacea to solving all of the business and IT problems that exist in organizations today. However, done right, big data and analytics not only improve existing business processes, but also enable the business to use analytics to create new products and services and remain competitive.

About BI Research BI Research is a research and consulting company whose goal is to help organizations understand and exploit new developments in business intelligence, data integration and data management.

Documents

Converting Hype into Value Final - IBM Big Data & Analytics Hub · 2013-10-18 · Converting Hype into Value With Big Data and Analytics Colin White, BI Research ... The objective