Business Intelligence ……industry perspective

Kishaloya Roychowdhury

Koushik Das

Background : -

• IT Enablement came into existence targeting improvement of enterprise operations through

• Automation• Decreasing delays• Increasing accuracy & reducing ‘rework’• Reducing cost• Providing more room to explore new ways of revenue

• In older days• Business was less complex (geography bound, easier needs, limited & known

customer base, less ‘competition’, less ‘regulations’, less diversity in ‘product landscape’ etc.)

• Information volume was less & could be manually managed• Simple ‘management’ like Cost & Profit was enough to run a business• It was difficult to envision the ‘outcome’

• IT systems were mostly operational systems• Management Information System used to replace manual management

reporting

MIS Reporting - Overview

• A management information system (MIS) is a subset of the overall internal controls of a business covering the application of people, documents, technologies, and procedures to solve business problems such as costing a product, service or a business-wide strategy

• Management information systems are distinct from regular information systems in that they are used to analyze other information systems applied in operational activities in the organization.

• It has been described as, "MIS 'lives' in the space that intersects technology and business. MIS combines technology with business to get people the information they need to do their jobs better/faster/smarter.

Old MIS Systems• Generally a few summary reports and a few

detailed reports grouped for a business function and menu having multiple such groups

• No well thought of framework for organizing, automating and analyzing business methodologies, metrics, processes and systems that drive business performance

• Difficult to figure out ‘cause and effect’ relationships

• Manual detection of problem points from a group of detailed reports

• Decision making more dependent on intuition

New Generation MIS Systems (also termed as performance management systems)

• Based on a sound framework for organizing, automating and analyzing business methodologies, processes and systems that drive business performance.

• Business processes are aligned with Strategies and KPIs are aligned with business processes

• Status indicators (KPI) set with defined target and/or tolerance ranges

• KPIs are published into a dashboard / scorecard with the ability to drill down to detailed analysis or trend reports.

Importance of Reporting & Analytics

• Common needs of reporting & analytics from ages –• Understand the health of the Business at any organizational levels• Informed decision making at tactical & strategic level• Regulatory Compliance

• Today’s need under the backdrop of ‘global competition’, ‘economic rollercoaster’

• Optimized but cost effective operations• Differentiation in the marketplace• Revenue protection and sustainable growth

Early adopters ride the wave

ANTICIPATE

TRANSFORM

AWARE

BI – Some real life industry needs

• Retail – – Customer Intelligence– Product Pricing & Store Optimization– Right budgeting

• Finance –– Right channel adoption– Intelligent customer service– Reduce financial risk

• Healthcare –– Right care at the right time in the right setting– Disease management & Case management– Removal of behavioral barrier of doctors, patients

• Cross Industry –– Cost reduction & safe revenue – (waste management & innovative practice

adoption)– Regulatory Compliance– Performance Management– Risk & Fraud Management

Challenges faced in the Industries

• It’s a sea of information• IT landscape (hardware, software, application) grew too large• No enterprise-wide standard (process, information, IT)• Information is not ‘trusted’• Cross business function information gathering is dependent on huge manual

effort & error prone

• Some common issues faced by decision makers• Data is scattered everywhere across our organization. Where do I look ?• It takes forever to get the information I need to do my job• When I do get it, it’s wrong• We have mountains of data, but I can’t figure out what’s important• It takes so long to get the data that I don’t have any time left over to analyze it• I want it to be easy to see my data in every possible combination. Just let me point and

click my way to an answer• I want a historical view of the business or make future projections• How can I plan based on the lessons learned and future projection

KPIs : selected measures of business performance

Carefully selected set of measures derived from strategies, goals and objectives that represents a tool to communicating strategic direction to the organization for motivating change.

These form the basis to plan, budget, structure the organization and to control results.

Customer Measures

% Sales of New ProductsCustomers AcquiredCustomer SatisfactionCustomer Retention

Financial Measures

Market Share% Revenue from New ProductsTransportation costs (costs/mile)

R & D and Human Resource

New Product IntroductionManagement SkillsEmployee Turnover

Internal Process Measures

Product Time to MarketUnit Manufacturing CostDays Supply to inventory

% revenue from new products

It is the ratio of money gained or lost (realized or unrealized) on the product relative to the amount of money invested on the same

Customer retention rates

Among the total customer, what are customers who are staying with the company after specific period of time. Its generally calculated yearly basis

Customer satisfaction Customer satisfaction is a measure of how products and services supplied by a company meet or surpass customer expectation, for a particular service/product line. Generally estimated through survey and using a scoring model on a pre-defined scale.

Conceptual View of Enterprise Business Intelligence

BIReal time BI, Embedded Analytics, Operational Dashboard

Tactical BI

Strategic BI

Operational BI

“Capture”

Distributed Operational Systems, External Data Providers, Unstructured Data Sources

“Collate”

Centralized Information Warehouse

“Deliver”

Distributed Analytic Applications

Operate

PlanStrategize

Dashboards, Reporting, Analysis, Planning, Budgeting,Mining & Predictive Analysis

Scorecards, Reporting, Planning, Budgeting, Performance Management

Executives

Departments, Managers

Touch Points- Customer, Vendor,

Partner, Organization Units

Logical View of Enterprise Business Intelligence

LAN

Vendor

ENTERPRISEDWH

BATCH

Near RealTime

REAL TIMEDATA INTEGRATION

REAL TIMEOFFER OPTIMIZATION

TOUCHPOINTS

WRITE-BACK

WRITE-BACK

DYNAMIC SCORING& SEGMENTING

REAL TIMECAMPAIGN MANAGEMENT

REAL TIMEOFFER SELECTION

REVIEW

SEGMENTATION/PRICINGMODELING

REAL TIME CAMPAIGNRESPONSE INTEGRATION

REAL TIME SEGMENTMIGRATION

DATA MANAGEMENT DW MANAGEMENT DISTRIBUTION MANAGEMENT PERSONALZATION & DELIVERY

INFORMATION MANAGEMENT PROCESSES

BATCHDATA INTEGRATION

PORTAL

UNSTRUCTUREDDATA STORE

Conceptual DW Definition

• Data warehousing is a program dedicated to the delivery of ‘Enterprise wide view’ of information which advances decision making, improves business practices, and empowers workers.

• The components, or layers, include the following:– Business Architecture– Information Architecture– Applications Architecture– Data Architecture– Technology Infrastructure

• What a EDW is NOT– A single integrated database or computer application– Not a duplication of every piece of data that exists in the Corporation – Up-to-the-minute reporting environment– A place to clean-up source system data accuracy issues– A means to perform the data conversion process for legacy system

replacement projects

EDW versus Data Mart

EDW Data MartIntegrated (shared definitions)

Supports standard corporate definitions

Feeds Data Marts

Highest level of required granularity

Resides in a single integrated environment

Subject specific

Can be made of one or many datasets and/or data cubes

Accessed by the business users

Generally summarized

May reside on various computer platforms and environments

OLTP (online transaction processing) vs OLAP (online analytical processing)

• Organized around applications• Nonintegrated data• Different key structures• Different naming conventions• Different file formats

• No time series analysis• Data relationships constantly change• Changes are instantaneous• Limited history, 60-90 days

• Place an order for a product• Look up price for a product• Apply discount• Assign shipper• Trigger inventory pick-list• Verify shipment of product• Create invoice for the product• Apply credit to sales representative

• Organized around subject areas• Integrated data• Standardized key structures• Standardized naming conventions• Standardized file formats

• Time series analysis• Data is static over time• Series of data snapshots• Snapshots create historical database, often greater

than two years

• What type of customers are ordering this product?• Who are my top 10% accounts? By name, by

revenue, by profitability, by region?• What have been the product purchase patterns over

the past three years?• How are these different by customer segments? By

sales rep? By store?• Which shippers have the best on time delivery

records ?• How does this vary by shipment size? By season of

year?

Data organization & integration

Time Handling

Usage Examples

Essential for running the companyEssential for watching the company

OLTP OLAP

Information Transformation

Information

Knowledge

Intelligence

Operational System

Data Warehouse

Data Marts

BI solutions

Data Sources

• ERP or Custom implementations supporting operational need:

– HCM– CRM– Sales & Marketing– Order, Inventory – Procurement etc

• Manual systems mainly either in skill areas or around niche business functions:

– Planning & Budgeting– Customer profiling– Sales Rep Incentive Calculations etc

• Third Party Data: – Credit rating– Competitor data – Prescription data– Address & demographic data– Market research data etc.

ETL vs EAI

Areas EAI ETL

Definition Technology solution that enables systems to communicate

Process designed by users to extract, transform, and load data from one or more sources to a target data repository

Performance Optimization

System is aimed at reducing the response time for a single user request or update

System is aimed at reducing total time to create the unified historical record

Integration Applications Data

Focus Operational & Strategic Operational

Business Case IT, e-business, Better Workflow, Data entry once

Business Intelligence, Decision making, large volume, complex transformation, data quality

Time Real Time Batch (moving to real time)

Data Transactional-small Historical-enormous

Metadata Limited--Message metadata Rich--dimensional metadata

Transformations Format oriented--Code supported

Analytic, Joins, Aggregations, function & formulae based

Volume Single transactionsMessages/second (KB)

Days or weeks of dataRecords per min (GB)

Targets OLTP APICode supported

Relational StructuresNative connectivityCodeless

Extracts Data Using API’s Directly from database or using application adapters

System Admin Involvement

EAI requires no system administrator involvement. Once implemented, EAI is a technology solution that is transparent to end users.

ETL requires extensive system administrator involvement

ENTERPRISE BUS

EXTRACT

TRANSFORM

LOADMetadata

ETLTransformation

EDW

Operational Data Store

• An operational data store (or "ODS") is a database designed to integrate data from multiple sources to make analysis and reporting easier. An ODS is usually designed to contain low level or atomic (indivisible) data (such as transactions and prices) with limited history that is captured "real time" or "near real time“.

• According to Bill Inmon, the originator of the concept, an ODS is "a subject-oriented, integrated, volatile, current-valued, detailed-only collection of data in support of an organization's need for up-to-the-second, operational, integrated, collective information."

• In practice ODS tend to be more reflective of source structures in order to speed implementations and provide a truer representation of production data.

• An "ODS" is not a replacement or substitute for an enterprise data warehouse but in turn could become a source.

Operational Data Store Data Warehouse

Characteristics Data Focused Integration from Transaction Processing Systems, A better integrated picture of source systems

Subject Oriented, Integrated, Non-Volatile, Time Variant

Age Of The Data Current, Near Term (Today, LastWeek’s) Historic (Last Month, Qtrly, FiveYears)

Primary Use Day-To-Day Decisions, Tactical Reporting, Current Operational Results

Long-Term Decisions, Strategic Reporting, Trend Detection

Frequency Of Load Real Time, Near Real Time, Twice Daily , Daily, Weekly

Daily, Weekly, Monthly, Quarterly, Bi-Yearly, Yearly

Staging Area

• Definition: Staging Area is a temporary location where data from source systems is copied and processed before loading into the target system, most often a data warehouse.

• Minimizing processing on source systems – Extract only once– Proper timing of different extracts within source system schedules– Both table-centric and document-centric extraction can be applied as necessary

• Source data within own control– Incremental– Delta identification (Inserts, Updates, and Deletes)

• Reduce record set to be processed– From source systems– For downstream processes

• True delta: only those records that have truly changed– CRC (Cyclic Redundancy Checksum)– Column by column comparison

• Challenges in true delta identification e.g.– NULL comparisons (Null does not equal Null)– When only the column used to identify a source modification changes– Source system challenges

– Freedom of storage format and abstraction• Data format consistency, e.g.

– CCYYMMDD format for dates, Trim trailing spaces, NULL replacement, Data type conversions– Demoralization, Document-centric, Summarization

– Audit trail– Data Quality

Real time data needs

• The barrier between transactional systems — which run the business day to day — and decision support — which traditionally have engaged business intelligence issues around product, customer, and market trends — is fading away under the pressure of new and ever more demanding business scenarios in customer service, product distribution, and market dynamics

• A call center agent who has a customer on the phone at risk of going to the competition has 15 seconds to turn the situation around.

• Analytics are used to optimize operations. For companies like FedEx, package dynamics are not just transactional; they are critical path — literally in a strategic and tactical sense — requiring the redeployment of resources such as aircraft to optimize operations.

• Supplier scorecards — on-time deliveries, returns, defects, incomplete orders — reduce revenue losses from out-of-stock items and reduce markdown losses from overstocks.

• For those enterprises that have physical inventory, reducing inventory through a demand planning or forecasting data warehouse results in significant cost reductions.

• Data updates can only be as fast as the business processes that produce data.

• Data consumption can only be as fast as the warehouse.

Right Latency is the right thing to implement

Type Definition How it works Example

Simulated An end user at a work station executing self-service query and reporting or what-if analysis. Updates and roll-up calculations are performed in batch, delivered in interactive “think time.”

The results have been pre-computed and stored in the data warehouse for latter delivery as if the calculation were done in real time, but it is not.

Customerrecommendation

Right time A catch-all phrase meaning near-real time — tied to a specific technology such as change data capture to a database log.

Allows for a variety for response times, none committing to synchronous processing — allows for distribution by an ETL tool or message broker

Web log analysis

Real time The answer is absolutely the most up-to-date information physically possible in terms of both update and access.

Resources such as databases, networks, and CPUs are locked synchronously until a commit point is reached, at which time other concurrent processing may proceed.

Fraud detection

On time Data is updated and delivered according to policies, service-level agreement, or consensus.

Business groups tell IT how often they need to update and access data, and IT delivers data on that schedule.

Inventory